TY - JOUR
T1 - A Convergence Study of SGD-Type Methods for Stochastic Optimization
AU - Xiao , Tiannan
AU - Yang , Guoguo
JO - Numerical Mathematics: Theory, Methods and Applications
VL - 4
SP - 914
EP - 930
PY - 2023
DA - 2023/11
SN - 16
DO - http://doi.org/10.4208/nmtma.OA-2022-0179
UR - https://global-sci.org/intro/article_detail/nmtma/22116.html
KW - SGD, momentum SGD, Nesterov acceleration, time averaged SGD, convergence analysis, non-convex.
AB - <p style="text-align: justify;">In this paper, we first reinvestigate the convergence of the vanilla SGD
method in the sense of $L^2$ under more general learning rates conditions and a more
general convex assumption, which relieves the conditions on learning rates and does
not need the problem to be strongly convex. Then, by taking advantage of the Lyapunov function technique, we present the convergence of the momentum SGD and
Nesterov accelerated SGD methods for the convex and non-convex problem under $L$-smooth assumption that extends the bounded gradient limitation to a certain extent.
The convergence of time averaged SGD was also analyzed.</p>