TY - JOUR
T1 - Convergence of Stochastic Gradient Descent Schemes for Łojasiewicz-Landscapes
AU - Dereich , Steffen
AU - Kassing , Sebastian
JO - Journal of Machine Learning
VL - 3
SP - 245
EP - 281
PY - 2024
DA - 2024/09
SN - 3
DO - http://doi.org/10.4208/jml.240109
UR - https://global-sci.org/intro/article_detail/jml/23416.html
KW - Stochastic gradient descent, Stochastic approximation, Robbins-Monro,
Almost sure convergence, Łojasiewicz-inequality.
AB - <p style="text-align: justify;">In this article, we consider convergence of stochastic gradient descent schemes (SGD), including
momentum stochastic gradient descent (MSGD), under weak assumptions on the underlying landscape. More
explicitly, we show that on the event that the SGD stays bounded we have convergence of the SGD if there is
only a countable number of critical points or if the objective function satisfies Łojasiewicz-inequalities around
all critical levels as all analytic functions do. In particular, we show that for neural networks with analytic
activation function such as softplus, sigmoid and the hyperbolic tangent, SGD converges on the event of
staying bounded, if the random variables modelling the signal and response in the training are compactly
supported.</p>