论文标题
非凸面景观中随机自适应算法的渐近研究
Asymptotic study of stochastic adaptive algorithm in non-convex landscape
论文作者
论文摘要
本文研究了广泛用于优化和机器学习的自适应算法的一些渐近特性，其中包括Adagrad和Rmsprop，它们参与了大多数BlackBox深度学习算法。我们的设置是非凸面景观优化的观点，我们考虑一个一次性尺度参数化，我们考虑在微型批次中可以使用或不使用这些算法的情况。我们采用随机算法的观点，并在使用降低的阶梯尺寸的观点朝目标函数的临界点集时，确定这些方法几乎确定的收敛。通过对噪声的温和额外假设，我们还获得了朝向该功能最小化器集的收敛性。在我们的研究中，我们还以\ cite {ghadimilan}的作品的方式获得了方法的“收敛速率”。
This paper studies some asymptotic properties of adaptive algorithms widely used in optimization and machine learning, and among them Adagrad and Rmsprop, which are involved in most of the blackbox deep learning algorithms. Our setup is the non-convex landscape optimization point of view, we consider a one time scale parametrization and we consider the situation where these algorithms may be used or not with mini-batches. We adopt the point of view of stochastic algorithms and establish the almost sure convergence of these methods when using a decreasing step-size point of view towards the set of critical points of the target function. With a mild extra assumption on the noise, we also obtain the convergence towards the set of minimizer of the function. Along our study, we also obtain a "convergence rate" of the methods, in the vein of the works of \cite{GhadimiLan}.
