pytorch-optimizer v2.5.0

Change Log

Feature

• Implement AdaFactor optimizer, #107
  • Adaptive Learning Rates with Sublinear Memory Cost
• Implement NovoGrad optimizer, #109
  • Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Implement Apollo optimizer, #108
  • An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization
• Implement Lion optimizer, #113
  • Symbolic Discovery of Optimization Algorithms