pytorch-optimizer v2.5.1

Change Log

Feature

• Implement Ali-G optimizer, #115, #116
  • Adaptive Learning Rates for Interpolation with Gradients
• Implement create_optimizer() to build the optimizer, #116

Bug

• __str__ method, #118, #119 (thanks to @Interpause)

pytorch-optimizer v2.5.0

Change Log

Feature

• Implement AdaFactor optimizer, #107
  • Adaptive Learning Rates with Sublinear Memory Cost
• Implement NovoGrad optimizer, #109
  • Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks
• Implement Apollo optimizer, #108
  • An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization
• Implement Lion optimizer, #113
  • Symbolic Discovery of Optimization Algorithms

pytorch-optimizer v2.4.2

Change Log

Bug

• Fix to deep-copy inverse preconditioners

Deps

• Support Pytorch 2.0, #106 (related to #105)

Docs

• Update Scalable Shampoo docstring (more parameter guides), #106
  • documentation : https://pytorch-optimizers.readthedocs.io/en/latest/optimizer_api.html#scalableshampoo

pytorch-optimizer v2.4.1

Change Log

Feature

• Rename the new Shampoo to ScalableShampoo. #103
• Implement the old(?) version of Shampoo optimizer. #103
• Support SVD method to calculate the inverse pth root matrix. #103
  • to boost the M^{-1/p} calculation, performs batched SVD when available.
• Implement AdamS optimizer. #102
• Support stable weight decay option for Adai optimizer. #102

Bug

• Fix compute_power_svd() to get a singular value. #104

pytorch-optimizer v2.4.0

Change Log

Feature

• Implement D-Adaptation optimizers (DAdaptAdaGrad, DAdaptAdam, DAdaptSGD), #101
  • Learning rate free learning for SGD, AdaGrad and Adam
  • original implementation: https://github.com/facebookresearch/dadaptation
• Shampoo optimizer
  • Support no_preconditioning_for_layers_with_dim_gt (default 8192)

Improvement

• refactor/improve matrix_power(), unroll the loop due to the performance, #101
• speed-up/fix power_iter(), not to deep-copy mat_v. #101

Docs

• D-Adaptation optimizers & Shampoo utils

pytorch-optimizer v2.3.1

Change Log

Feature

• more add-ons for Shampoo optimizer, #99
  • implement moving_average_for_momentum
  • implement decoupled_weight_decay
  • implement decoupled_learning_rate
  • supports more grafting (RMSProp, SQRT_N)
  • supports more PreConditioner (ALL, INPUT)

Docs

• apply pydocstyle linter, #91

Refactor

• deberta_v3_large_lr_scheduler, #91

ETC

• add more Ruff rules (ICN, TID, ERA, RUF, YTT, PL), #91

pytorch-optimizer v2.3.0

Change Log

Feature

• re-implement Shampoo Optimizer (#97, related to #93)
  • layer-wise grafting (none, adagrad, sgd)
  • block partitioner
  • preconditioner
• remove casting to fp16 or bf16 inside of the step() not to lose consistency with the other optimizers. #96
• change some ops to in-place operations to speed up. #96

Fix

• fix exp_avg_var when amsgrad is True. #96

Refactor

• change linter from Pylint to Ruff, #97

pytorch-optimizer v2.2.1

Change Log

Feature

• Support max_grad_norm (Adan optimizer)
• Support gradient averaging (Lamb optimizer)
• Support dampening, nesterov parameters (Lars optimizer)

Refactor

• move step parameter from state to group. (to reduce computation cost & memory)
• load betas by group, not a parameter.
• change to in-place operations.

Fix

• fix when momentum is 0 (Lars optimizer)

pytorch-optimizer v2.2.0

Change Log

• Implement GSAM (Surrogate Gap Guided Sharpness-Aware Minimization) optimizer, ICLR 22

pytorch-optimizer v2.1.1

Change Log

#90

Feature

• Support gradient centralization for Adai optimizer
• Support AdamD debias for AdaPNM optimizer
• Register custom exceptions (e.g. NoSparseGradientError, NoClosureError, ...)

Documentation

• Add API documentation

Bug

• Fix SAM optimizer