[Feature] Implement AdaFactor optimizer

[kozistr]

Problem (Why?)

Implement AdaFactor optimizer.

Solution (What/How?)

• Implement AdaFactor optimizer

Other changes (bug fixes, small refactors)

nope

Notes

nope

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.72%. Comparing base (fff34af) to head (0533250).
Report is 1624 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #107    +/-   ##
========================================
  Coverage   99.71%   99.72%            
========================================
  Files          39       40     +1     
  Lines        3125     3236   +111     
========================================
+ Hits         3116     3227   +111     
  Misses          9        9            

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.