Metrics for categorical forecasting

[jonasblanc]

Checklist before merging this PR:

• Mentioned all issues that this PR fixes or addresses.
• Summarized the updates of this PR under Summary.
• Added an entry under Unreleased in the Changelog.

Fixes #.

Summary

Implement metrics for categorical forecasting.

TODO:

• extend list of supported categorical metrics
• categorical specific tests
• adapt metrics decorator to support per class score and cross-class averaging

Other Information

[codecov]

Codecov Report

Attention: Patch coverage is 74.44444% with 23 lines in your changes missing coverage. Please review.

Project coverage is 95.03%. Comparing base (8e60174) to head (96492e7).

Files with missing lines	Patch %	Lines
darts/metrics/categorical_metrics.py	74.44%	23 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2767      +/-   ##
==========================================
- Coverage   95.23%   95.03%   -0.21%     
==========================================
  Files         145      146       +1     
  Lines       15092    15182      +90     
==========================================
+ Hits        14373    14428      +55     
- Misses        719      754      +35     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[dennisbader]

Thanks @jonasblanc, this is a nice start 🚀

Some thoughts:

• remove per time step metric support for all metrics, except maybe the accuracy itself
• add support for probabilistic series:
  • if preds are sampled, take the label with highest count
  • if preds are likelihood parameters, take the label with highest probability
• for a first collection, we should have accuracy, precision, recall, f1
• if we don't add timeseries support for confusion matrix, etc. then we should make these functions private

[dennisbader]

We could call it classification.py then it's a bit more nice to import:

from darts.metrics.classification import acc

[jonasblanc]

Maybe forecasting_classification in order not to be confused with up-coming TS classification ?

[dennisbader]

Here you give already the aggregated value over time. But your metric is defined as a per time step metric since it accepts the time_reduction.

Either you return here the time dependent accuracy (e.g. boolean 1 / 0 whether it's a hit per time step) and add a mean accuracy metric that aggregates (see metrics.ae and metrics.mae as an example), or you call it macc directly and remove the time_reduction.

[dennisbader]

these metrics below should not allow the time_reduction but should always return the aggregated metric