Skip to content

Move skl eval_metric and early_stopping rounds to model params. #6751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 28, 2021
Merged

Move skl eval_metric and early_stopping rounds to model params. #6751

merged 2 commits into from
Oct 28, 2021

Conversation

trivialfis
Copy link
Member

@trivialfis trivialfis commented Mar 15, 2021 •
edited
Loading

Close #6735.
Close #6445.

See the comment below for a brief summary.

@trivialfis
Copy link
Member Author

A summary of the changes in this PR:

A new parameter custom_metric is added to train and cv to distinguish the behaviour from the old feval. And feval is deprecated. The new custom_metric receives transformed prediction when the built-in objective is used. This enables XGBoost to use cost functions from other libraries like scikit-learn directly without going through the definition of the link function.

eval_metric and early_stopping_rounds in sklearn interface are moved from fit to __init__ and is now saved as part of the scikit-learn model. The old ones in fit function are now deprecated. The new eval_metric in __init__ has the same new behaviour as custom_metric.

Added more detailed documents for the behaviour of custom objective and metric.

@codecov-commenter
Copy link

codecov-commenter commented Oct 22, 2021 •
edited
Loading

Codecov Report

Merging #6751 (ee810ec) into master (1568599) will increase coverage by 0.09%.
The diff coverage is 92.40%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #6751      +/-   ##
==========================================
+ Coverage   83.71%   83.80%   +0.09%     
==========================================
  Files          13       13              
  Lines        3892     3939      +47     
==========================================
+ Hits         3258     3301      +43     
- Misses        634      638       +4
Impacted Files Coverage Δ
python-package/xgboost/dask.py 82.72% <83.33%> (-0.07%) ⬇️
python-package/xgboost/sklearn.py 91.25% <89.74%> (+0.10%) ⬆️
python-package/xgboost/training.py 94.83% <96.00%> (-0.14%) ⬇️
python-package/xgboost/callback.py 97.71% <100.00%> (+<0.01%) ⬆️
python-package/xgboost/core.py 84.55% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7593fa9...ee810ec. Read the comment docs.

@trivialfis
Copy link
Member Author

I will move the callback parameter too in next PR. This one is huge as it's.

trivialfis
trivialfis commented Oct 24, 2021
View reviewed changes
@trivialfis
Move skl eval_metric early_stopping rounds to model params.
1263ef6 
These 2 parameters are now model parameters that can be set at constructor and
`set_params` method.

Doc and test.

Fix return.

non breaking.

mypy.

Configuration.

Add test/example.

Use output-margin only when objective is provided.

hint.

Deprecation notice.

Document.

Lint.

test.

mypy.

Add document.

Cleanup.

Add custom_metric.

Fix softmax.

test.

Prevent softmax.

Lint.

fix.

Doc.

Contents.

Note.

More documents.

Format.

Fix.

Fix dask.

Fix dask.

Check RF.

Apply suggestions from code review
@trivialfis trivialfis merged commit 45aef75 into dmlc:master Oct 28, 2021
@trivialfis trivialfis deleted the metric-decorator branch October 28, 2021 09:20
@trivialfis trivialfis mentioned this pull request Oct 28, 2021
Merged
@harupy harupy mentioned this pull request Oct 28, 2021
Merged
27 tasks
trivialfis added a commit to trivialfis/xgboost that referenced this pull request Nov 9, 2021
@trivialfis
[backport] Move skl eval_metric and early_stopping rounds to mode…
f935b8d 
…l params. (dmlc#6751)

A new parameter `custom_metric` is added to `train` and `cv` to distinguish the behaviour from the old `feval`.  And `feval` is deprecated.  The new `custom_metric` receives transformed prediction when the built-in objective is used.  This enables XGBoost to use cost functions from other libraries like scikit-learn directly without going through the definition of the link function.

`eval_metric` and `early_stopping_rounds` in sklearn interface are moved from `fit` to `__init__` and is now saved as part of the scikit-learn model.  The old ones in `fit` function are now deprecated. The new `eval_metric` in `__init__` has the same new behaviour as `custom_metric`.

Added more detailed documents for the behaviour of custom objective and metric.
@trivialfis trivialfis mentioned this pull request Nov 9, 2021
Closed
@BenEfrati BenEfrati mentioned this pull request Jan 22, 2023
Merged
@trivialfis trivialfis mentioned this pull request Oct 24, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RAMitchell RAMitchell RAMitchell approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[RFC] Move eval_metric from XGBModel.fit to __init__ and accept sklearn style metrics. Use transformed prediction in custom evaluation metric.
3 participants
@trivialfis @codecov-commenter @RAMitchell