Support GPU input in `XGBClassifier`; deprecate the use of label encoder

[pseudotensor]

conda, xgboost 1.2.0 , rapids 0.14, Ubuntu 18.04 LTS, GeForce RTX 2080, cuda 10.0.

While the non-sklearn API can take cudf, the sklearn API fails with the error in the title.

Any data set works, just choose for creditcard.csv . For the target given can choose attached dataset
creditcard.csv.zip

i.e. this works:

import cudf
import pynvml
import numpy as np
import pandas as pd
import xgboost as xgb
from math import sqrt
from sklearn.model_selection import KFold, StratifiedKFold, GroupKFold
from sklearn.metrics import mean_squared_error,f1_score, accuracy_score
train = cudf.read_csv("creditcard.csv")
test = cudf.read_csv("creditcard.csv")
y = train['default payment next month']
train = train.drop(['default payment next month'], axis=1)
test = test.drop(['default payment next month'], axis=1)
from cuml import train_test_split
X_train, X_test, y_train, y_test = train_test_split(train, y, test_size=0.2, random_state=123)
dtrain = xgb.DMatrix(X_train,y_train)
dval   = xgb.DMatrix(X_test, y_test)
dtest  = xgb.DMatrix(test)
evallist = [(dval, 'validation'), (dtrain, 'train')]
num_round=5000
trained_model = xgb.train(
                        {
                          'learning_rate': 0.01,
                          'colsample_bytree' : 0.25,
                          'max_depth': 10,
                          'objective': 'binary:logistic',
                          'silent': True,
                          'tree_method':'gpu_hist',
                        },
                        dtrain,num_round, evallist,verbose_eval=250)
prediction = trained_model.predict(dtest)

but this does not work:

import cudf
import pynvml
import numpy as np
import pandas as pd
import xgboost as xgb
from math import sqrt
from sklearn.model_selection import KFold, StratifiedKFold, GroupKFold
from sklearn.metrics import mean_squared_error,f1_score, accuracy_score
train = cudf.read_csv("creditcard.csv")
test = cudf.read_csv("creditcard.csv")
y = train['default payment next month']
train = train.drop(['default payment next month'], axis=1)
test = test.drop(['default payment next month'], axis=1)
from cuml import train_test_split
X_train, X_test, y_train, y_test = train_test_split(train, y, test_size=0.2, random_state=123)

params = {'learning_rate': 0.01, 'colsample_bytree' : 0.25, 'max_depth': 10, 'silent': True, 'tree_method':'gpu_hist','n_estimators': 5000, 'verbose_eval': 250}
model = xgb.XGBClassifier(**params)
eval_set = [(X_test, y_test), (X_train, y_train)]
model.fit(X_train, y_train, eval_set=eval_set)
prediction = model.predict(test)

fails with:

>>> model.fit(X_train, y_train, eval_set=eval_set)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jon/miniconda3/lib/python3.6/site-packages/xgboost/sklearn.py", line 858, in fit
    self._le = XGBoostLabelEncoder().fit(y)
  File "/home/jon/miniconda3/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 219, in fit
    y = column_or_1d(y, warn=True)
  File "/home/jon/miniconda3/lib/python3.6/site-packages/sklearn/utils/validation.py", line 749, in column_or_1d
    shape = np.shape(y)
  File "<__array_function__ internals>", line 6, in shape
  File "/home/jon/miniconda3/lib/python3.6/site-packages/cudf/core/series.py", line 734, in __array_function__
    return cudf_func(*args, **kwargs)
TypeError: 'property' object is not callable
>>> 

I expect I'm doing something wrong since cudf functionality has been in xgboost for a long time now and I assume many people use the sklearn API, but I cannot see what I'm doing wrong. Thanks!

[hcho3]

@pseudotensor The eval_set argument should be in correct format:

eval_set = [(X_train, y_train), (X_test, y_test)]
model.fit(X_train, y_train, eval_set=eval_set)

See https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBClassifier.fit

[pseudotensor]

Hi @hcho3 , sorry, typo on my side. I fixed the above but the error still remains.

One does not even need to pass the eval_set. The problem is with "y", not the eval set.

[pseudotensor]

Note that model = xgb.XGBRegressor(**params) works, but model = xgb.XGBClassifier(**params) does not. I had similar error for xgboost 1.0.0, where label encoding the y failed with cudf error. Will post...

[hcho3]

It appears that the label encoder does not work with cuDF. Tagging @trivialfis

[pseudotensor]

I didn't see anyone post the same error, so wondering what to try. But, I can try rapids 0.15 or 0.16 to see if it's a bug in cudf that was fixed.

[pseudotensor]

No, rapids 0.15 fails in different but related way:

>>> model.fit(X_train, y_train)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jon/minicondadai_rapids15/lib/python3.8/site-packages/xgboost/sklearn.py", line 791, in fit
    self._le = XGBoostLabelEncoder().fit(y)
  File "/home/jon/minicondadai_rapids15/lib/python3.8/site-packages/sklearn/preprocessing/_label.py", line 239, in fit
    y = column_or_1d(y, warn=True)
  File "/home/jon/minicondadai_rapids15/lib/python3.8/site-packages/sklearn/utils/validation.py", line 72, in inner_f
    return f(**kwargs)
  File "/home/jon/minicondadai_rapids15/lib/python3.8/site-packages/sklearn/utils/validation.py", line 833, in column_or_1d
    y = np.asarray(y)
  File "/home/jon/minicondadai_rapids15/lib/python3.8/site-packages/numpy/core/_asarray.py", line 83, in asarray
    return array(a, dtype, copy=False, order=order)
  File "/home/jon/minicondadai_rapids15/lib/python3.8/site-packages/cudf/core/frame.py", line 1636, in __array__
    raise TypeError(
TypeError: Implicit conversion to a host NumPy array via __array__ is not allowed, To explicitly construct a GPU array, consider using cupy.asarray(...)
To explicitly construct a host array, consider using .to_array()
>>> 

[hcho3]

The issue is that sklearn's label encoder cannot accept a cuDF array. Right now, XGBClassifier uses sklearn's label encoder to map the labels to 0..[num_class] range.

[pseudotensor]

Can I ask a general related question? To what extent is cudf supported in xgboost? Are there documented issues for lack of support? Or is plan to fully support? Are there tests for cudf specifically? I'd suppose the full suite of tests could be done but input data X, y, eval_set could be cudf and that should catch a bunch of issues.

[hcho3]

@pseudotensor We have a suite of tests to ensure compatibility with cuDF. See this for an example. The inability to use cuDF arrays with XGBClassifier is actually a known issue: #5541.

[pseudotensor]

@hcho3 I see, thanks. I didn't find it in the issues since no explicit error was provided.

I guess April 15 is a long time ago. So I suppose there are many aspects of xgboost that don't support cudf?

[hcho3]

@pseudotensor The issue is really confined with XGBClassifier, as it embeds sklearn's label encoder. Other parts of XGBoost, such as xgb.train or xgb.DMatrix support cuDF fully.

I will write a short patch to throw an explicit error when a cuDF array is fed into XGBClassifier. In the long term, we'll need to revise XGBClassifier so that it doesn't rely on sklearn's label encoder. Some alternatives:

• Force users to perform label encoding on their own. This will break existing code using XGBClassifier.
• If the label is a cuDF series, convert it automatically into a NumPy array. This is inefficient (causing PCIe transfer) but does not break existing code.
• Use cuML's label encoder. cuML is a big dependency, so it will have to be an optional feature.

[pseudotensor]

It would be nice to be able to just skip the label encoder by choice, which would be work-around for this issue as well. Often times in a pipeline of work, xgboost would anyways be given an already-encoded target.

[hcho3]

@pseudotensor Thanks a lot for the suggestion. For this release (1.3.0), it should be feasible to add an option to skip the label encoder.

[hcho3]

@pseudotensor What do you think of the idea of eventually deprecating the use of label encoder inside XGBClassifier? The presence of label encoder makes it tricky for us to support additional array types. For now, when the option skip_label_encoder is set to False, we can throw a DeprecationWarning.

I wonder how difficult it is for users to modify their training code; potentially we may need a long deprecation period.

[pseudotensor]

Yes, sounds reasonable on both deprecation and long period.