xgboost 1.1.1 pred failed, while 0.90 pred success

[zwqjoy]

1line_inst:
0 999:2000.000000

#model1.bin train with xgb0.90
#model2.bin train with xgb1.1.1

CODE1

import xgboost as xgb

print(xgb.__version__)
pred = xgb.DMatrix("1line_inst")

bst2 = xgb.Booster({'nthread': 4})  # init model
bst2.load_model('model2.bin')  # load data
print(bst2.predict(pred))

OUTPUT1

1.1.1
[15:16:14] 4x998 matrix with 2105 entries loaded from 1line_inst
Traceback (most recent call last):
  File "pred_zxb.py", line 12, in <module>
    print(bst2.predict(pred))
  File "/Users/zengwenqi/DXM/DXM-codebase/baidu/rimrdp/pipelines/venv/lib/python3.7/site-packages/xgboost/core.py", line 1580, in predict
    ctypes.byref(preds)))
  File "/Users/zengwenqi/DXM/DXM-codebase/baidu/rimrdp/pipelines/venv/lib/python3.7/site-packages/xgboost/core.py", line 190, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [15:16:14] /Users/travis/build/dmlc/xgboost/src/learner.cc:1070: Check failed: learner_model_param_.num_feature == p_fmat->Info().num_col_ (1104 vs. 998) : Number of columns does not match number of features in booster.
Stack trace:
  [bt] (0) 1   libxgboost.dylib                    0x0000000118c101c0 dmlc::LogMessageFatal::~LogMessageFatal() + 112
  [bt] (1) 2   libxgboost.dylib                    0x0000000118cbda2a xgboost::LearnerImpl::ValidateDMatrix(xgboost::DMatrix*) const + 282
  [bt] (2) 3   libxgboost.dylib                    0x0000000118cbdb13 xgboost::LearnerImpl::PredictRaw(xgboost::DMatrix*, xgboost::PredictionCacheEntry*, bool, unsigned int) const + 67
  [bt] (3) 4   libxgboost.dylib                    0x0000000118cadecc xgboost::LearnerImpl::Predict(std::__1::shared_ptr<xgboost::DMatrix>, bool, xgboost::HostDeviceVector<float>*, unsigned int, bool, bool, bool, bool, bool) + 732

CODE2

import xgboost as xgb

print(xgb.__version__)
pred = xgb.DMatrix("1line_inst")

bst2 = xgb.Booster({'nthread': 4})  # init model
bst2.load_model('model1.bin')  # load data
print(bst2.predict(pred))

OUTPUT2

0.90
[15:20:51] 1x1000 matrix with 1 entries loaded from 1line_test
[0.0208639]

[trivialfis]

So, it seems your model is trained on a dataset with different shape?

[zwqjoy]

@trivialfis but the pred inst only have one feature, index 999, the 2 models training with max index feats is 2000+

[trivialfis]

There were some heuristics around this that never got documented nor tested. I'm trying to re-establish them in a way that can be precisely documented.

[trivialfis]

learner_model_param.num_feature == p_fmat->Info().num_col (1104 vs. 998)

Also, I think there's something wrong here.

the 2 models training with max index feats is 2000+

From the error message your model has 1104 number of feature while your training dataset has 2000+ features.

[hcho3]

@trivialfis @RAMitchell

From #5856 (comment):

In the older versions we had some heuristics to predict on this kind of mismatching number of features, like treating them as missing if the input has less trailing features, or ignore them if input has more. After a offline chat with @RAMitchell , neither of us want to restore that heuristic as it brings more issues than benefits.

I agree that the heuristic has issues, but we need to make a provision for reading LIBSVM files with deficient number of features. The LIBSVM files unfortunately do not store the number of columns, and the column dimension is inferred from the maximum feature ID that occurs. For example, it should be possible to feed in the LIBSVM file

1 0:100

into a model trained with 10 features. The correct way to interpret this LIBSVM file is to assign 100 for the first feature and mark all the other features as missing. IMHO, rejecting LIBSVM files like this would make the built-in LIBSVM reader unusable.

[ranInc]

I also encountered this issue, and the same model works on the same data on 0.9 as described here.

It might have a similarity with the following issue I opened that is happening more often (to me at least):
https://github.com/dmlc/xgboost/issues/5848

[trivialfis]

@hcho3 Could you please help establishing the heuristics we need to support? I'm quite confused by the old heuristics.

• We should support predicting on DMatrix with less features than model. Those features will be treated as missing.
• Should we support predicting on DMatrix with more features than model?
• Should we output a warning on mismatching number of features? With libsvm, this can be quite verbose.

[trivialfis]

Also, on Python side, there's a validate feature option, should we lower it down to C++ and make it a parameter?

[hcho3]

We should support predicting on DMatrix with less features than model. Those features will be treated as missing.

Yes.

Should we support predicting on DMatrix with more features than model?

No. We should raise an error in this case.

Should we output a warning on mismatching number of features? With libsvm, this can be quite verbose.

No. If we only adopt the first heuristic, the behavior should be fairly predictable, I think.

on Python side, there's a validate feature option, should we lower it down to C++ and make it a parameter?

Now that feature names and types in the C++ layer, we can. I'll let you decide.

[lucagiovagnoli]

I've just encountered this issue as well. The newer xgboost > 1.0.0 is printing a lot of warnings to screen:

WARNING: /xgboost/src/learner.cc:979: Number of columns does not match number of features in booster

We should support predicting on DMatrix with less features than model. Those features will be treated as missing

I agree that we should support this case.

If it helps, we never saw this issue on xgboost=0.82

[hcho3]

@lucagiovagnoli I think you have a different issue. XGBoost uses zero-based indexing for features by default, so the diabetes dataset would be recognized as having 9 features. To use one-based indexing, append ?indexing_mode=1 to the file path:

dtrain = xgb.DMatrix('./diabetes.libsvm?indexing_mode=1')

Alternatively, you may use sklearn.datasets.load_svmlight_file().

[lucagiovagnoli]

Hi @hcho3 , sorry I had deleted the comment before you replied because I noticed it was my mistake :)
Thanks for the insight, I'll try that!

Original comment for reference:

Actually, this is happening using this dataset with no missing column: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/diabetes
I wonder if the problem is that columns in the libsvm file are numbered from 1 to 8 without using zero?

[trivialfis]

Will look into this.