[jvm-packages] Xgboost4spark 1.1.1 broken and consistently does not work

[ranInc]

Some models’ predictions fail on the following error:
Check failed: weights_.Size() == num_row_ (15363 vs. 15362) : Size of weights must equal to number of rows.
The numbers in the error are of course not always the same (but the subtraction is always 1).
The same data/model works on xgboost 0.9.

[trivialfis]

Size of weight is either equal to number of groups or number of rows. As you can come to this check, I assume you are not doing ranking.

xgboost/src/data/data.cc

Line 389 in eb067c1

return;

[ranInc]

I am not doing ranking.
I still did not understand what is wrong here - or why this model works on 0.9.

[trivialfis]

The number of weights should be equal to number of rows. Since the weight is defined for each data instance.

In later version of XGBoost we are adding a lots of checks to prevent user errors. If you are using Python or R XGBoost will even check your parameters.

[ranInc]

But I am not using weights....
there is not weightcol.

[ranInc]

Because of this bug I cannot upgrade to 1.1.1, and to spark 3.0 - because this error seems to also happen in 1.0.0.
I want to upload a parquet file with data and a model that produces this error, so you can investigate the issue - but I don't see where I can do that.

Do you any speculation to when this can be fixed?
Also is there a way to go around the issue?

[trivialfis]

Do you have something I can run and reproduce? The bug you described doesn't show up on our tests.

[ranInc]

I uploaded a zip file with the data folder and model folder.
replace "/tmp/xgboost_test/data" to wherever you unzip the data folder,
and "/tmp/xgboost_test/model" to wherever you unzip the model folder.
this is the scala test code:

import org.apache.spark.ml.PipelineModel
import org.apache.spark.sql.SparkSession

object XgboostTest {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder().enableHiveSupport().master("local").getOrCreate();
    try{
      val data = spark.read.parquet(("/tmp/xgboost_test/data"))
      val model = PipelineModel.load("/tmp/xgboost_test/model")
      val predictions = model.transform(data)
      predictions.persist()
      predictions.count()
      predictions.show()
    }finally {
      spark.close()
    }
  }
}

xgboost_test.zip

[trivialfis]

Thanks! Let me check that later this week. I'm not familiar with spark so might take some time.

[ranInc]

Hi,
any update?

[trivialfis]

Not yet.

[ranInc]

still nothing?