[CI] Flaky unit test for SHAP

[hcho3]

https://xgboost-ci.net/blue/organizations/jenkins/xgboost-win64/detail/PR-6019/6/pipeline

[2020-08-31T06:09:47.714Z] C:\tools\miniconda3\envs\win64_6ad3a277178e49eab07729d2452d1373\lib\site-packages\hypothesis\core.py:642: 
[2020-08-31T06:09:47.714Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[2020-08-31T06:09:47.714Z] 
[2020-08-31T06:09:47.714Z] self = <hypothesis.core.StateForActualGivenExecution object at 0x000002A00CD50F08>
[2020-08-31T06:09:47.714Z] message = "Hypothesis test_shap(self=<test_gpu_prediction.TestGPUPredict testMethod=test_shap>, num_rounds=141, dataset=sparse, ...el_tree': 10}, all_rows=True) produces unreliable results: Falsified on the first call but did not on a subsequent one"
[2020-08-31T06:09:47.714Z] 
[2020-08-31T06:09:47.714Z]     def __flaky(self, message):
[2020-08-31T06:09:47.714Z]         if len(self.falsifying_examples) <= 1:
[2020-08-31T06:09:47.714Z] >           raise Flaky(message)
[2020-08-31T06:09:47.714Z] E           hypothesis.errors.Flaky: Hypothesis test_shap(self=<test_gpu_prediction.TestGPUPredict testMethod=test_shap>, num_rounds=141, dataset=sparse, param={'max_depth': 10, 'max_leaves': 106, 'num_parallel_tree': 10}, all_rows=True) produces unreliable results: Falsified on the first call but did not on a subsequent one
[2020-08-31T06:09:47.714Z] 
[2020-08-31T06:09:47.714Z] C:\tools\miniconda3\envs\win64_6ad3a277178e49eab07729d2452d1373\lib\site-packages\hypothesis\core.py:847: Flaky
[2020-08-31T06:09:47.714Z] --------------------------------- Hypothesis ----------------------------------
[2020-08-31T06:09:47.714Z] Falsifying example: test_shap(
[2020-08-31T06:09:47.714Z]     self=<test_gpu_prediction.TestGPUPredict testMethod=test_shap>,
[2020-08-31T06:09:47.714Z]     num_rounds=141,
[2020-08-31T06:09:47.714Z]     dataset=sparse,
[2020-08-31T06:09:47.714Z]     param={'max_depth': 10, 'max_leaves': 106, 'num_parallel_tree': 10},
[2020-08-31T06:09:47.714Z]     all_rows=True,
[2020-08-31T06:09:47.714Z] )
[2020-08-31T06:09:47.714Z] Failed to reproduce exception. Expected: 
[2020-08-31T06:09:47.714Z] Traceback (most recent call last):
[2020-08-31T06:09:47.714Z]   File "C:\Users\Administrator\workspace\xgboost-win64_PR-6019\tests\python-gpu\test_gpu_prediction.py", line 219, in test_shap
[2020-08-31T06:09:47.714Z]     assert np.allclose(np.sum(shap, axis=len(shap.shape) - 1), margin, 1e-3, 1e-3)
[2020-08-31T06:09:47.714Z] AssertionError: assert False
[2020-08-31T06:09:47.714Z]  +  where False = <function allclose at 0x000002A07A04A948>(array([ 68.19425 , 233.31032 , 188.2829  , ...,  68.03375 ,  11.593885,\n        51.257465], dtype=float32), array([ 68.19389 , 233.31035 , 188.28214 , ...,  68.03336 ,  11.594096,\n        51.25665 ], dtype=float32), 0.001, 0.001)
[2020-08-31T06:09:47.714Z]  +    where <function allclose at 0x000002A07A04A948> = np.allclose
[2020-08-31T06:09:47.714Z]  +    and   array([ 68.19425 , 233.31032 , 188.2829  , ...,  68.03375 ,  11.593885,\n        51.257465], dtype=float32) = <function sum at 0x000002A07A031438>(array([[-1.3257432e-01, -4.1069903e+00,  1.6655694e-01, ...,\n         7.6787562e+00,  6.7797709e+00,  3.4963994e+00],\n...6780e-01,  3.2375448e+00, -4.1314743e-02, ...,\n         2.7299562e-01,  5.3053784e-01,  3.4963984e+00]], dtype=float32), axis=(2 - 1))
[2020-08-31T06:09:47.714Z]  +      where <function sum at 0x000002A07A031438> = np.sum
[2020-08-31T06:09:47.714Z]  +      and   2 = len((2000, 101))
[2020-08-31T06:09:47.714Z]  +        where (2000, 101) = array([[-1.3257432e-01, -4.1069903e+00,  1.6655694e-01, ...,\n         7.6787562e+00,  6.7797709e+00,  3.4963994e+00],\n...6780e-01,  3.2375448e+00, -4.1314743e-02, ...,\n         2.7299562e-01,  5.3053784e-01,  3.4963984e+00]], dtype=float32).shape

[hcho3]

Related: https://xgboost-ci.net/blue/organizations/jenkins/xgboost-win64/detail/PR-6088/1/pipeline

[2020-09-06T05:39:25.074Z] self = <hypothesis.core.StateForActualGivenExecution object at 0x000001ED02A61648>
[2020-09-06T05:39:25.074Z] message = "Hypothesis test_shap(self=<test_gpu_prediction.TestGPUPredict testMethod=test_shap>, num_rounds=117, dataset=sparse, ...el_tree': 10}, all_rows=True) produces unreliable results: Falsified on the first call but did not on a subsequent one"
[2020-09-06T05:39:25.074Z] 
[2020-09-06T05:39:25.074Z]     def __flaky(self, message):
[2020-09-06T05:39:25.074Z]         if len(self.falsifying_examples) <= 1:
[2020-09-06T05:39:25.074Z] >           raise Flaky(message)
[2020-09-06T05:39:25.074Z] E           hypothesis.errors.Flaky: Hypothesis test_shap(self=<test_gpu_prediction.TestGPUPredict testMethod=test_shap>, num_rounds=117, dataset=sparse, param={'max_depth': 10, 'max_leaves': 136, 'num_parallel_tree': 10}, all_rows=True) produces unreliable results: Falsified on the first call but did not on a subsequent one
[2020-09-06T05:39:25.074Z] 
[2020-09-06T05:39:25.074Z] C:\tools\miniconda3\envs\win64_53ff2bf064624d89b9d79f44530433ee\lib\site-packages\hypothesis\core.py:847: Flaky
[2020-09-06T05:39:25.074Z] --------------------------------- Hypothesis ----------------------------------
[2020-09-06T05:39:25.074Z] Falsifying example: test_shap(
[2020-09-06T05:39:25.074Z]     self=<test_gpu_prediction.TestGPUPredict testMethod=test_shap>,
[2020-09-06T05:39:25.074Z]     num_rounds=117,
[2020-09-06T05:39:25.074Z]     dataset=sparse,
[2020-09-06T05:39:25.074Z]     param={'max_depth': 10, 'max_leaves': 136, 'num_parallel_tree': 10},
[2020-09-06T05:39:25.074Z]     all_rows=True,
[2020-09-06T05:39:25.074Z] )
[2020-09-06T05:39:25.074Z] Failed to reproduce exception. Expected: 
[2020-09-06T05:39:25.074Z] Traceback (most recent call last):
[2020-09-06T05:39:25.074Z]   File "C:\Users\Administrator\workspace\xgboost-win64_PR-6088\tests\python-gpu\test_gpu_prediction.py", line 219, in test_shap
[2020-09-06T05:39:25.074Z]     assert np.allclose(np.sum(shap, axis=len(shap.shape) - 1), margin, 1e-3, 1e-3)
[2020-09-06T05:39:25.074Z] AssertionError: assert False
[2020-09-06T05:39:25.074Z]  +  where False = <function allclose at 0x000001ED7BFAEC18>(array([ 67.100845, 132.87697 , 189.37752 , ...,  67.79396 ,  11.182817,\n        49.87058 ], dtype=float32), array([ 67.10081 , 132.87706 , 189.37799 , ...,  67.79424 ,  11.182876,\n        49.869823], dtype=float32), 0.001, 0.001)
[2020-09-06T05:39:25.074Z]  +    where <function allclose at 0x000001ED7BFAEC18> = np.allclose
[2020-09-06T05:39:25.074Z]  +    and   array([ 67.100845, 132.87697 , 189.37752 , ...,  67.79396 ,  11.182817,\n        49.87058 ], dtype=float32) = <function sum at 0x000001ED7BF9B708>(array([[ 0.19836748, -5.121308  ,  0.19831742, ...,  0.73095405,\n         7.3354983 ,  3.8634229 ],\n       [-0.2283282... ],\n       [-0.0455513 ,  4.0112267 ,  1.7401932 , ...,  0.8874571 ,\n         1.9192814 ,  3.8634229 ]], dtype=float32), axis=(2 - 1))
[2020-09-06T05:39:25.074Z]  +      where <function sum at 0x000001ED7BF9B708> = np.sum
[2020-09-06T05:39:25.074Z]  +      and   2 = len((2000, 101))
[2020-09-06T05:39:25.074Z]  +        where (2000, 101) = array([[ 0.19836748, -5.121308  ,  0.19831742, ...,  0.73095405,\n         7.3354983 ,  3.8634229 ],\n       [-0.2283282... ],\n       [-0.0455513 ,  4.0112267 ,  1.7401932 , ...,  0.8874571 ,\n         1.9192814 ,  3.8634229 ]], dtype=float32).shape

[RAMitchell]

Thanks, looks like a problem specifically with the sparse case. Will see if I can find anything.

[trivialfis]

https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-6105/22/pipeline

[hcho3]

@RAMitchell A unit test for SHAP feature interaction is failing: https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-6202/1/pipeline/220

[hcho3]

Another failure: https://xgboost-ci.net/blue/organizations/jenkins/xgboost-win64/detail/PR-6216/1/pipeline

[2020-10-09T03:43:42.919Z] > assert np.allclose(np.sum(shap, axis=(len(shap.shape) - 1, len(shap.shape) - 2)), margin,

[2020-10-09T03:43:42.919Z] 1e-3, 1e-3)

[2020-10-09T03:43:42.919Z] E AssertionError: assert False

[2020-10-09T03:43:42.919Z] E + where False = <function allclose at 0x0000027D56FDE708>(array([22.34262 , 35.4537 , 6.800888, ..., 26.148102, 7.017563,\n 50.312687], dtype=float32), array([22.34258 , 35.453533 , 6.8009005, ..., 26.148138 , 7.017548 ,\n 50.31148 ], dtype=float32), 0.001, 0.001)

[2020-10-09T03:43:42.919Z] E + where <function allclose at 0x0000027D56FDE708> = np.allclose

[2020-10-09T03:43:42.919Z] E + and array([22.34262 , 35.4537 , 6.800888, ..., 26.148102, 7.017563,\n 50.312687], dtype=float32) = <function sum at 0x0000027D56FC21F8>(array([[[-1.7425357e-01, -2.4234520e-02, -6.3748877e-03, ...,\n 0.0000000e+00, -2.3086108e-03, 0.0000000e+00]...00e+00, 0.0000000e+00, 0.0000000e+00, ...,\n 0.0000000e+00, 0.0000000e+00, 3.1190395e+00]]], dtype=float32), axis=(2, 1))

[2020-10-09T03:43:42.919Z] E + where <function sum at 0x0000027D56FC21F8> = np.sum

[2020-10-09T03:43:42.919Z]

[2020-10-09T03:43:42.919Z] tests\python-gpu\test_gpu_prediction.py:223: AssertionError

[trivialfis]

@RAMitchell Can we re-enable sparse test and close this?

[RAMitchell]

I think it should be ok now.