Skip to content

[SPARK-3162][MLlib] Local Tree Training Pt 1: Refactor RandomForest.scala into utility classes #19758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from

Conversation

smurching
Copy link
Contributor

@smurching smurching commented Nov 15, 2017

What changes were proposed in this pull request?

Breaks up #19433 to help unblock #19666; after this PR is merged, #19666 can be merged.

This PR contains the changes made to migrate functionality from RandomForest.scala into the following utility classes:

  • AggUpdateUtils
  • ImpurityUtils
  • SplitUtils

The PR also adds tests for split selection logic in TreeSplitUtilsSuite.

A follow-up PR will include the other changes from #19433:

  • Local decision tree data structures & tests
  • Local tree training logic & tests

How was this patch tested?

Adds unit tests for split selection logic in TreeSplitUtilsSuite

@SparkQA
Copy link

SparkQA commented Nov 15, 2017

Test build #83910 has finished for PR 19758 at commit b93f9f3.

  • This patch fails to generate documentation.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 15, 2017

Test build #83911 has finished for PR 19758 at commit b4a5f3b.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 15, 2017

Test build #83914 has finished for PR 19758 at commit b6291e1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Contributor

@WeichenXu123 WeichenXu123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work. I mainly reviewed the new added testsuite part.

// label: 2 --> values: 2
// Expected split: feature value 1 on the left, values (0, 2) on the right
val values = Array(1, 1, 0, 2, 2)
val featureArity = values.max + 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to make the test more strict, can you increase the featureArity, numExamples and numClasses ? e.g., featureArity = 6 and numExamples = 10 and numClasses = 5

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WeichenXu123 thanks for the feedback! Definitely agree that the test is a little weak right now.

IMO it's mainly weak due to the low feature arity (there only three possible splits, so the right one could be picked by chance). I think increasing the number of classes/examples substantially might make the test harder to reason about, but not opposed to that either - let me know what you think.

What about something like:

    val values = Array(0, 1, 2, 3, 2, 2, 4)
    val labels = Array(0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 2.0)
    // label: 0 --> values: 0, 1
    // label: 1 --> values: 2, 3
    // label: 2 --> values: 2, 2, 4
    // Expected split: feature values (0, 1) on the left, values (2, 3, 4) on the right

This way we still test multiclass classification & test the split-selection logic more rigorously.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is OK. thanks!

@SparkQA
Copy link

SparkQA commented Dec 1, 2017

Test build #84379 has finished for PR 19758 at commit 5bcccda.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@felixcheung
Copy link
Member

ping?
I'm mostly interested in SPARK-3162

@wenbochang
Copy link

Any updates? this PR seems to address critical issue: https://issues.apache.org/jira/browse/SPARK-3162

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants