Remove dependencies: StatsBase and Distributions

[joshday]

My PR to move params from Distributions to StatsBase now has 8 commits and 22 comments...

I think this is as good a time as any to visit the idea of going back to 0 dependencies, which was our original thought when we created LearnBase. We essentially only have StatsBase and Distributions in our require file for nobs and params/params!. Does anyone have a strong opinion on adding these ourselves and just not exporting them? Or other solutions?

[Evizero]

Well we need nobs defined here because MLDataPattern and MLDataUtils build on that fact heavily.

[joshday]

Revisiting this again. We were able to drop Distributions 🎉 thanks to params now living in StatsBase. Would you be okay with JuliaML packages that need params or nobs importing StatsBase directly? We pick up 4 dependencies for 2 names (not functionality), and that ratio kinda bugs me.

• StatsBase
  • DataStructures
  • SpecialFunctions
    • BinDeps

[Evizero]

for completeness, this is what we import and reexport from StatsBase

import StatsBase: nobs, fit, fit!, predict, params, params!

Well the unfortunate thing at the state that we are now (disregarding potential future reasons) is that MLDataPattern advertises that its only needed to extend the functions from LearnBase. (AFAIK its nobs and getobs) in order to add support for a custom datatype.

I guess it would be possible to say "extend LearnBase.getobs and StatsBase.nobs", but thats also not ideal.

I do see your point though.

[joshday]

We could drop fit for learn (even though I'm the one that added fit...), nobs and params could be imported by packages that need them, but predict is pretty essential. I guess it wouldn't be worth the effort to hack around a StatsBase dependency.

[oxinabox]

I am reopenning this for the modern world.
Can we think about this again?

I just want to overload default_obsdim for my new container type.
And I don't want to end up with StatsBase at all really,
in my dependency tree.

[joshday]

It's difficult because StatsBase has claimed some key functions like predict and nobs that LearnBase really can't go without. I think StatsBase needs a "StatsCore" package that just provides the namespace (empty function definitions, just like LearnBase), but I'm not sure how well received that would be.

[oxinabox]

I agree,
but until we live in that world,
I don't see it as that bad for a package wanting to overload fit to have to import StatsBase even if it is already importing LearnBase.

Infact, the argument could be made that it is better practice to only overload things when importing them directly from their original namespace,
not from someone reexporting them

[joshday]

That's fair. I'm on board with removing StatsBase here

[skanskan]

What's the difference between StatsBase, Distributions, StatsFuns and StatsModels?

[oxinabox]

StatsBase is core functions for working with statistics, like calculating the weighted std-dev.
Distributions is for working with probability distributions, sampling and logpdf etc.
StatsFuns is like SpecialFunctions.jl but for statistics, so small functions like logit.
StatsModels is a DSL for decribing relationships one one want to model, mostly used for linear models and that kinda thing.

[juliohm]

Reviving this thread again... (third time I guess?)

I am planning to revamp the LearnBase.jl package and upgrade it into a more general interface for statistical learning. In this plan the names fit and predict seem too narrow, and I agree with @joshday that we could perhaps call them learn and perform or something along these lines to avoid clashes with StatsBase.jl and Statistics stdlib?

I would like to ask you where else the LearnBase.jl package is being used inside the JuliaML organization. To what extent do we need to be backwards compatible? I see that the project doesn't have a Project.toml yet, and that is a good sign that it is not being actively used elsewhere.

I will open a separate issue to share a proposal regarding an updated LearnBase.jl API.

[joshday]

I see that the project doesn't have a Project.toml yet, and that is a good sign that it is not being actively used elsewhere.

LearnBase hasn't required much updating since really all it does is claim names (although I'm pretty surprised there's no Project.toml!). I use it in several packages (OnlineStats, SparseRegression), but I started removing LearnBase from both within the last month.

[juliohm]

@joshday why is it that you are removing it from your packages? Do you feel that it is not suiting your needs?