Reduce dependency footprint

[gballet]

This issue is created as a result of an ongoing conversation with @raulk over the switch from devp2p to libp2p in Ethereum; I have made two attempts at making a libp2p version

• A first attempt that only covers one protocol (Whisper) Whisper v6 libp2p v2 ethereum/go-ethereum#16419, As you can see it has 590+ files, most of which are dependencies.
• Another attempt that is still being developed is based on blankhost; It's already breaking the 350+ file mark.

Since we vendor most of our code, it makes a lot of members of our core-team un-happy. Here's a non-exhaustive list of pain points that apparently is going to be useful for a design meeting that you will be having soon:

• One of them is that it makes a nightmare of a PR review
• It brings a lot of duplicates with packages that we already vendored or rely on in a different version.
• Modularity is great if it possible to only pick the modules that one needs. We don't need most of the things that we pulled.

Hope that is helps with the discussion, if you need more details let me know.

[ghost]

Yes, this feedback helps a lot. Thanks for opening the issue. And yes, we are planning to discuss this next week.

/ cc @whyrusleeping @bigs

[mhluongo]

Ditto from the Keep team- @rargulati has been grappling with it this week

[whyrusleeping]

@gballet @mhluongo make sure to only vendor .go files (no markdown, yaml, _test.go, etc). If I rememeber correctly, this is how i've seen a few different vendoring tools do it. They basically do an import of the code via the go importer and pull in only those files. I suspect that would help significantly.

[whyrusleeping]

Modularity is great if it possible to only pick the modules that one needs. We don't need most of the things that we pulled.

Let's figure out where things are getting pulled in unwanted, and modularize to avoid that.

[ghost]

Raw notes from a discussion we had about this problem tonight: libp2p/notes#4

No specific conclusions were reached, but the information may be helpful for context.

[gballet]

Understood, indeed we could remove around 150 files by just deleting all LICENSE and README.md ! This is not going to be enough, but it's sure to help.

[Stebalien]

FYI, you can use:

go list -f '{{.Deps}}'

To list all used packages and then use copy only files ending in *.go. However, you'll need to run this once per GOOS/GOARCH pair.

[aschmahmann]

@raulk any thoughts here or status updates?

[raulk]

We did one consolidation a few months ago, when we introduced go-libp2p-core. And the plan is to do another consolidation by merging go-libp2p-core into go-libp2p, along with default implementations of replaceable components like the peerstore, connmgr, network, etc. Potentially even hosting transports, secure channels, and muxers (pluggable components) that do not introduce much/no dependency footprint (e.g. tcp, noise, tls, mplex). Things like WebRTC, WS, Yamux, etc. would stay in their separate repos. That’s the best marriage I can think of between modularity/extensibility and codebase simplicity.

[marten-seemann]

This should have been resolved by our more recent rounds of repo consolidation. go-libp2p essentially is a mono-repo now.