Discussion of go-libp2p file bloat, vendoring issues

[ghost]

(Raw notes from discussion 8-Oct-2018 @lgierth @raulk @mgoelzer @Kubuxu @vyzo +add yourself)

History

• go-libp2p extracted from IPFS initially
  • more can be extracted (the host, identify, tests)
  • bitswap relies on go-libp2p package which pulls in everything
  • raulk: godepth and gx --deps or (gx --tree)

What about pulling interfaces out into an interfaces package?

• Arguments against: (1) just a huge bunch of work (2) interfaces will change a lot (Kuba, but Lars disagrees)
  • Explanation of (2): we would have to update the hashes
• Raul: but this is weird in any language? will feel unfamiliar?
• Each set of repos (like all muxers) has a go mod dedicated to it
  • Them implementers can pull in what they need
  • Lars: ok, say I'm going to use only yamux, will it pull in all the other muxers?
    • Kuba: no, bc a module is defined by a text file, so it would pull in only yamux

What about go modules?

• We're not dumping gx anytime soon
• gx could live on as a lock file, but go modules could be used to modularize the library
• go mod stable is 2 go versions away (~9 months from now)

Mono repo - would it help downstreams?

Could Ethereum Whisper exclude all non-dotgo files?

• Yes, because we are building with go build so compile process cannot pull in anything that is not a .go
• Could also exclude all *_test.go

Competing libraries on HN:

• (forget name) - a one-off p2p library with no upgradability, muxing
• need to convince community that the modularity we've created is worth the bloat

What about just having a go-libp2p-interface package?

Central location for how to use libp2p for different use case

• Problem: there is no central place that explains "what does libp2p provide for me?", "what repos do I need to pull in to get just {transport, identity provider}

What is eth's goal?

• Whisper v6
• Two versions:
  • devp2p (default)
  • libp2p (behind an experimenrtal flag)
    • But just this tiny test option is forcing them to pull in 590 files!

Short-term actions we could take

• We don't want to force them to change how they do their build (ie, force them to remove non-.go files)
• Fix this: root go-libp2p package still has dependencies in there
• Going through dependencies are completely unusused
  • Which ones are that use only a tiny fraction of a big library (could just copy the code)
• Putting interfaces into one place
• go modules - figure out how this would look with go modules

[lanzafame]

I will quickly add a few points I raised with @raulk over lunch in Glasgow.

What about just having a go-libp2p-interface package?

Technically, this already exists, in part, in the specs repo, they obviously need further detail and I think the should fill the role of the place to go to figure out what libp2p module to use for different use cases. I believe that placing the concrete go definition of those interfaces in a single repo on serves to separate them further from the code that is the reference implementation for that interface.

At the end of the day, eth maybe vendoring a lot of files but that is the trade-off for the modularity of libp2p which as another user of libp2p (ipfs-cluster), it is super useful knowing what is being used by each package and only having to bring in what is required.