[BOUNTY] Implement UnixFSv1.5 in go-ipfs

[Stebalien]

This issue has a bounty!

Successfully closing this issue by producing a production-ready, mergeable PR can earn you not only the undying love of the IPFS community — it can net you a financial reward. See the current list of bounty issues and their values here.

The need in brief

Add UnisFSv1.5 metadata support per the spec https://github.com/ipfs/specs/blob/master/UNIXFS.md#data-format, which includes: specifying metadata, respecting existing metadata, and displaying metadata.

This aims to support large data import use cases (ex package managers), who need to retain file metadata, particularly last modified time (mtime), in order to selectively sync only data that has changed. Up until now if you wanted to host a large data set on IPFS, like a package manager's repository, it would be difficult to update - but with file mtime you can only reimport the changed files, making things much more efficient.

Deliverable

• Implement the new mode & mtime fields and obey the new rules around default modes & mtimes as per the metadata spec.
• Plumb this metadata all the way through from ipfs add on one IPFS node to ipfs get on another.
• An end-to-end sharness test that validates that this metadata is preserved.

Requirements

• This feature must be interoperable with js-ipfs's implementation of the UnixFSv1.5 spec.
• The behavior of ipfs add must not change except when the user has requested that metadata be preserved via the appropriate flags (see the implementation details below). Specifically, the final hash produced by ipfs add must not change.
• The final hash of files added with this feature enabled must match the hash produced by javascript.

Guidelines

• Please use the IPFS Go Contributing Guidelines as your north star — adherence to these guidelines are crucial when it comes to awarding a potential bounty!
• Please DO NOT bundle a general code refactor with the completion of this bounty!

[momack2]

For context, here is the js-ipfs unixfsv1.5 implementation done by @achingbrain: ipfs/js-ipfs#2621, and the blog post describing the shipped work: https://blog.ipfs.io/2020-02-13-js-ipfs-0-41/#unixfs-v1-5

If you want to start working on this, ping @Stebalien for a quick walkthrough of where to start and what already exists to build on top!

[Stebalien]

Implementation Notes

Implementing this feature will touch several sub-components of go-ipfs:

• The go-ipfs-files library will need to be updated to:
  • Support modification times and modes.
    • When reading from and writing to disk.
    • When reading from the web (WebFile).
    • When sending/receiving over HTTP using the "multipart" file types.
  • Expose modification times and modes via the Node interface.
• Implementations of the Node interface from go-ipfs-files will need to be updated:
  • github.com/ipfs/go-unixfs/file/unixfile.go
  • github.com/ipfs/go-ipfs-http-client/apifile.go
• The protobuf definitions in github.com/ipfs/go-unixfs/pb/unixfs.proto will need to be updated and regenerated according to the spec.
• The go-unixfs (de)serialization logic in go-unixfs will also need to be updated to (de)serialize modes/mtimes.
• The go-ipfs file adder (github.com/ipfs/go-ipfs/core/coreunix/add.go) will need to be extended to handle modification times.
• The go-ipfs "add" command (ipfs add) will need two new flags (defaulting to false):
  • --preserve-mtime -- record the file modification time when adding.
  • --preserve-mode -- record the file mode when adding.

The new go-ipfs-files Node interface should be:

// Node is a common interface for files, directories and other special files
type Node interface {
	io.Closer

	// Mode returns the file's Mode
	Mode() os.FileMode

	// ModTime returns the files last modification time.
	//
	// If the last modification time is unknown/unspecified, ok is false and
	// the mtime is the Unix epoch (default timestamp).
	ModTime() (mtime time.Time, ok bool)

	// Size returns size of this file (if this file is a directory, total size of
	// all files stored in the tree should be returned). Some implementations may
	// choose not to implement this
	Size() (int64, error)
}

[kalmi]

@Stebalien Hi, @momack2 said to ping you for a walkthrough if one would like to start working on this, but I think it is safe to assume that your last comment is that walkthrough, and I will just go ahead. Thanks!

[achingbrain]

ipfs add also needs the following new options:

• --mode - specify a file mode
• --mtime - specify the seconds component of the mtime
• --mtime-nsecs - specify the nanosecond component of the mtime

And there are two new mfs commands:

• ipfs files touch [path] (update mtime to the current time)
  • --mtime - optionally specify the seconds component of the mtime to update to
  • --mtime-nsecs - optionally specify the nanosecond component of the mtime to update to
• ipfs files chmod [mode] [path]