Skip to content

Commit 74bf836

Browse files
StebalienStebalien
authored
Merge pull request #5864 from nitishm/docs/mfs/add-code-flow
docs/code-flow : Add code flow documentation for add cmd.
2 parents 532c9bd + 269550b commit 74bf836

File tree

2 files changed

+103
-0
lines changed

2 files changed

+103
-0
lines changed

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -403,6 +403,7 @@ Some places to get you started on the codebase:
403403
- libp2p: https://github.com/libp2p/go-libp2p
404404
- DHT: https://github.com/libp2p/go-libp2p-kad-dht
405405
- PubSub: https://github.com/libp2p/go-libp2p-pubsub
406+
- [IPFS : The `Add` command demystified](https://github.com/ipfs/go-ipfs/tree/master/docs/add-code-flow.md)
406407

407408
### CLI, HTTP-API, Architecture Diagram
408409

‎docs/add-code-flow.md

+102
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# IPFS : The `Add` command demystified
2+
3+
The goal of this document is to capture the code flow for adding a file (see the `coreapi` package) using the IPFS CLI, in the process exploring some datastructures and packages like `ipld.Node` (aka `dagnode`), `FSNode`, `MFS`, etc.
4+
5+
## Concepts
6+
- [Files](https://github.com/ipfs/docs/issues/133)
7+
8+
---
9+
10+
**Try this yourself**
11+
>
12+
> ```
13+
> # Convert a file to the IPFS format.
14+
> echo "Hello World" > new-file
15+
> ipfs add new-file
16+
> added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u new-file
17+
> 12 B / 12 B [=========================================================] 100.00%
18+
>
19+
> # Add a file to the MFS.
20+
> NEW_FILE_HASH=$(ipfs add new-file -Q)
21+
> ipfs files cp /ipfs/$NEW_FILE_HASH /new-file
22+
>
23+
> # Get information from the file in MFS.
24+
> ipfs files stat /new-file
25+
> # QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy47CnJDgvs8u
26+
> # Size: 12
27+
> # CumulativeSize: 20
28+
> # ChildBlocks: 0
29+
> # Type: file
30+
>
31+
> # Retrieve the contents.
32+
> ipfs files read /new-file
33+
> # Hello World
34+
> ```
35+
36+
## Code Flow
37+
38+
**[`UnixfsAPI.Add()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreapi/unixfs.go#L31)** - *Entrypoint into the `Unixfs` package*
39+
40+
The `UnixfsAPI.Add()` acts on the input data or files, to build a _merkledag_ node (in essence it is the entire tree represented by the root node) and adds it to the _blockstore_.
41+
Within the function, a new `Adder` is created with the configured `Blockstore` and __DAG service__`.
42+
43+
- **[`adder.AddAllAndPin(files)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L403)** - *Entrypoint to the `Add` logic*
44+
encapsulates a lot of the underlying functionality that will be investigated in the following sections.
45+
46+
Our focus will be on the simplest case, a single file, handled by `Adder.addFile(file files.File)`.
47+
48+
- **[`adder.addFile(file files.File)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L450)** - *Create the _DAG_ and add to `MFS`*
49+
50+
The `addFile(file)` method takes the data and converts it into a __DAG__ tree and adds the root of the tree into the `MFS`.
51+
52+
https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L508-L521
53+
54+
There are two main methods to focus on -
55+
56+
1. **[`adder.add(io.Reader)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L115)** - *Create and return the **root** __DAG__ node*
57+
58+
This method converts the input data (`io.Reader`) to a __DAG__ tree, by splitting the data into _chunks_ using the `Chunker` and organizing them in to a __DAG__ (with a *trickle* or *balanced* layout. See [balanced](https://github.com/ipfs/go-unixfs/blob/6b769632e7eb8fe8f302e3f96bf5569232e7a3ee/importer/balanced/builder.go) for more info).
59+
60+
The method returns the **root** `ipld.Node` of the __DAG__.
61+
62+
2. **[`adder.addNode(ipld.Node, path)`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L366)** - *Add **root** __DAG__ node to the `MFS`*
63+
64+
Now that we have the **root** node of the `DAG`, this needs to be added to the `MFS` file system.
65+
Fetch (or create, if doesn't already exist) the `MFS` **root** using `mfsRoot()`.
66+
67+
> NOTE: The `MFS` **root** is an ephemeral root, created and destroyed solely for the `add` functionality.
68+
69+
Assuming the directory already exists in the MFS file system, (if it doesn't exist it will be created using `mfs.Mkdir()`), the **root** __DAG__ node is added to the `MFS` File system using the `mfs.PutNode()` function.
70+
71+
- **[MFS] [`PutNode(mfs.Root, path, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/ops.go#L86)** - *Insert node at path into given `MFS`*
72+
73+
The `path` param is used to determine the `MFS Directory`, which is first looked up in the `MFS` using `lookupDir()` function. This is followed by adding the **root** __DAG__ node (`ipld.Node`) in to this `Directory` using `directory.AddChild()` method.
74+
75+
- **[MFS] Add Child To `UnixFS`**
76+
- **[`directory.AddChild(filename, ipld.Node)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L350)** - *Add **root** __DAG__ node under this directory*
77+
78+
Within this method the node is added to the `Directory`'s __DAG service__ using the `dserv.Add()` method, followed by adding the **root** __DAG__ node with the given name, in the `directory.addUnixFSChild(directory.child{name, ipld.Node})` method.
79+
80+
- **[MFS] [`directory.addUnixFSChild(child)`](https://github.com/ipfs/go-mfs/blob/v0.1.18/dir.go#L375)** - *Add child to inner UnixFS Directory*
81+
82+
The node is then added as a child to the inner `UnixFS` directory using the `(BasicDirectory).AddChild()` method.
83+
84+
> NOTE: This is not to be confused with the `directory.AddChild(filename, ipld.Node)`, as this operates on the `UnixFS` `BasicDirectory` object.
85+
86+
- **[UnixFS] [`(BasicDirectory).AddChild(ctx, name, ipld.Node)`](https://github.com/ipfs/go-unixfs/blob/v1.1.16/io/directory.go#L137)** - *Add child to `BasicDirectory`*
87+
88+
> IMPORTANT: It should be noted that the `BasicDirectory` object uses the `ProtoNode` type object which is an implementation of the `ipld.Node` interface, seen and used throughout this document. Ideally the `ipld.Node` should always be used, unless we need access to specific functions from `ProtoNode` (like `Copy()`) that are not available in the interface.
89+
90+
This method first attempts to remove any old links (`ProtoNode.RemoveNodeLink(name)`) to the `ProtoNode` prior to adding a link to the newly added `ipld.Node`, using `ProtoNode.AddNodeLink(name, ipld.Node)`.
91+
92+
- **[Merkledag] [`AddNodeLink()`](https://github.com/ipfs/go-merkledag/blob/v1.1.15/node.go#L99)**
93+
94+
The `AddNodeLink()` method is where an `ipld.Link` is created with the `ipld.Node`'s `CID` and size in the `ipld.MakeLink(ipld.Node)` method, and is then appended to the `ProtoNode`'s links in the `ProtoNode.AddRawLink(name)` method.
95+
96+
- **[`adder.Finalize()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L200)** - *Fetch and return the __DAG__ **root** from the `MFS` and `UnixFS` directory*
97+
98+
The `Finalize` method returns the `ipld.Node` from the `UnixFS` `Directory`.
99+
100+
- **[`adder.PinRoot()`](https://github.com/ipfs/go-ipfs/blob/v0.4.18/core/coreunix/add.go#L171)** - *Pin all files under the `MFS` **root***
101+
102+
The whole process ends with `PinRoot` recursively pinning all the files under the `MFS` **root**

0 commit comments

Comments
 (0)