Ipfs daemon hangs when MFS root is not available locally

[nlko]

Version information:

go-ipfs version: 0.4.23-
Repo version: 7
System version: amd64/linux
Golang version: go1.13.7

Description:

Hello,

When I start the daemon it now hangs after displaying this (I tried this several times and also restarted the computer):

$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.4.23-
Repo version: 7
System version: amd64/linux
Golang version: go1.13.7

The execution hangs there forever. I tried some ipfs commands that failed returning Error: cannot acquire lock: Lock FcntlFlock of ~/.ipfs/repo.lock failed: resource temporarily unavailable

After an hour, if I press Ctrl+C it stops and displays :

22:46:53.828 ERROR   cmd/ipfs: error from node construction:  could not build arguments for function "reflect".makeFuncStub (/usr/lib/go/src/reflect/asm_amd64.s:12): failed to build *mfs.Root: function "github.com/ipfs/go-ipfs/core/node".Files (pkg/mod/github.com/ipfs/go-ipfs@v0.4.23/core/node/core.go:74) returned a non-nil error: error loading filesroot from DAG: failed to get block for QmeXwawbVyYrFh6ZjjjbBzzY8gZAcnt7UfLyY2ZhG6MkYQ: context canceled daemon.go:337

Error: could not build arguments for function "reflect".makeFuncStub (/usr/lib/go/src/reflect/asm_amd64.s:12): failed to build *mfs.Root: function "github.com/ipfs/go-ipfs/core/node".Files (pkg/mod/github.com/ipfs/go-ipfs@v0.4.23/core/node/core.go:74) returned a non-nil error: error loading filesroot from DAG: failed to get block for QmeXwawbVyYrFh6ZjjjbBzzY8gZAcnt7UfLyY2ZhG6MkYQ: context canceled

Here is how I managed to reach this point:

• Everything was working fine, the daemon was running for more than 24h and I was doing some normal mfs operation like everyday using ipfs-desktop;
• The disk where the repo was located ran out of space for a couple of minutes.
• The reason why the disk ran out of space has nothing to do with ipfs : I was using dd (for a big copy) In the meantime I was watching a video from the mfs.
• From that point I wasn't able to perform ipfs add comands. It failed stating that the disk was full according to a .LOG file. (which is wasn't anymore, more than 180G available).
• I tried to restart the daemon and the daemon hung like I said previously.

Any idea how I can at least recover the content of the repo ?

[hsanjuan]

If you don't start your daemon, and run ipfs files ls /, what happens?

[nlko]

I prior checked with that the daemon isn't running with ps and grep.

$ ipfs files ls /
Error: could not build arguments for function "reflect".makeFuncStub (/usr/lib/go/src/reflect/asm_amd64.s:12): failed to build *mfs.Root: function "github.com/ipfs/go-ipfs/core/node".Files (pkg/mod/github.com/ipfs/go-ipfs@v0.4.23/core/node/core.go:74) returned a non-nil error: error loading filesroot from DAG: merkledag: not found

[hsanjuan]

@Stebalien is there a way to replace the MFS root? I have no knowledge of it, but I could hack something together. It's not the first time we see this.

[Stebalien]

We should refuse to start, but we shouldn't hang. This error indicates that the repo has been corrupted somehow (or maybe wasn't initialized properly?). We should:

1. Track down why this might be happening.
2. Fetch the MFS root with an "offline" DAG service on start, instead of trying to fetch it from the network.

[nlko]

It complains the QmeXwawbVyYrFh6ZjjjbBzzY8gZAcnt7UfLyY2ZhG6MkYQ block is missing. If I understand well it is looking for a file name L6/CIQPBH3WIZMW4EJVPZ4DDBCASE7JMK6F4RPT3BAYO2TEQPBIO4XBL6I.data which is missing in the filesystem.

Is it possible that somehow it has been able to remember what was the last hash of the root MFS even if it wasn't able to properly store the corresponding block ? (like I said I was running out of disk for some time just before I started to have trouble with ipfs.)

[Stebalien]

Do you enable garbage collection (ipfs daemon --enable-gc or ipfs repo gc)? If so, I believe we have some races where the garbage collector may remove the root if run at the same time.

[nlko]

When it complained that the disk was full (during an ipfs add) I manually ran the gc.
Then I restarted the daemon because the ipfs add was still failing (event if at that time I was capable of accessing mfs files).
But restarting the daemon hung.

When I run the gc it removed some blocks. Most of my data (at least 60 GB out of 70+GB) was in mfs. The size of the repo folder is still more than 70GB.

So my data is still there but since I lost the MFS root block I'm unable to access it.

It's not the first time I end up with a problem with the mfs. During the last month I had several times mfs corruptions (https://discuss.ipfs.io/t/update-root-cid). I believe because I was using the mfs write.

The root cid is a quite sensible peace of information. Maybe keeping track of some older root cid would allow to do some roll-back latter on (ideally they could be stored outside in a human readable rotating file and the gc would respect them).

I consider myself as new to IPFS but the way I see it is that mfs is a layer on top of IPFS. Maybe the lack of mfs should not prevent the ipfs daemon to start (in a kind of reduced mode) and perform basic add/pin operations. It's also maybe a feature that is not useful in some context like on a ipfs-cluster server.

[nlko]

I wrote a piece of code that looks for the biggest tree in the repo.

With it I finally found a DAG tree with 75GB.

The content of the DAG contains the links (folders and files) I had at the root mfs.

Now that I have a CID that looks valid, how can I change the mfs root CID that the ipfs daemon should look for ?

[hsanjuan]

@nlko , can you try https://github.com/hsanjuan/mfs-replace-root ? Hopefully this allows you to move on with your stuff. Figuring out if this was due to a GC-related race or simply a bad error when trying to write on a full disk may take a while, and I understand resetting your repo is not an option.

• Track down why this might be happening.

• Fetch the MFS root with an "offline" DAG service on start, instead of trying to fetch it from the network.

It is important to do this but from the user point of view, it would be good to include a way to forcefully set the mfs root whenever something like this arises (as part of ipfs files commands), imho.