Skip to content

docs: add snapshot instructions #224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 29, 2022
Merged

docs: add snapshot instructions #224

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
df7ffc2
docs: add snapshot instructions
Apr 28, 2022
3fb56ce
docs: add snapshot instructions
Apr 28, 2022
df97e17
fix: use x-goog-hash rather than etag
Apr 28, 2022
b46ba52
fix: specify 0.10.3
Apr 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/docs/joining-a-testnet.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ curl https://rpc-capricorn.fetch.ai:443 | jq '.result.genesis' > ~/.fetchd/confi
fetchd start --p2p.seeds=fec822ecf6e503a694a709ce663fd0c6da5fda3e@connect-capricorn.fetch.ai:36956
```

Your local node will then start to synchronise itself with the network, replaying all blocks and transactions up to the current block. Depending on the age of the network and your hard disk speed, this could take a while.
Your local node will then start to synchronise itself with the network, replaying all blocks and transactions up to the current block. Depending on the age of the network and your hard disk speed, this could take a while. Consider using [chain snapshots](../snapshots/) to speed up this process.

To know when your node as finished syncing, you can query it's status from its RPC API:

Expand Down
33 changes: 18 additions & 15 deletions docs/docs/networks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ The chain identifier of our production network is fetchhub-4.
| REST Endpoint | <https://rest-fetchhub.fetch.ai:443> |
| Block Explorer | [https://explore-fetchhub.fetch.ai](https://explore-fetchhub.fetch.ai) |
| Token Faucet | N/A |
| Genesis | `curl https://raw.githubusercontent.com/fetchai/genesis-fetchhub/fetchhub-4/fetchhub-4/data/genesis_migrated_5300200.json --output ~/.fetchd/config/genesis.json` |
| Genesis | `curl https://raw.githubusercontent.com/fetchai/genesis-fetchhub/fetchhub-4/fetchhub-4/data/genesis_migrated_5300200.json --output ~/.fetchd/config/genesis.json` |
| Seed Node(s) | 17693da418c15c95d629994a320e2c4f51a8069b@connect-fetchhub.fetch.ai:36456,a575c681c2861fe945f77cb3aba0357da294f1f2@connect-fetchhub.fetch.ai:36457,d7cda986c9f59ab9e05058a803c3d0300d15d8da@connect-fetchhub.fetch.ai:36458 |
| Snapshots | <https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-pruned.tgz> <br /> <https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-full.tgz> <br /> <https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-archive.tgz> |

## Mainnet Archives

Expand Down Expand Up @@ -99,17 +100,19 @@ This network is running the same software as our mainnet (`fetchhub-3`), and is

This network is used for testing the future upgrade to mainnet.

| Parameter | Value |
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Chain ID | dorado-1 |
| Denomination | atestfet |
| Decimals | 18 (1testfet = 1000000000000000000atestfet) |
| Min Gas Prices | 1000000000atestfet |
| Version | [v0.10.x (fetchd >= v0.10.x)](https://github.com/fetchai/fetchd/releases/tag/v0.10.0) |
| RPC Endpoint | <https://rpc-dorado.fetch.ai:443> |
| GRPC Endpoint | <https://grpc-dorado.fetch.ai:443> |
| REST Endpoint | <https://rest-dorado.fetch.ai:443> |
| Block Explorer | [https://explore-dorado.fetch.ai/](https://explore-dorado.fetch.ai/) |
| Ledger Explorer | [https://browse-dorado.fetch.ai/](https://browse-dorado.fetch.ai/) |
| Token Faucet | Use block explorer |
| Seed Node(s) | [email protected]:36556,[email protected]:36557,[email protected]:36558 |
| Parameter | Value |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Chain ID | dorado-1 |
| Denomination | atestfet |
| Decimals | 18 (1testfet = 1000000000000000000atestfet) |
| Min Gas Prices | 1000000000atestfet |
| Version | [v0.10.x (fetchd >= v0.10.3)](https://github.com/fetchai/fetchd/releases/tag/v0.10.0) |
| RPC Endpoint | <https://rpc-dorado.fetch.ai:443> |
| GRPC Endpoint | <https://grpc-dorado.fetch.ai:443> |
| REST Endpoint | <https://rest-dorado.fetch.ai:443> |
| Block Explorer | [https://explore-dorado.fetch.ai/](https://explore-dorado.fetch.ai/) |
| Ledger Explorer | [https://browse-dorado.fetch.ai/](https://browse-dorado.fetch.ai/) |
| Token Faucet | Use block explorer |
| Genesis | `curl https://storage.googleapis.com/fetch-ai-testnet-genesis/genesis-dorado-827201.json --output ~/.fetchd/config/genesis.json` |
| Seed Node(s) | [email protected]:36556,[email protected]:36557,[email protected]:36558 |
| Snapshots | <https://storage.googleapis.com/fetch-ai-testnet-snapshots/dorado-pruned.tgz> <br /> <https://storage.googleapis.com/fetch-ai-testnet-snapshots/dorado-full.tgz> <br /> <https://storage.googleapis.com/fetch-ai-testnet-snapshots/dorado-archive.tgz> |
52 changes: 52 additions & 0 deletions docs/docs/snapshots.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Chain State Snapshots

As blockchains get longer, the process of syncing from the genesis block begins to take many hours, or even days to complete.
In circumstances where a faster sync is required, various snapshots of the fetchd chain state data are available for download, to more quickly bootstrap a node.

Snapshots are available for both mainnet and the most recent testnet. The URLs can be obtained from the [network page](../networks/).
We aim to update snapshots on a daily basis.

The example below uses the pruned mainnet snapshot, but can be adapted as required for full or archive nodes.

## Using a snapshot

### Stop your node

If you are already running fetchd, it is important that you stop it before proceeding. Instructions for this are highly installation dependent and beyond the scope of this document, but could be as simple as a Ctrl-C.
If you have not already initialised your node, follow the instructions for [joining a testnet](../joining-a-testnet/) (modifying for mainnet as appropriate), then return to this page before starting fetchd.

### Reset your node

WARNING: This will irreversibly erase your node's state database. Ensure you take whatever backups you deem appropriate before proceeding.

If using fetchd <= 0.10.3
`fetchd unsafe-reset-all`

If using fetchd >= 0.10.4
`fetchd tendermint reset-state`

### Download and install the snapshot

Many options here! The example below assumes a bash-like environment, uses a single connection for downloading, confirms the md5sum of the downloaded data against that of the original, and does not land the original compressed data to disk. This is a good starting point, but depending on your local environment you may wish to make adaptations that eg sacrifice disk space and extra md5sum complexity for the benefit of parallel downloads with aria2. Entirely up to you!

```bash
# (optional) show the timestamp of the latest available snapshot
echo "Latest available snapshot timestamp : $(curl -s -I https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-pruned.tgz | grep last-modified | cut -f3- -d' ')"

# download, decompress and extract state database
curl -v https://storage.googleapis.com/fetch-ai-mainnet-snapshots/fetchhub-4-pruned.tgz -o- 2>headers.out | tee >(md5sum > md5sum.out) | gunzip -c | tar -xvf - --directory=~/.fetchd

# (optional, but recommended) compare source md5 checksum provided in the headers by google, with the one calculated locally
[[ $(awk -F\" '/etag:/{ print $2 }' headers.out) == $(awk '{ print $1 }' md5sum.out) ]] && echo "OK - md5sum match" || echo "ERROR - md5sum MISMATCH"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's some cool stuff here!

To double-check, is using the etag fine here? Seeing on https://cloud.google.com/storage/docs/hashes-etags#xml-api it might not guarantee to be the md5 - maybe x-goog-hash to be preferred (they actually have the same value when I tested, but not sure it's meant to stay like this)

Also wondered if md5 was enough here given how easy is to tamper with it nowadays. But I guess it's ok, the consensus is supposed to deal with the extra validation that this state is pristine

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well spotted, thanks! I will fix...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for md5 collisions... something they do (in a slightly different context) on Osmosis is to publish their hashes on the blockchain itself. I'll have a think about whether I can push a memo with the (assumed immutable) GCS x-goog-generation version number vs md5sum, that should give a very good confidence that a downloaded snapshot was created by my k8s pod, even in the scenario where an adversary has write access to our bucket.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://fetchai.atlassian.net/browse/ECO-1297 created


# (optional) show the creation date of the downloaded snapshot
echo "Downloaded snapshot timestamp: $(grep last-modified headers.out | cut -f3- -d' ')"
```

### Restart your node

Again, this entirely depends on your local installation, but a simple example for mainnet might be...

```bash
fetchd start --p2p.seeds 17693da418c15c95d629994a320e2c4f51a8069b@connect-fetchhub.fetch.ai:36456,a575c681c2861fe945f77cb3aba0357da294f1f2@connect-fetchhub.fetch.ai:36457,d7cda986c9f59ab9e05058a803c3d0300d15d8da@connect-fetchhub.fetch.ai:36458`.
```
1 change: 1 addition & 0 deletions docs/mkdocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ nav:
- Installation: 'building.md'
- Versions: 'versions.md'
- Networks: 'networks.md'
- Snapshots: 'snapshots.md'
- Commandline Client:
- Introduction: 'cli-introduction.md'
- Keys: 'cli-keys.md'
Expand Down