`hsm_secret` generation from a seed-phrase

[positiveblue]

Hello,

I saw that c-lightning was participating Lightning HackSprint May 2020 and I was interested in helping with one of the "challenges".

The goal of my contribution would be the creation of a launcher or plugin that takes a seed-phrase and generates the associated hsm_secret file.

I am pretty new on the topic so I would like to summarize what I understood after doing a bit of research + reading part of the c-lightning code and try to move forward from there.

When a new lightning node starts up, it launches a daemon (hsmd.c) which creates/loads an hms_secret file. That file contains the 256 bits (32 bytes) used for creating the master private key of a HD Wallet as it is specified in the BIP32. During the creation of the key the bytes are randomly generated using libsodium's cryptographic randomness routine. By now there is no way to derivative those random bytes from a seed in order to backup the wallet.

The idea is to implement some process that allow the node to derive those 32 bytes using a seed-phrase like input.

There are some widely approaches to this like using a 24 word list as specified in the BIP39. However, other lightning implementations like lnd use a similar approach (24 word list) but with another procedure called the aezeed cipher seed scheme.

The one based on aez comes with some sweet features like wallet timestamp and versioning in the seed but I am not sure if those are needed in this case, they are a nice have, or they are not needed at all.

My idea is to discuss which scheme is better for this use case and implement it as an external plugin (I did not investigate yet how plugins are loaded in the program).

Not sure if someone else is already taking care of this, there is no issue for it. I will add @cdecker who seems to be the coordinator for this.

[positiveblue]

@cdecker

@mrostecki and I have been discussing the best way to implement this feature. By now our solution involves a plugin which should be executed before hsmd.c starts up. The daemon uses the function maybe_create_new_hsm to read from a file called hsm_secret which contains the 256 bit (which map to the hsm_secret) in the program. The idea is to generate a hsm_secret from a seed-phrase . Using BIP39 and given 24 words we will create that file (hsm_secret) which will contain the 32 bytes.

If there were no instances of c-lightning already running in the wild we would just need to generate 24 random words during the hsmdc startup process, "print them" and derive the 256 entropy bits from it. However, right now the 256 bits are generated using a random libsodium function so we also need an inverse function: given 256 bits (a hsm_secret file from a running instance) -> 24 words. Using that function we will be able to backup the instances that are already running c-ligthning without using the plugin.

What do you think about it?

[vadorovsky]

We could to use BIP39 functions from libwally-core:

https://wally.readthedocs.io/en/release_0.7.4/bip39/
https://github.com/ElementsProject/libwally-core/blob/master/include/wally_bip39.h

libwally-core is already used in hsmd.c for BIP32:

https://github.com/ElementsProject/lightning/blob/master/hsmd/hsmd.c#L53

Example project using BIP39 functionality from libwally:

https://github.com/greenaddress/garecovery/blob/master/garecovery/mnemonic.py

[positiveblue]

After reading more carefully the BIP39:

To create a binary seed from the mnemonic, we use the PBKDF2 function with a mnemonic sentence (in UTF-8 NFKD) used as the password and the string "mnemonic" + passphrase (again in UTF-8 NFKD) used as the salt. The iteration count is set to 2048 and HMAC-SHA512 is used as the pseudo-random function. The length of the derived key is 512 bits (= 64 bytes).

This seed can be later used to generate deterministic wallets using BIP-0032 or similar methods. 

Which basically means that, to get the seed, we have to HMAC-SHA512 the derived data from the word list (seed-phrase). That makes it impossible for us to write an inverse function for the people who are already running c-lightning and used "random data" as the bip32 seed.

[vadorovsky]

Looking at this test can be helpful:

https://github.com/ElementsProject/libwally-core/blob/master/src/ctest/test_clear.c

[vadorovsky]

Maybe we can generate the master with mnemonic like that?

u8 bip39_seed[BIP39_SEED_LEN_512];
u8 entropy[BIP39_ENTROPY_LEN_256];
size_t bip39_seed_len;
struct words *words;
char *mnemonic;

randombytes_buf(entropy, sizeof(entropy));

bip39_get_wordlist("en", &words);
bip39_mnemonic_from_bytes(words, entropy, sizeof(entropy), &mnemonic);
status_debug("mnemonic: %s", mnemonic);

if (bip39_mnemonic_to_seed(mnemonic, passphrase, bip39_seed, sizeof(bip39_seed),
                           &bip39_seed_len) != WALLY_OK)
    status_failed(STATUS_FAIL_INTERNAL_ERROR,
                  "Can't convert mnemonic to seed");
if (bip32_key_from_seed(bip39_seed, bip39_seed_len,
                        bip32_key_version.bip32_privkey_version,
                        0, &master_extkey) != WALLY_OK)
    status_failed(STATUS_FAIL_INTERNAL_ERROR,
                  "Can't create bip32 master key");

What do you think? Now I will try to think about restoring from mnemonic.

[darosior]

Hi, I may be a bit late but I think it's a good fit for hsmtools. However it would have been great to have mnemonics compatible with other clients... (see #3687)

[positiveblue]

Hi, @darosior we just had a call (@cdecker, @mrostecki and me). Let me summarize what we talked about so you have a bit more of context about the solution that we propose.

To be on the same page, lets sketch the process for getting the master node for the HD wallet tree using BIP39 and BIP32.

Generate X random words ------------> derive Y random bytes ------------> derive the master node for HD Wallets

[Image from BIP32]

Right now c-lightning uses 32 (Y == 32) bytes to derive the master node. The bytes are stored in a file called hsm_secret. If the file is not present, it is generated using a libsodium's cryptographic randomness routine. The program loads the content of the file at start up time.

Because that file contains 32 bytes generated at random it is safe to change directly the code in hsmd.c file and start the process one step before. Generate 24 words at random (in BIP39 if you want 32 bytes you need 24 words) and derive the 32 bytes using BIP39. This change will make the process compatible with other apps using mnemonic seed-phrases. (Although not LND which uses azeed*).

After that, we will add the --use-custom-seed (name may change). The hsmd.c will then read 24 words + passphrase (empty string by default) from standard input and generate the derived hsm_file. The flow of hsmd.c does not have to change, during the call to maybe_create_new_hsm it won't create a new hsm_secret file and load_hms will load the one that already exists. It is important that the flag does not override any hsm_secret file which is already present in the system (potential data lose). In that case, the node simply fail at startup time. It is also important to make it compatible with the --encrypted-hsm flag)

The last step will be to "print" the mnemonic during the startup time when no seed is provided by the user and there is no hsm_secret file present in the folder. The best way to do this has to be decided
yet. I am not sure if it should be possible to print the mnemonic at any point or only when we are generating the hsm_secret for first time. If the node has been running for a while but the process did not exit yet we may be able to print the mnemonic if we save the words in memory. However, if the node restarts for any reason we will be able to recover the wallet (using the hsm_secret file) and operate normally but the mnemonic will be lost for ever.

Given that many other lightning implementations use c-lightning under the hood (easy to create bindings) it could be interesting to write this logic as a little program that reads from std input and writes the hsm_secret file. In that case other lightning implementations can use it without having to call the hsmd daemon.

[vadorovsky]

The following matrix shows cases we should handle in the new seed/masterkey generation logic. Thanks @positiveblue for it!

To be precise, by seed we mean mnemonic (as an optional input from user).

random is the current behavior of c-lightning - using BIP32 seed. no seed is the case when c-lightning will switch to BIP39 seeds and user will not provde mnemonic (so it will be generated and printed).