Bad snapshot created on warehouse mainnet-beta nodes running v1.0.14

[sakridge]

Problem

[2020-04-10T10:17:54.252736523Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 65536
[2020-04-10T10:17:54.253330868Z INFO  solana_ledger::blockstore] Opening database at "/home/sol/snapshot-check/rocksdb"
[2020-04-10T10:17:54.458421763Z INFO  solana_ledger::blockstore] "/home/sol/snapshot-check/rocksdb" open took 205ms
[2020-04-10T10:17:54.458503174Z INFO  solana_ledger::bank_forks_utils] Initializing snapshot path: "/home/sol/snapshot-check/snapshot"
[2020-04-10T10:17:54.459968072Z INFO  solana_ledger::bank_forks_utils] Loading snapshot package: "/home/sol/snapshot-check/snapshot-4770249-4
[2020-04-10T10:18:05.021372721Z INFO  solana_ledger::snapshot_utils] snapshot untar took 10.6s
[2020-04-10T10:18:05.022530605Z INFO  solana_ledger::snapshot_utils] snapshot version: 1.1.0
[2020-04-10T10:18:05.022846772Z INFO  solana_ledger::snapshot_utils] Loading bank from "/home/sol/snapshot-check/snapshot/.tmpArAsi3/snapshot
[2020-04-10T10:18:05.032469937Z INFO  solana_ledger::snapshot_utils] Rebuilding accounts...
[2020-04-10T10:18:07.419037126Z INFO  solana_ledger::snapshot_utils] Rebuilding status cache...
[2020-04-10T10:18:07.447397802Z INFO  solana_ledger::snapshot_utils] Loaded bank for slot: 4770249
[2020-04-10T10:18:07.447625073Z INFO  solana_runtime::accounts_db] total_stores: 1133, newest_slot: 4770249, oldest_slot: 0, max_slot: 477024
[2020-04-10T10:18:07.447772246Z INFO  solana_metrics::metrics] metrics configuration: host=https://metrics.solana.com:8086 db=mainnet-beta us
[2020-04-10T10:18:07.447994191Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=1133i
[2020-04-10T10:18:07.476960141Z WARN  solana_runtime::accounts_db] mismatched bank hash for slot 4770249: 687b6oxzAben8tmZvGVxZ6LmnPiB3tqcYRW
[2020-04-10T10:18:07.477002898Z WARN  solana_runtime::accounts] verify_bank_hash failed: MismatchedBankHash
thread 'main' panicked at 'Snapshot bank for slot 4770249 failed to verify', ledger/src/snapshot_utils.rs:461:9

Proposed Solution

Debug and fix.

[mvines]

I've collected the relevant snapshots and logs at:

sol@warehouse-us-west1-b:~/github-issue-9431$ ls -l
total 899884
drwxrwxr-x 2 sol sol      4096 Apr 10 16:58 bad
drwxrwxr-x 2 sol sol      4096 Apr 10 17:04 good
-rw-r--r-- 1 sol sol 921468745 Apr 10 17:06 validator.log

and

sol@warehouse-europ-west4-c:~/github-issue-9431$ ls -l
total 926540
drwxrwxr-x 2 sol sol      4096 Apr 10 17:27 bad
drwxrwxr-x 2 sol sol      4096 Apr 10 17:31 good
-rw-r--r-- 1 sol sol 948762931 Apr 10 17:26 validator.log

[sakridge]

Bad snapshot has:

key: HUW8DQVebeF8UApvNmG3kVJ8U1b6aYimC3RDbmmoMjni hash: 71PuKpg

[sakridge]

HUW8DQVebeF8UApvNmG3kVJ8U1b6aYimC3RDbmmoMjni => 
StoredAccount { 
meta: StoredMeta { write_version: 462550075, 
pubkey: HUW8DQVebeF8UApvNmG3kVJ8U1b6aYimC3RDbmmoMjni, 
data_len: 0 }, 
account_meta: Account
Meta { lamports: 616999995000, owner: 11111111111111111111111111111111, executable: false,
rent_epoch: 11 }, 
data: [], offset: 27224,
 hash: 71PuKpgepmxwBcnqGA5wpTHAJNGQKPDdTJAxZnNTdQ2X }

[sakridge]

accounts list: RwLockReadGuard { lock: RwLock { 
data: (1,
  [(4337312,
   AccountInfo { store_id: 17350773, offset: 27224, lamports: 616999995000 })]) } } 
index: 0

[sakridge]

good one, has an update that cleans up the account in slot 4762184:

accounts list: RwLockReadGuard { lock: RwLock { 
data: (2, [
(4337312, 
   AccountInfo { store_id: 17350773, offset: 27224, lamports: 616999995000 }),
(4762184, 
   AccountInfo { store_id: 19206749, offset: 27088, lamports: 0 })]) } } 
index: 1

 4337312 account: Some(StoredAccount { meta: StoredMeta { write_version: 462550075, pubkey: HUW8DQVebeF8UApvNmG3kVJ8U1b6aYimC3RDbmmoMjni, data_len: 0 },
account_meta: 
AccountMeta { lamports: 616999995000, owner: 11111111111111111111111111111111, executable: false, rent_epoch: 11 }, data: [], offset: 27224, 
hash: 71PuKpgepmxwBcnqGA5wpTHAJNGQKPDdTJAxZnNTdQ2X })

  4762184 account: Some(StoredAccount { meta: StoredMeta { write_version: 515166176, pubkey: 
HUW8DQVebeF8UApvNmG3kVJ8U1b6aYimC3RDbmmoMjni, data_len: 0 }, 
account_meta: 
AccountMeta { lamports: 0, owner: 11111111111111111111111111111111,
executable: false, rent_epoch: 0 }, 
data: [], offset: 27088, hash: 11111111111111111111111111111111 
})

[sakridge]

cc @ryoqun