"FATAL: Failed to spend" error during initial sync #283

talebi · 2025-03-24T12:31:29Z

I have been using v1.10 on a Windows machine for a while with no issues until a power failure caused the "db in inconsistent state" error. then I had to resync but every time I try I get an error at different stages like:

[2025-03-24 22:49:36.037] <Controller> Processed height: 256000, 28.8%, 113.8 blocks/sec, 27904.8 txs/sec, 72246.2 addrs/sec
[2025-03-24 22:49:46.235] <Controller> Processed height: 257000, 28.9%, 98.0 blocks/sec, 31380.6 txs/sec, 74520.3 addrs/sec
[2025-03-24 22:49:54.599] <Controller> Processed height: 258000, 29.0%, 119.6 blocks/sec, 28980.2 txs/sec, 78229.2 addrs/sec
[2025-03-24 22:50:03.201] <Controller> FATAL: Failed to spend: f63add26fae45e935ab8a2bedaf1af40437c121e049df582c46569145c817476:1 (spending txid: 91735f64d2eeac80e67376d1a2bfe7a462469610aa87698bd108fd0c8b6cb04b)

The database is now likely in an inconsistent state. To recover, you will need to delete the datadir and do a full resynch. Sorry!

[2025-03-24 22:50:03.202] Stopping Stats HTTP Servers ...
[2025-03-24 22:50:03.202] Stopping Controller ...
[2025-03-24 22:50:03.236] Stopping BitcoinDMgr ...
[2025-03-24 22:50:03.239] Closing storage ...
[2025-03-24 22:50:04.068] Shutdown complete

I am using a quick config file and at this stage totally clueless as what to do.

The text was updated successfully, but these errors were encountered:

cculianu · 2025-03-24T12:38:13Z

Oh wow. On windows huh? I wonder if something with the new rocksdb in 1.11+ somehow is broken on windows.

I did a few full synchs on windows on small testnets and they worked ok. You are on Bitcoin CIA version or on Bitcoin Cash or what?

cculianu · 2025-03-24T12:41:43Z

Also you aren't using the utxo-cache option right?

talebi · 2025-03-24T12:45:25Z

I use it on the mainnet. have tried both with and without utxo-cache, with same results

cculianu · 2025-03-24T12:46:26Z

I use it on the mainnet. have tried both with and without utxo-cache, with same results

Mainnet what? BTC (Bitocin CIA) or BCH (Bitcoin Cash) or LTC (Litecoin CIA)? Fulcrum supports all 3.

talebi · 2025-03-24T12:47:08Z

sorry meant mainnet on bitcoin core

cculianu · 2025-03-24T12:48:50Z

Ok, I'll set up my dev machine here on windows to do a full sync there using Bitcoin CIA version. The problem started happening on 1.11 and above or what? Which version should I use? Also if you go back to 1.10 does it "just work"?

talebi · 2025-03-24T12:52:29Z

thanks. so the problem started on 1.10 (after resync due to power failure) but then upgraded to the latest 1.12 but the same happens. the max I got to was around 56% but generally the error happens around 20-30% mark

talebi · 2025-03-24T12:57:38Z

also the first few lines of logs as per below (if it helps)

PS C:\Fulcrum-1.12.0-win64> .\Fulcrum.exe .\fulcrum-quick-config.conf
[2025-03-24 22:37:52.566] Enabled JSON parser: simdjson
[2025-03-24 22:37:52.566] simdjson implementations:
[2025-03-24 22:37:52.567]     haswell: Intel/AMD AVX2  [supported]
[2025-03-24 22:37:52.567]     westmere: Intel/AMD SSE4.2  [supported]
[2025-03-24 22:37:52.567]     fallback: Generic fallback implementation  [supported]
[2025-03-24 22:37:52.567] active implementation: haswell
[2025-03-24 22:37:52.568] jemalloc: version 5.3.0-0-g54eaed1
[2025-03-24 22:37:52.568] Qt: version 5.15.13
[2025-03-24 22:37:52.568] rocksdb: version 9.2.1-08f9322
[2025-03-24 22:37:52.568] simdjson: version 0.6.0
[2025-03-24 22:37:52.568] ssl: OpenSSL 3.3.0 9 Apr 2024
[2025-03-24 22:37:52.568] zmq: libzmq version: 4.3.3, cppzmq version: 4.10.0
[2025-03-24 22:37:52.568] UPnP: miniupnpc 2.3.0 (API version: 19)
[2025-03-24 22:37:52.568] Fulcrum 1.12.0 (Release 7743cca) - Mon Mar 24, 2025 22:37:52.568 AUS Eastern Daylight Time - starting up ...
[2025-03-24 22:37:52.569] Loading database ...
[2025-03-24 22:37:54.717] DB memory: 512.00 MiB
[2025-03-24 22:37:54.717] Verifying headers ...
[2025-03-24 22:37:54.717] DB version: v3
[2025-03-24 22:37:54.717] BitcoinDMgr: starting 3 bitcoin RPC clients ...
[2025-03-24 22:37:54.718] BitcoinDMgr: started ok
[2025-03-24 22:37:54.718] Stats HTTP: starting 1 server ...
[2025-03-24 22:37:54.718] Starting listener service for HttpSrv 127.0.0.1:8080 ...
[2025-03-24 22:37:54.721] Service started, listening for connections on 127.0.0.1:8080
[2025-03-24 22:37:54.722] <BitcoinDMgr> Coin: BTC
[2025-03-24 22:37:54.830] <Controller> Chain: main
[2025-03-24 22:37:54.830] <Controller> Block height 889225, downloading new blocks ...
[2025-03-24 22:37:54.830] <Controller> utxo-cache: Not enabled
[2025-03-24 22:37:55.060] <Controller> Processed height: 1000, 0.1%, 4390.4 blocks/sec, 4473.7 txs/sec, 4587.7 addrs/sec

cculianu · 2025-03-24T12:58:18Z

Thanks for the log.

thanks. so the problem started on 1.10 (after resync due to power failure) but then upgraded to the latest 1.12 but the same happens. the max I got to was around 56% but generally the error happens around 20-30% mark

This is very disconcerting and indicates some strange low-level DB error with rocksdb.. is my going hypothesis. Will do a full synch today on ciacoin + windows and let you know.

cculianu · 2025-03-25T09:20:28Z

Well so I set it up and am running it on Windows. I'm on Windows10. Connected to bitcoind core 27.0. It's synching just fine so far -- It's at block height 567,000 and counting.

cculianu · 2025-03-25T16:22:34Z

So far I'm at block 665,000 and it's still working ok.

cculianu · 2025-03-25T22:01:30Z

Block 720k. No errors so far.

I think this will likely just succeed to sync. Can you tell me more details about what you are doing? It is mysterious to me that you would consistently get these errors?

Also is your database in some dir that is on a strange filesystem (like a network share or something)?

talebi · 2025-03-26T01:16:09Z

I have a fairly decent machine (intel i7 14th gen, 64GB memory) running Win 11. the database is in C drive somewhere (so no strange share or anything). below are my quick config if it helps:

datadir = C:\FulcrumData\mainnet
bitcoind = 127.0.0.1:8332
rpcuser = someuser
rpcpassword = someP@ss
tcp = 0.0.0.0:50001
peering = false
announce = false
admin = 8000  # <-- 1.2.3.4:8000 notation also accepted here
donation = bitcoincash:qplw0d304x9fshz420lkvys2jxup38m9symky6k028
stats = 8080  # <-- 1.2.3.4:8080 notation also accepted here
utxo_cache = 0

thanks for checking.

cculianu · 2025-03-26T06:22:09Z

Hmm.. and it's ntfs and not fat32 or something like that, right?

I haven't ever once run Windows 11. I'm on Windows 10. Jeez I hope that's not it..

talebi · 2025-03-26T12:14:03Z

yes its ntfs.
just trying to remove a few apps I recently installed to see if it helps. I will report back if I succeed

cculianu · 2025-03-26T12:53:03Z

I wonder if antivirus apps somehow are screwing with the files in the datadir. Maybe see if you can make the datadir for Fulcrum immune to antivirus real-time checking?

cculianu · 2025-03-26T21:31:22Z

Well the sync finished here just fine on my Windows 10 box. No issues. I sincerely hope you get to the bottom of this but since I cannot reproduce it I really can't help.

I don't think it's a problem with Fulcrum itself.. but who knows.

talebi · 2025-03-26T23:09:10Z

I have disabled the windows defender temporarily and hasn't broken so far. not sure if that helped or removing recent apps, but so far so good (74%)

talebi · 2025-04-07T01:47:42Z

just to update, after 2 weeks of struggling (disabling defender, programs etc) I still get the error. I've decided to move on to other options. thanks for all your help much appreciated

cculianu · 2025-04-07T04:21:21Z

Well if you ever figure it out do let me know... very unusual. Never seen anything like it.

Sroose · 2025-04-16T15:42:27Z

I confirm I see the same on v1.12 on Ubuntu.

Fulrum was running stable for a few months on 1.11, but I got a full disk so I had to remove the database and started fully fresh with 1.12.

First attempt was with utxo_cache = 3000 (I have plenty of RAM), this failed after 2.5 hours:
[2025-04-16 15:01:38.499] <Controller> FATAL: batch merge fail for block height 560230: Corruption: block checksum mismatch: stored(context removed) = 1234764856, computed = 1055331840, type = 4
Second attempt was with utxo_cache = 1000, this failed after 3 minutes (at blockheight 221000):

[2025-04-16 15:08:24.099] <Controller> FATAL: Failed to spend: 03f338a8479cd0f94952da693157a533a51440af9232ab8a386e4e48d5851019:1 (spending txid: 467a1c2cb7f8433cd243772e99593f1d47d2f41e8dd80e81b64edabedcfbd462)
The database is now likely in an inconsistent state. To recover, you will need to delete the datadir and do a full resynch. Sorry!

Third attempt was with utxo_cache disabled, like I used to have it. However, it again crashed after 5 minutes at the same tx:

[2025-04-16 15:15:55.910] <Controller> FATAL: Failed to spend: 03f338a8479cd0f94952da693157a533a51440af9232ab8a386e4e48d5851019:1 (spending txid: 467a1c2cb7f8433cd243772e99593f1d47d2f41e8dd80e81b64edabedcfbd462)
The database is now likely in an inconsistent state. To recover, you will need to delete the datadir and do a full resynch. Sorry!

Tried again and still the same, now it consistently crashes at this location.

After each attempt I started with a complete empty .fulcrum data folder. The tx it fails at is a very old one, if there would be a bitcoincore data issue I would assume it did not pass there the first time. So it seems a bit random while at the same time it's now each time the same tx..

Note I did not perform any library/ubuntu updates in a while that would cause this change.

cculianu · 2025-04-16T15:50:44Z

Are you running in docker or anything like that? Also what FS on Ubuntu?

Sroose · 2025-04-16T16:21:02Z

No docker, just the plain precompiled binaries.
FS is type ext4 (rw,noatime), Samsung NVME SSD, LUKS

btw, I reverted to 1.11 and it fails at the same tx now

update:

bitcoin-cli getrawtransaction 03f338a8479cd0f94952da693157a533a51440af9232ab8a386e4e48d5851019 true
No such mempool or blockchain transaction. Use gettransaction for wallet transactions.

seems like now the bitcoin db is probbaly the culprit :'(

cculianu · 2025-04-16T17:19:50Z

Interesting as fuck. And it's always that tx. I really need to investigate this. Wow.

cculianu · 2025-04-16T17:20:10Z

Is -txindex enabled on the bitcoind side? (enabled in bitcoin.conf as txindex=1)??

cculianu · 2025-04-16T17:21:13Z

I actually suspect some form of corruption on the bitcoind side somehow.. tbh... that propagates forward to Fulcrum This is quite a pickle!

Sroose · 2025-04-16T17:23:01Z

yes i have txindex on. Note that i'm using this setup for years now. This year Ive had some more bad luck with corruptions though. Maybe I should try building bitcoin core db from scratch as well

cculianu · 2025-04-16T19:35:21Z

You can try rebuilding with -reindex-chainstate as you may very well know (but that can take a day).

Something is fishy with the blocks I suspect but I cannot be sure.

Sroose · 2025-04-24T16:41:28Z

the cat is out of the bag: some memory modules apparently give bitflips, resulting in the corruption of both bitcoin and fulcrum databases.

cculianu · 2025-04-24T17:25:44Z

Oh man glad it wasn’t this software

But sorry to hear you gotta buy new ram

How did you figure it out? You ran memtet86 or other software to test ram ?

Sroose · 2025-04-24T17:32:18Z

yes, memtest86 (this required a reboot and boot from memtest usb)
But for whoever is in the same boat and wants to test live: stress-ng --vm 2 --vm-bytes 80% -t 5m
I got errors quite quickly from this live test as well so I guess the memory was really messed up.

Good that it was definitely not your software 👍

cculianu added the observation required Dev unable to reproduce situation described by user label Mar 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"FATAL: Failed to spend" error during initial sync #283

"FATAL: Failed to spend" error during initial sync #283

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 •

edited

Loading

cculianu commented Mar 24, 2025

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 •

edited

Loading

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 •

edited

Loading

talebi commented Mar 24, 2025

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 •

edited

Loading

cculianu commented Mar 25, 2025

cculianu commented Mar 25, 2025

cculianu commented Mar 25, 2025

talebi commented Mar 26, 2025

cculianu commented Mar 26, 2025

talebi commented Mar 26, 2025

cculianu commented Mar 26, 2025 •

edited

Loading

cculianu commented Mar 26, 2025

talebi commented Mar 26, 2025

talebi commented Apr 7, 2025

cculianu commented Apr 7, 2025

Sroose commented Apr 16, 2025 •

edited

Loading

cculianu commented Apr 16, 2025

Sroose commented Apr 16, 2025 •

edited

Loading

cculianu commented Apr 16, 2025

cculianu commented Apr 16, 2025

cculianu commented Apr 16, 2025

Sroose commented Apr 16, 2025

cculianu commented Apr 16, 2025

Sroose commented Apr 24, 2025

cculianu commented Apr 24, 2025

Sroose commented Apr 24, 2025

"FATAL: Failed to spend" error during initial sync #283

"FATAL: Failed to spend" error during initial sync #283

Comments

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 • edited Loading

cculianu commented Mar 24, 2025

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 • edited Loading

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 • edited Loading

talebi commented Mar 24, 2025

talebi commented Mar 24, 2025

cculianu commented Mar 24, 2025 • edited Loading

cculianu commented Mar 25, 2025

cculianu commented Mar 25, 2025

cculianu commented Mar 25, 2025

talebi commented Mar 26, 2025

cculianu commented Mar 26, 2025

talebi commented Mar 26, 2025

cculianu commented Mar 26, 2025 • edited Loading

cculianu commented Mar 26, 2025

talebi commented Mar 26, 2025

talebi commented Apr 7, 2025

cculianu commented Apr 7, 2025

Sroose commented Apr 16, 2025 • edited Loading

cculianu commented Apr 16, 2025

Sroose commented Apr 16, 2025 • edited Loading

cculianu commented Apr 16, 2025

cculianu commented Apr 16, 2025

cculianu commented Apr 16, 2025

Sroose commented Apr 16, 2025

cculianu commented Apr 16, 2025

Sroose commented Apr 24, 2025

cculianu commented Apr 24, 2025

Sroose commented Apr 24, 2025

cculianu commented Mar 24, 2025 •

edited

Loading

cculianu commented Mar 24, 2025 •

edited

Loading

cculianu commented Mar 24, 2025 •

edited

Loading

cculianu commented Mar 24, 2025 •

edited

Loading

cculianu commented Mar 26, 2025 •

edited

Loading

Sroose commented Apr 16, 2025 •

edited

Loading

Sroose commented Apr 16, 2025 •

edited

Loading