bitcoind RPC: Make mempool syncing more efficient #465
+104
−16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Querying the mempool unfortunately takes multiple sequential steps: first we call `getrawmempool` to retrieve a list of all entries and then retrieve all of them via `getmempoolentry`. Previously, we would error out if a previously retrieved `Txid` wouldn't be available anymore at the second step. Here, we ensure smooth continuation by simply skipping any entries we can't retrieve anymore.
tnull
requested review from
jkczyz and
arik-so
and removed request for
jkczyz
tnull
force-pushed
the
2025-02-fix-mempool-access
branch
from
ccc699e to
a8dbe2e
Compare
Previously, we would retransmit entry and transaction data whenever polling for mempool data. Here we introduce simple caches for both, so most data is only retrieved in bulk on the first iteration after startup.
.. including `TRACE` logs of relevant interval times
tnull
force-pushed
the
2025-02-fix-mempool-access
branch
from
a8dbe2e to
150ea3d
Compare
arik-so
reviewed
Feb 11, 2025
arik-so
reviewed
Feb 11, 2025
arik-so
reviewed
Feb 11, 2025
| match self.get_raw_transaction(&entry.txid).await { | ||
| Ok(Some(tx)) => { | ||
| mempool_txs_cache.insert(entry.txid, (tx.clone(), entry.time)); | ||
| txs_to_emit.push((tx, entry.time)); | ||
| }, | ||
| Ok(None) => { |
There was a problem hiding this comment.
should the case of having a txid among the mempool entries but not being able to find its full tx hex be handled somehow?
There was a problem hiding this comment.
Well, we handle it by skipping :)
Note that we used to log/error out in some of these races, but its exactly the point to get rid of that behavior in this PR. In fact, turns out you'll regularly run into these races between getrawmempool/getmempoolentry and getmempoolentry/getrawtransaction, since entries are dropped often from the mempool. So we can just skip processing them and will re-detect them if they would reappear in the mempool.
arik-so
approved these changes
Feb 12, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Querying the mempool unfortunately takes multiple sequential steps:
first we call
getrawmempoolto retrieve a list of all entries and thenretrieve all of them via
getmempoolentry.Previously, we would error out if a previously retrieved
Txidwouldn'tbe available anymore at the second step. Here, we ensure smooth
continuation by simply skipping any entries we can't retrieve anymore.
Also we would retransmit entry and transaction data whenever
polling for mempool data. Here we introduce simple caches for both, so
most data is only retrieved in bulk on the first iteration after
startup.