bug: palomad crashes with "fatal error: concurrent map read and map write"

[freak12techno]

What is happening?

Sometimes palomad crashes with the following error: fatal error: concurrent map read and map write.
If the node doesn't have the autorestart enabled (actually the case for us), it might lead to a validator getting jailed quite quickly.
Namely, we got this error twice in 4 hours, here's the logs of the last one: https://gist.github.com/freak12techno/5eda82de1c69760b53b99d2a1fb84295
Might be an error on Tendermint side, but not sure, need to investigate.

Paloma and pigeon versions and logs

validator@paloma-main ~ ❯ cosmovisor version
cosmovisor version: v1.5.0
1:41AM INF running app args=["version"] module=cosmovisor path=/home/validator/.paloma/cosmovisor/upgrades/v2.4.6/bin/palomad
v2.4.6

pigeond shouldn't be relevant here I guess, but let me know if that's needed.

How to reproduce?

No idea, it just happens randomly.

What is the expected behaviour?

Concurrent read and map doesn't happen => node doesn't crash

[taariq]

FYI @byte-bandit Looks like comet-reated issue. Should we upgrade Paloma to get upgraded version of comet? I see we also have this issue: https://github.com/palomachain/paloma/security/dependabot/49

[taariq]

@freak12techno have you enabled custom mempool flag in the configuration config.toml or app.toml files?

[freak12techno]

@taariq we're pretty much using the default config here.