Weird fishtest fastchess issues

[Disservin]

Describe the issue

Ever since we enabled illegal moves and illegal pv moves checks in fishtest which cause the run to stop there have been a few rare occurences of that happening on otherwise "okay" looking patches..

https://tests.stockfishchess.org/actions?action=&user=&text=%22fastchess+says%3A%22

Most recently

25‑02‑20 18:52:54 stop_run TueRens-56cores-144c3c5d-5a91 Risk_tolerance_calculat-bc87289/798 fastchess says: 'Warning; Illegal pv move e1e2 pv; info depth 1 seldepth 6 multipv 1 score cp -42 nodes 51 nps 51000 hashfull 0 tbhits 0 time 1 pv e1e2'

which is somewhat weird because black (master) apparently made that move and there is no black piece anywhere near that

[Event "Batch 798: Risk_tolerance_calculation vs master"]
[Site "https://tests.stockfishchess.org/tests/view/67b2f25138b351d1ce6405db"]
[Date "2025.02.20"]
[Round "116"]
[White "New-3c79eadda08d623caa7954f168e5c4869788f6e4"]
[Black "Base-fa6c30af814fe91e6a6c2d1bcaa8d951e3724ae7"]
[Result "1-0"]
[SetUp "1"]
[FEN "r1bqk2r/pppp1ppp/2nb3n/4p1N1/2BPP3/8/PPP2PPP/RNBQK2R w KQkq - 3 6"]
[GameDuration "00:02:12"]
[GameStartTime "2025-02-20T19:50:41 +0100"]
[GameEndTime "2025-02-20T19:52:53 +0100"]
[PlyCount "38"]
[Termination "illegal move"]
[TimeControl "78.678+0.787"]

1. dxe5 {+1.14/24 14.923s} Nxe5 {-1.10/24 18.817s} 2. Bb3 {+1.08/18 0.846s}
O-O {-1.00/17 0.833s} 3. Nc3 {+1.07/19 2.837s} Kh8 {-1.06/21 7.201s}
4. f4 {+1.00/24 15.698s} Ng6 {-0.92/17 1.099s} 5. f5 {+1.10/16 0.765s}
Nf4 {-0.85/20 1.555s} 6. Nf3 {+1.41/19 1.726s} Nxg2+ {-1.35/21 4.251s}
7. Kf1 {+1.40/18 1.378s} Nh4 {-1.30/20 1.879s} 8. Rg1 {+1.38/20 1.230s}
Nxf3 {-1.30/19 1.415s} 9. Qxf3 {+1.30/19 1.199s} Ng8 {-1.21/21 4.060s}
10. Bg5 {+1.23/21 1.642s} Be7 {-1.15/18 1.249s} 11. Bxe7 {+1.33/19 1.057s}
Nxe7 {-1.22/24 11.364s} 12. Nb5 {+1.73/19 1.939s} d5 {-0.76/20 3.423s}
13. Qc3 {+1.72/23 8.461s} Rg8 {-0.82/19 1.528s} 14. Nxc7 {+1.83/16 0.926s}
Bxf5 {-0.18/16 1.111s} 15. Nxa8 {+2.02/19 1.569s} Bxe4 {-0.17/18 1.014s}
16. Nc7 {+1.93/20 1.050s} Nf5 {0.00/24 3.356s} 17. Re1 {+1.86/19 1.626s}
Qh4 {0.00/35 1.370s} 18. Nxd5 {+2.87/17 1.157s} Bxd5 {+0.42/15 0.979s}
19. Bxd5 {+4.63/19 1.102s} e1e2 {-0.38/23 4.315s, Black makes an illegal move} 1-0

the evaluation from master is also "weird" relatively drawish score while a good amount ahead, browser fish agrees with +2...

needs some further digging to findout what went wrong here or in the other occurences

[Disservin]

while i can't explain the weird scores, a bitflip in i.e. sf's stm would explain e1e2 (i think that's a relatively okay move to play here for an engine?)

[vdbergh]

Is it conceivable that fc somehow missed the actual black move and then took the white move for the black one (I have not opened the pgn).

[vondele]

IIUC, so far we just have 1 occurrence of this issue with a non-suspect patch, so I think we can wait and see. If it is a suspect worker, we'll see it occur more often on that hardware, while if it is SF, it should show up on multiple workers.

Would actually be interesting to know if the worker has ECC memory or not.

[vdbergh]

To me it really seems a case of Murphy's law . This event occurred right after I made resurrecting a failed run illegal (this has since been reverted). I suspect that the phenomenon of a worker, which is otherwise behaving entirely normally, causing a single correct patch to fail is extremely rare,

[Disservin]

To me it really seems a case of Murphy's law . This event occurred right after I made resurrecting a failed run illegal (this has since been reverted). I suspect that the phenomenon of a worker, which is otherwise behaving entirely normally, causing a single correct patch to fail is extremely rare,

same with my patch which enabled the reporting of illegal moves which was added right before a patch which causes this :D

[Disservin]

Would actually be interesting to know if the worker has ECC memory or not.

"Unfortunately, the RAM is not ECC.."

[xu-shawn]

Perhaps this is something #5681 could help with, if fishtest starts annotating pgn with node count?