Add new package with methods to save and load compressor state #63
Conversation
This enables other packages to use the compressor without jemalloc
Also allow groups_to_compress to be used if min_state_group is None (previously had to set it to something like -1 which was ugly)
auto_compressor/src/state_saving.rs
Outdated
| let sql = r#" | ||
| SELECT level_num, max_size, current_length, current_head, last_compressed | ||
| FROM state_compressor_state as s | ||
| LEFT JOIN state_compressor_progress as p ON s.room_id = p.room_id |
There was a problem hiding this comment.
This LEFT JOIN means that we don't necessarily have an entry in state_compressor_progress, and so last_compressed could be null, which I don't think we're expecting?
There was a problem hiding this comment.
Well you don't want to be able to restart the compressor if progress is NULL and level state is non-empty, but neither do you want to start the compressor from scratch (which would happen if we put a JOIN here)
Consider the following scenario:
1: compressor is run on a room with default level sizes of 100,50,25
2: that room's entry in state_compressor_progress is deleted by accident
3: compressor is run on that same room with default level sizes of 100,50 (i.e. just two levels)
When we save the compressor state, we don't overwrite the contents of the level 3 (the one with max size 25) so when the compressor loads state from the database on the next run it will load that third level with corrupted data in.
I've now added a bail! so it spots this!
- Use PRIMARY KEY rather than UNIQUE() - Use USING for join rather than AS xyz ON - Remove unneccessary ::< on get() - Use column names rather than indices on get() - Add comments about usize <-> i32 conversion - Use INSERT ON ... CONFLICT ... DO UPDATE SET rather than DELETE ... INSERT - Use TRUNCATE rather than DELETE FROM to clear tables
Builds on #62