Skip to content

Serialisation of (online) state for online detectors #604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 48 commits into from
Jan 24, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
2171395
Initial implementation of save/load state for LSDDDriftOnlineTorch
Sep 2, 2022
68883d4
Add tests for saving state
Sep 7, 2022
944a65d
WIP: LSDDDriftOnlineTF implementation
Nov 3, 2022
672fbcc
State save/load/reset for online LSDD (tf) and MMD (pt and tf)
Nov 7, 2022
9411d54
Online FET and CVM implementation
Nov 7, 2022
d3c397a
Update LSDD and MMD to reset via reset() rather than reset_state()
Nov 7, 2022
06a032b
use .get for state_dict in mmd/lsdd for consistency
Nov 7, 2022
9ea9339
WIP: saving state via save_detector. Need to add tests
Nov 7, 2022
faa483b
Fixes, and tests for save_detector save_state kwarg
Nov 7, 2022
ec48bb6
Merge branch 'master' into feature/save_state
Nov 23, 2022
6deb17f
Fix tests
Nov 23, 2022
272ef3a
Update changelog
Nov 24, 2022
e9c3685
Update docs
Nov 24, 2022
f73fbbf
POC of refactoring. LSDDDriftOnlineTorch only
Nov 30, 2022
09c4305
Refactor initialise methods (and tests)
Dec 14, 2022
1774ee9
Move save/load_state to base classes
Dec 14, 2022
d9d2e25
Manage online test seeds with fixed_seed
Dec 15, 2022
975094e
Add missing _state.py files
Dec 15, 2022
363b435
Fix ops deps tests
Dec 15, 2022
cfc8918
Remove misplaced offline_state code
Dec 15, 2022
30bc57a
Fix test_saving
Dec 15, 2022
1d81d35
Merge branch 'master' into feature/save_state
Jan 4, 2023
2d9c5cf
Remove uneccesary Framework import
Jan 4, 2023
bb53453
Simplify online state tests
Jan 4, 2023
bb4035a
Incorperate some updates from feature/save_offline_state
Jan 5, 2023
430ebd5
Revert 21519e4, but keep changes to tests and additions to online_sta…
Jan 10, 2023
91e1b61
Test saving of state for all online detectors in test_saving.py
Jan 10, 2023
9e84410
Added logging messages
Jan 10, 2023
5dea156
Remove old _set_state_path method
Jan 10, 2023
785365d
Make StatefulDetector inherit from ConfigurableDetector
Jan 10, 2023
7ce6476
Replace indexing with tf.gather
Jan 11, 2023
a2bce2b
Merge branch 'fix/tf_indexing_bug' into feature/save_state
Jan 11, 2023
b2806cc
Add filepath check back to _load_detector_config
Jan 12, 2023
2d4b6a3
Rename StatefulDetector to StatefulDetectorOnline
Jan 12, 2023
67915b8
Add online_state_keys attr to StatefulDetectorOnline
Jan 12, 2023
feb9fda
Revert "Add online_state_keys attr to StatefulDetectorOnline"
Jan 12, 2023
aa6b0d3
Rename rest to reset_state
Jan 12, 2023
15d1dfa
Update typings
Jan 13, 2023
da56714
Remove extra parentheses
Jan 16, 2023
6240fdf
Remove unused seed's in tests
Jan 16, 2023
5daf1b1
Add BaseDriftOnline class
Jan 16, 2023
c7d72c7
Add description for BaseUniDriftOnline
Jan 16, 2023
d190589
Remove save_state kwarg from save_detector
Jan 16, 2023
4de6fad
Move state methods to StateMixin
Jan 17, 2023
7b824d8
Update docs
Jan 18, 2023
d888276
Only mention .predict() in saving page
Jan 18, 2023
1026713
Move conftest.py's containing seed fixture to local test dirs
Jan 18, 2023
828e486
Add missing conftest
Jan 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/cd/methods/onlinecvmdrift.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@
"cd.load_state('checkpoint_t1')\n",
"```\n",
"\n",
"At any point, the state may be reset with the `reset_state` method."
"At any point, the state may be reset to `t=0` with the `reset_state` method. When saving the detector with `save_detector`, the state will be saved, unless `t=0` (see [here](../../overview/saving.md#online-detectors))."
]
}
],
Expand Down
2 changes: 1 addition & 1 deletion doc/source/cd/methods/onlinefetdrift.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@
"cd.load_state('checkpoint_t1')\n",
"```\n",
"\n",
"At any point, the state may be reset with the `reset_state` method."
"At any point, the state may be reset to `t=0` with the `reset_state` method. When saving the detector with `save_detector`, the state will be saved, unless `t=0` (see [here](../overview/saving.md#online-detectors))."
]
},
{
Expand Down
2 changes: 1 addition & 1 deletion doc/source/cd/methods/onlinelsdddrift.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@
"cd.load_state('checkpoint_t1')\n",
"```\n",
"\n",
"At any point, the state may be reset with the `reset_state` method."
"At any point, the state may be reset to `t=0` with the `reset_state` method. When saving the detector with `save_detector`, the state will be saved, unless `t=0` (see [here](../overview/saving.md#online-detectors))."
]
},
{
Expand Down
2 changes: 1 addition & 1 deletion doc/source/cd/methods/onlinemmddrift.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@
"cd.load_state('checkpoint_t1')\n",
"```\n",
"\n",
"At any point, the state may be reset with the `reset_state` method."
"At any point, the state may be reset to `t=0` with the `reset_state` method. When saving the detector with `save_detector`, the state will be saved, unless `t=0` (see [here](../overview/saving.md#online-detectors))."
]
},
{
Expand Down
32 changes: 20 additions & 12 deletions doc/source/overview/saving.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,26 +143,34 @@ the scikit-learn API.
```
````

## Saving state
## Online detectors

[Online drift detectors](../cd/methods.md#online) are stateful, with their state updated upon each `predict` call.
When saving an online detector, the `save_state` option controls whether to include the detector's state:
[Online drift detectors](../cd/methods.md#online) are stateful, with their state updated each timestep `t` (each time
`.predict()` or `.state()` is called). {func}`~alibi_detect.saving.save_detector` will save the state of online
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean score() instead of state()? On that note, do we even document the usage/use cases of score()? If not, perhaps should leave it out.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good spot thanks. Also fair point about not really documenting it. I'll remove.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed .state() in d888276

detectors to disk if `t > 0`. At load time, {func}`~alibi_detect.saving.load_detector` will load this state.
For example:

```python
from alibi_detect.cd import LSDDDriftOnline
from alibi_detect.saving import save_detector, load_detector

# Init detector (self.t = 0)
dd = LSDDDriftOnline(x_ref, ert, window_size)
# Init detector (t=0)
dd = LSDDDriftOnline(x_ref, window_size=10, ert=50)

# Perform predict call to update state (e.g. self.t = self.t + 1)
dd.predict(x)
# Run 2 predictions
pred_1 = dd.predict(x_1) # t=1
pred_2 = dd.predict(x_2) # t=2

# Save detector with its state included
save_detector(dd, filepath, save_state=True)
# Save detector (state will be saved since t>0)
save_detector(dd, filepath)

# Load stateful detector (i.e. self.t = 1)
dd_new = load_detector(filepath)
# Load detector
dd_new = load_detector(filepath) # detector will start at t=2
```

Note that `load_detector` will load an online detector's state if it exists within the save directory.
To save a clean (stateless) detector, it should be reset before saving:

```python
dd.reset_state() # reset to t=0
save_detector(dd, filepath) # save the detector without state
```