Package review PR #1

KarthikSubbarao · 2024-04-29T19:40:11Z

This is a PR which merges current changes in unstable into an empty branch in order to help with a review code of the entire codebase.

src/commands/bloom.rs

hpatro · 2024-05-01T22:03:26Z

src/commands/bloom.rs

+    let mut result = Vec::new();
+    match value {
+        Some(bf) => {
+            for item in input_args.iter().take(argc).skip(idx) {
+                result.push(RedisValue::Integer(bf.add_item(item.as_slice())));
+            }
+            Ok(RedisValue::Array(result))
+        }
+        None => {
+            if nocreate {
+                return Err(RedisError::Str("ERR not found"));
+            }
+            let mut bf = BloomFilterType::new_reserved(fp_rate, capacity, expansion);
+            for item in input_args.iter().take(argc).skip(idx) {
+                result.push(RedisValue::Integer(bf.add_item(item.as_slice())));
+            }
+            match filter_key.set_value(&BLOOM_FILTER_TYPE, bf) {
+                Ok(_) => Ok(RedisValue::Array(result)),
+                Err(_) => Err(RedisError::Str(ERROR)),
+            }
+        }


Could we check the bloom filter exists or not first and then insert the data in a single flow. Don't like the code duplication around insertion.

I can create a separate function for multi adds and call it from both flows. I wanted to handle both through the same flow, however it will be out of reference scope and result in a moved error

hpatro · 2024-05-01T22:08:06Z

src/commands/bloom_util.rs

+pub struct BloomFilterType {
+    pub expansion: u32,
+    pub fp_rate: f32,
+    pub filters: Vec<BloomFilter>,


Does the underlying support scalable bloom filters or do we need to support this mechanism?

This particular library does not have auto scaling

https://docs.rs/bloomfilter/1.0.13/bloomfilter/struct.Bloom.html

src/commands/bloom_util.rs

hpatro · 2024-05-01T22:12:18Z

src/commands/bloom_util.rs

+                let new_capacity = filter.capacity * self.expansion;
+                let mut new_filter = BloomFilter::new(self.fp_rate, new_capacity);
+                // Add item.
+                new_filter.bloom.set(item);
+                new_filter.num_items += 1;
+                self.filters.push(new_filter);


Shouldn't we fail the request if the expansion is much higher than we can support?

expansion really means expansion_rate. I can rename this variable if that would help clarify this.

I don't think there is a limit on the number of sub filters an object can have. But (if we want) we can define a config for this to either silently fail expansion (by allowing a set) beyond a limit of X sub filters per object OR explicitly return an error (without a set).

One other aspect we should handle this checking / rejecting based on memory overhead of every operation that creates a new BloomFilter object (BF.ADD, BF.MADD, BD.RESERVE, BF.INSERT, RDB Load). Before any of these operations, we should probably check the memory usage and reject the operations if there is not sufficient space. We have a mechanism to compute the estimated additional memory overhead per creation.

src/wrapper/bloom_callback.rs

src/commands/bloom_data_type.rs

Signed-off-by: Viktor Söderqvist <[email protected]> Co-authored-by: Madelyn Olson <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

… objects. Update RDB load/save Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: VanessaTang <[email protected]>

#37) Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: KarthikSubbarao <[email protected]>

Signed-off-by: zackcam <[email protected]>

Signed-off-by: Nihal Mehta <[email protected]>

…om filter can reach the desired size (#41) * Adding optional arg to BF.INSERT to allow users to check if their bloom filter can reach the desired size Signed-off-by: zackcam <[email protected]> * Fixing ATLEASTCAPACITY calculation as well as adding MAXCAPACITY functionality for info Signed-off-by: zackcam <[email protected]> --------- Signed-off-by: zackcam <[email protected]>

…ity caluclation matches actual capacity (#43) Signed-off-by: zackcam <[email protected]>

…44) file as it can be taken from a dependency and moving metric increments to after creates Signed-off-by: zackcam <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

Signed-off-by: VanessaTang <[email protected]>

… as (#49) adding asan capability to ./build.sh. Updating version of valkey-module as well so memory leak no longer occurs Signed-off-by: zackcam <[email protected]>

Signed-off-by: zackcam <[email protected]>

compatible with non scaling filters as well Signed-off-by: zackcam <[email protected]>

…t platforms + Define default/unstable version, stage + add placeholder release notes (#52) Signed-off-by: KarthikSubbarao <[email protected]>

* Cleaning up Valkey_test_case. Removing ValkeyInfo and ValkeyClient. Making waits consistent. Adding cleanup for AOF folder/files Signed-off-by: zackcam <[email protected]> * Adding functionality for integration tests to work for pytest 6 and 7 Signed-off-by: zackcam <[email protected]> * Decoupling server start from valkey_test_case.py and instead setting it for valkey_bloom_test_case.py Signed-off-by: zackcam <[email protected]> * Removing unneccesary args in startup and duplicate function Signed-off-by: zackcam <[email protected]> --------- Signed-off-by: zackcam <[email protected]>

Signed-off-by: zackcam <[email protected]>

Signed-off-by: Karthik Subbarao <[email protected]>

…github and removing it locally (#56) Signed-off-by: zackcam <[email protected]>

* Adding bloom command json files. These files contain base information about bloom comamnds which will be used by the valkey website Signed-off-by: zackcam <[email protected]> * Update summaries for bloom commands to be more descriptive Taking feedback and applying suggested changes to the summaries of bloom commands Co-authored-by: KarthikSubbarao <[email protected]> Signed-off-by: zackcam <[email protected]> * Apply suggestions from code review Signed-off-by: KarthikSubbarao <[email protected]> * Update .gitignore Signed-off-by: KarthikSubbarao <[email protected]> --------- Signed-off-by: zackcam <[email protected]> Signed-off-by: KarthikSubbarao <[email protected]> Co-authored-by: KarthikSubbarao <[email protected]>

* Cleaning up usage of valkey-test-framework, small updates to build scripts, and cmd handler Signed-off-by: Karthik Subbarao <[email protected]> * Update tests Signed-off-by: Karthik Subbarao <[email protected]> --------- Signed-off-by: Karthik Subbarao <[email protected]>

… so exit does not need to be called (#58) Signed-off-by: zackcam <[email protected]>

KarthikSubbarao force-pushed the unstable branch 3 times, most recently from 61aa1fb to 0e4d4d1 Compare April 30, 2024 18:14

KarthikSubbarao changed the title ~~Package review~~ Package review PR May 1, 2024

KarthikSubbarao self-assigned this May 1, 2024

hpatro reviewed May 1, 2024

View reviewed changes

KarthikSubbarao force-pushed the unstable branch from 055728d to 7027032 Compare July 16, 2024 03:25

KarthikSubbarao force-pushed the unstable branch from c87fd97 to 869a254 Compare August 23, 2024 17:20

KarthikSubbarao pushed a commit that referenced this pull request Aug 23, 2024

Add information about RFC process (#1)

657f83c

Signed-off-by: Viktor Söderqvist <[email protected]> Co-authored-by: Madelyn Olson <[email protected]>

KarthikSubbarao added 21 commits September 10, 2024 23:06

Initial commit

5237a10

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.ADD, BF.EXISTS, BF.CARD

a96c08a

Signed-off-by: Karthik Subbarao <[email protected]>

Update README

be97a25

Signed-off-by: Karthik Subbarao <[email protected]>

Optimize bloom operations to use a reference to already created Bloom…

5ebc1cb

… objects. Update RDB load/save Signed-off-by: Karthik Subbarao <[email protected]>

Remove older Bloom APIs using prev serialization apporach

8557b55

Signed-off-by: Karthik Subbarao <[email protected]>

Add Dev Profile in Cargo.toml for debugging

1729520

Signed-off-by: Karthik Subbarao <[email protected]>

Create Module Datatype and support RDB load, save, free

be80d9d

Signed-off-by: Karthik Subbarao <[email protected]>

Update README.md

ed4d0f9

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.RESERVE and expansion config

2db0e4b

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.INFO

778c219

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.MEXISTS

b11970e

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.MADD

bdf5f0d

Signed-off-by: Karthik Subbarao <[email protected]>

Update error handling / messages and update expansion logic

8f5e3fc

Signed-off-by: Karthik Subbarao <[email protected]>

Add auto scaling support for bloom filters

5545680

Signed-off-by: Karthik Subbarao <[email protected]>

Fix RDB Save for scaled filters

42d015c

Signed-off-by: Karthik Subbarao <[email protected]>

Refactoring

481ecac

Signed-off-by: Karthik Subbarao <[email protected]>

Update TODOs

ff60817

Signed-off-by: Karthik Subbarao <[email protected]>

Fix mem_usage calculation

1c1b6ab

Signed-off-by: Karthik Subbarao <[email protected]>

Update Cargo.toml

d00fa99

Signed-off-by: Karthik Subbarao <[email protected]>

minor refactoring

053d5e9

Signed-off-by: Karthik Subbarao <[email protected]>

Add support for BF.INSERT and fix multi add logic

fbdfb5b

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from fcaad77 to 6a514bf Compare December 19, 2024 18:27

YueTang-Vanessa and others added 2 commits January 2, 2025 19:22

Add test for loading rdb onto non-bloom cluster (#36)

1c949fb

Signed-off-by: VanessaTang <[email protected]>

Large bloom object handling + Rename to BloomFilterType to BloomObject (

ab158bb

#37) Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from 9859dd6 to ab158bb Compare January 11, 2025 00:04

ACL Updates, BF.INSERT updates, other misc updates (#38)

a9b578e

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from d3d7522 to a9b578e Compare January 12, 2025 02:55

KarthikSubbarao and others added 7 commits January 11, 2025 20:01

Update README.md (#39)

1bb7d25

Signed-off-by: KarthikSubbarao <[email protected]>

Adding a bloom command keyspace test (#40)

591ab10

Signed-off-by: zackcam <[email protected]>

Add dump and restore test (#42)

20efd95

Signed-off-by: Nihal Mehta <[email protected]>

Fixing capacity calcualtion and adding a unit test to make sure capac…

56f631f

…ity caluclation matches actual capacity (#43) Signed-off-by: zackcam <[email protected]>

Code cleanup: removing unused dependencies in tests, removing digest (#…

8077d58

…44) file as it can be taken from a dependency and moving metric increments to after creates Signed-off-by: zackcam <[email protected]>

Add License file following Valkey (#46)

8f1c9f4

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch from f46179b to 8f1c9f4 Compare February 15, 2025 02:17

YueTang-Vanessa and others added 8 commits March 10, 2025 10:11

Add server version validation during module loading (#48)

1e6c23b

Signed-off-by: VanessaTang <[email protected]>

Making it so github workflows can see if we have memory leaks as well…

e94a2e7

… as (#49) adding asan capability to ./build.sh. Updating version of valkey-module as well so memory leak no longer occurs Signed-off-by: zackcam <[email protected]>

Adding usage of must_obey_client in wrapper to improve performace (#45)

12fa935

Signed-off-by: zackcam <[email protected]>

Adding tightening ratio to bf.info. Making BF.BLOOM-EXPANSION also (#50)

e308fed

compatible with non scaling filters as well Signed-off-by: zackcam <[email protected]>

Update development script's tests use module extensions from differen…

d23399a

…t platforms + Define default/unstable version, stage + add placeholder release notes (#52) Signed-off-by: KarthikSubbarao <[email protected]>

Adding a quick start guide for bloom (#54)

9d403c4

Signed-off-by: zackcam <[email protected]>

Set default unstable version and update readme.md (#55)

5d514e0

Signed-off-by: Karthik Subbarao <[email protected]>

KarthikSubbarao force-pushed the unstable branch 2 times, most recently from 391d947 to 5d514e0 Compare March 28, 2025 18:55

zackcam and others added 3 commits March 28, 2025 20:33

Updating build script and github workflow to get test framework from …

d6b8763

…github and removing it locally (#56) Signed-off-by: zackcam <[email protected]>

KarthikSubbarao force-pushed the unstable branch from 4c92c47 to 148930e Compare March 31, 2025 21:17

Fixing creation of server in test_rdb_restore_non_bloom_compatibility…

32b0218

… so exit does not need to be called (#58) Signed-off-by: zackcam <[email protected]>

KarthikSubbarao closed this Apr 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Package review PR #1

Package review PR #1

KarthikSubbarao commented Apr 29, 2024 •

edited

Loading

hpatro May 1, 2024

KarthikSubbarao May 1, 2024

hpatro May 1, 2024

KarthikSubbarao May 1, 2024

hpatro May 1, 2024

KarthikSubbarao May 1, 2024

KarthikSubbarao May 2, 2024

Package review PR #1

Package review PR #1

Conversation

KarthikSubbarao commented Apr 29, 2024 • edited Loading

hpatro May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 1, 2024

Choose a reason for hiding this comment

hpatro May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 1, 2024

Choose a reason for hiding this comment

hpatro May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 1, 2024

Choose a reason for hiding this comment

KarthikSubbarao May 2, 2024

Choose a reason for hiding this comment

KarthikSubbarao commented Apr 29, 2024 •

edited

Loading