Feature/persistent gateway storage #784

jstuczyn · 2021-09-21T16:41:20Z

This is something I started working on today while browsing for things to do that will definitely help in the long term (and also in the medium term when expanding on Coconut capabilities).

Basically the purpose of this pull request is going to be to unify the storage on the gateway with everything being stored in the single database. Prior to this change we had (well, technically still do as, at the time of writing this, it is still a draft):

shared keys being stored in sled database
messages for offline clients being stored in individual files directly in the file system (ughh)
available client bandwidth stored in non-persistent in-memory HashMap

As a super nice byproduct I think we will also be able to significantly simplify internal gateway logic and completely get rid of ClientsHandler and all channel communication to it. Even though I wrote it myself over a year ago, I still need to think for a moment to figure out what is going on there. I can only imagine what everyone else must be going through when reading that code (reason for writing it that way was twofold: my own inexperience and the way we were storing messages for offline clients). So future changes should be easier for everyone.

The remaining work includes:

Start using new storage in the ClientsHandler
Introduce managers + build API for bandwidth for the GatewayStorage
Use the new bandwidth manager
Figure out a way of staggering reading messages stored for offline clients (so that we would not be vulnerable to OOM attacks)
Remove the ClientsHandler and possibly introduce some centralised (or behind a lock) in-memory storage with open connections - I haven't yet investigated that part in depth
Update config file with the setup
Possibly some upgrade magic - not really sure if this is going to be worth the hussle
Cleanup
Localnet testing

Closes #11

Maybe it doesn't look like it right now, but once completed it will remove bunch of redundant checks for Nones etc

gateway/src/node/storage/inboxes.rs

neacsu · 2021-09-28T11:52:25Z

gateway/src/node/client_handling/clients_handler.rs

-                    e
-                );
+            if comm_channel.unbounded_send(messages).is_err() {
+                error!("Somehow we failed to stored messages to a fresh client channel - there seem to be a weird bug present!");


nit: Failed to store

This code no longer exists anyway : )

gateway/src/node/client_handling/websocket/connection_handler.rs

neacsu · 2021-09-29T09:13:36Z

gateway/src/node/storage/bandwidth.rs

+        sqlx::query!(
+            r#"
+                UPDATE available_bandwidth
+                SET available = available + ?


How does sqlx handle overflow situations?

That is a very interesting question so I've run an experiment. Basically sqlite will happily store data bigger than maximum size of int64 (so we never lose any "overflow" bandwidth). However, on query the results from sqlx are capped at the maximum value, meaning that in the unlikely scenario of client having more than few exabytes of traffic available, he is not going to see anything above it (but it would still exist in the database)

neacsu · 2021-09-29T09:13:45Z

gateway/src/node/storage/bandwidth.rs

+        sqlx::query!(
+            r#"
+                UPDATE available_bandwidth
+                SET available = available - ?


Same question

And same answer : )

* Sqlx struct stub * Initial schema * Initial error enum * Managed for persisted shared keys * Initial inbox manager * Comments * Using new database in clients handler * Extending gateway storage API * tokio::main + placeholder values * Removed old client store * Simplified logic of async packet processing * Renamed table + not null restriction * BandwidthManager * Removed sled dependency * Using centralised storage for bandwidth * Dead code removal * WIP connection_handler split and simplification Maybe it doesn't look like it right now, but once completed it will remove bunch of redundant checks for Nones etc * Further more explicit clients handler split * Minor cleanup * Temporary store for active client handles * Fixed error types * Error trait on iv and encrypted address * Authentication and registration moved to the handler * Removal of clients handler * Further logic simplification + returned explicit bandwidth values * Further cleanup and comments * Updated config with relevant changes * Basic bandwidth tracking in client * FreshHandle doc comments + fixed stagger issue * Removed side-effects from .map * More doc comments * Database migration on build * Increased default claimed bandwidth * Renaming * Fixed client determining available bandwidth * Removed dead sql table that might be used in the future * Windows workaround * Comment * Return error rather than cap credential

* Feature/persistent gateway storage (#784) * Sqlx struct stub * Initial schema * Initial error enum * Managed for persisted shared keys * Initial inbox manager * Comments * Using new database in clients handler * Extending gateway storage API * tokio::main + placeholder values * Removed old client store * Simplified logic of async packet processing * Renamed table + not null restriction * BandwidthManager * Removed sled dependency * Using centralised storage for bandwidth * Dead code removal * WIP connection_handler split and simplification Maybe it doesn't look like it right now, but once completed it will remove bunch of redundant checks for Nones etc * Further more explicit clients handler split * Minor cleanup * Temporary store for active client handles * Fixed error types * Error trait on iv and encrypted address * Authentication and registration moved to the handler * Removal of clients handler * Further logic simplification + returned explicit bandwidth values * Further cleanup and comments * Updated config with relevant changes * Basic bandwidth tracking in client * FreshHandle doc comments + fixed stagger issue * Removed side-effects from .map * More doc comments * Database migration on build * Increased default claimed bandwidth * Renaming * Fixed client determining available bandwidth * Removed dead sql table that might be used in the future * Windows workaround * Comment * Return error rather than cap credential * Feature/migrate hidden delegations (#786) * Remove migration code * Added function to iterate over delegation of variable type * Add unit tests * Refactored some naming and reused mix/gateway functionality * Borrow bucket instead of move * Linked with existing delegations function * Migration of left-over delegations * Remove unused imports * Put a gateway test as well, next to the mix one * Expose queries for all delegations * Change break point * Added client side calls to the new queries * Fix clippy * Added pagination and read check tests * Fix gateway test from the last commit * Test functions for (de)serialization of identity and owner (in)to storage keys * Add delegation function unit test * Feature guard import * Changed UnpackedDelegation from type to struct * Remove mutable parameter and put start_after in returned value * Made all delegations into iterator for OOM safety * Fix clippy * Add test for delegations iterator size in memory * Change map with if let for ease of read * Use DENOM instead of hardcoded value * Total mixnode stake accounting in the contract * GetTotalMixState query * Add total_mix_stake to validator api * cargo fmt * Handle errors properly * Gateway stake accounting * tests * Naming :) Co-authored-by: Jędrzej Stuczyński <[email protected]> Co-authored-by: Bogdan-Ștefan Neacşu <[email protected]> Co-authored-by: Drazen Urch <[email protected]>

jstuczyn added 30 commits September 21, 2021 12:06

Sqlx struct stub

91ddf6b

Initial schema

56ca1b9

Initial error enum

27e5ac0

Managed for persisted shared keys

403bcfb

Initial inbox manager

4076c5d

Comments

b994d8b

Using new database in clients handler

f53d084

Extending gateway storage API

1fe6c08

tokio::main + placeholder values

e1c0247

Removed old client store

8e896d5

Simplified logic of async packet processing

9fce084

Renamed table + not null restriction

650b159

BandwidthManager

11997ca

Removed sled dependency

34329d7

Using centralised storage for bandwidth

f871dc6

Dead code removal

d697e4c

WIP connection_handler split and simplification

f36e305

Maybe it doesn't look like it right now, but once completed it will remove bunch of redundant checks for Nones etc

Further more explicit clients handler split

02967b0

Minor cleanup

7a87616

Temporary store for active client handles

48548c3

Fixed error types

c9b7b24

Error trait on iv and encrypted address

de1ae7c

Authentication and registration moved to the handler

c2f462e

Removal of clients handler

48ef3e3

Further logic simplification + returned explicit bandwidth values

00c9d20

Further cleanup and comments

ce86348

Updated config with relevant changes

61f11b0

Basic bandwidth tracking in client

54dd567

FreshHandle doc comments + fixed stagger issue

6974e11

Removed side-effects from .map

859a690

jstuczyn added 5 commits September 23, 2021 16:19

More doc comments

07ebd97

Database migration on build

ef63ab3

Increased default claimed bandwidth

5f75b88

Renaming

24b5a16

Fixed client determining available bandwidth

a0faed3

jstuczyn marked this pull request as ready for review September 24, 2021 11:09

jstuczyn requested review from durch, futurechimp and neacsu as code owners September 24, 2021 11:09

jstuczyn added 2 commits September 24, 2021 12:11

Removed dead sql table that might be used in the future

f44024f

Windows workaround

3eceb34

neacsu reviewed Sep 29, 2021

View reviewed changes

jstuczyn added 2 commits September 29, 2021 11:27

Comment

b932635

Return error rather than cap credential

0b5212e

neacsu approved these changes Sep 29, 2021

View reviewed changes

jstuczyn merged commit 12637d9 into develop Sep 30, 2021

jstuczyn deleted the feature/persistent-gateway-storage branch September 30, 2021 08:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/persistent gateway storage #784

Feature/persistent gateway storage #784

jstuczyn commented Sep 21, 2021 •

edited

Loading

neacsu Sep 28, 2021

jstuczyn Sep 29, 2021

neacsu Sep 29, 2021

jstuczyn Sep 29, 2021

neacsu Sep 29, 2021

jstuczyn Sep 29, 2021

Feature/persistent gateway storage #784

Feature/persistent gateway storage #784

Conversation

jstuczyn commented Sep 21, 2021 • edited Loading

neacsu Sep 28, 2021

Choose a reason for hiding this comment

jstuczyn Sep 29, 2021

Choose a reason for hiding this comment

neacsu Sep 29, 2021

Choose a reason for hiding this comment

jstuczyn Sep 29, 2021

Choose a reason for hiding this comment

neacsu Sep 29, 2021

Choose a reason for hiding this comment

jstuczyn Sep 29, 2021

Choose a reason for hiding this comment

jstuczyn commented Sep 21, 2021 •

edited

Loading