feat(odap-plugin): backup gateway implementation

Signed-off-by: André Augusto <[email protected]>

Author: AndreAugusto11

packages/cactus-plugin-odap-hermes/README.md

@@ -52,7 +52,7 @@ Firstly let us identify the different entities involved in the protocol and what
 
 The sequence diagram of ODAP is pictured below.
 
-![odap-sequence-diagram](https://imgur.com/a/NN62LyL)
+![odap-sequence-diagram](https://i.imgur.com/AFNvL9v.png)
 
 ### API Endpoints
 This plugin uses OpenAPI to generate the API paths.
@@ -83,45 +83,27 @@ Alice and Bob, in blockchains A and B, respectively, want to make a transfer of
 
 ## Running the tests
 
-For developers that want to test separate steps/phases of the ODAP protocol, please refer to the following test files (client and server side along with the recovery procedure):
+[A test of the entire protocol with manual calls to the methods, i.e. without ledger connectors and Open API.](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap.test.ts)
 
-https://github.com/hyperledger/cactus/tree/main/packages/cactus-plugin-odap-hermes/src/test/typescript/unit/
+[A test of the entire protocol using Open API but with no ledger connectors.](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap-api-call.test.ts)
 
-A test of the entire protocol with manual calls to the methods, i.e. without ledger connectors and Open API:
+[A test of the entire protocol with ledger connectors (Fabric and Besu) and Open API.](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap-api-call-with-ledger-connector.test.ts)
 
-https://github.com/hyperledger/cactus/tree/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap-test.ts
+[A test with a simulated crash of the client gateway after the transfer initiation flow.](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/client-crash-after-transfer-initiation.test.ts)
 
-A test of the entire protocol using Open API but with no ledger connectors:
+[A test with a simulated crash of the client gateway after the lock of the asset in the source blockchain (Fabric).](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/client-crash-after-lock-asset.test.ts)
 
-https://github.com/hyperledger/cactus/tree/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap-api-call.test.ts
+[A test with a simulated crash of the client gateway after the deletion of the asset in the source blockchain (Fabric).](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/client-crash-after-delete-asset.test.ts)
 
-A test of the entire protocol with ledger connectors (Fabric and Besu) and Open API:
+[A test with a simulated crash of the server gateway after the creation of the the asset in the recipient blockchain (Besu).](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/server-crash-after-create-asset.test.ts)
 
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/odap-api-call-with-ledger-connector.test.ts
+[A test with a simulated crash of the server gateway after the transfer initiation flow.](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/server-crash-after-transfer-initiation.test.ts)
 
-A test with a simulated crash of the client gateway after the transfer initiation flow:
+[A test with a rollback after a timeout (client crashed).](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap-rollback.test.ts)
 
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/client-crash-after-transfer-initiation.test.ts
+[A test with a backup gateway resuming the protocol after the client gateway crashed.](https://github.com/hyperledger/cactus/tree/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/backup-gateway-after-client-crash.test.ts)
 
-A test with a simulated crash of the client gateway after the lock of the asset in the source blockchain (Fabric):
-
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/client-crash-after-lock-asset.test.ts
-
-A test with a simulated crash of the client gateway after the deletion of the asset in the source blockchain (Fabric):
-
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/client-crash-after-delete-asset.test.ts
-
-A test with a simulated crash of the server gateway after the creation of the the asset in the recipient blockchain (Besu):
-
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/server-crash-after-create-asset.test.ts
-
-A test with a simulated crash of the server gateway after the transfer initiation flow:
-
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/server-crash-after-transfer-initiation.test.ts
-
-A test with a rollback after a timeout (client crashed):
-
-https://github.com/hyperledger/cactus/blob/main/packages/cactus-plugin-odap-hermes/src/test/typescript/integration/odap/odap-rollback.test.ts
+For developers that want to test separate steps/phases of the ODAP protocol, please refer to [these](https://github.com/hyperledger/cactus/blob/2e94ef8d3b34449c7b4d48e37d81245851477a3e/packages/cactus-plugin-odap-hermes/src/test/typescript/unit/) test files (client and server side along with the recovery procedure).
 
 ## Usage
 

packages/cactus-plugin-odap-hermes/src/main/json/openapi.json

@@ -195,6 +195,18 @@
                     "assetProfile": {
                         "$ref":"#/components/schemas/AssetProfile"
                     },
+                    "allowedSourceBackupGateways": {
+                        "type": "array",
+                        "items": {
+                            "type": "string"
+                        }
+                    },
+                    "allowedRecipientBackupGateways": {
+                        "type": "array",
+                        "items": {
+                            "type": "string"
+                        }
+                    },
                     "sourceBasePath": {
                         "type":"string"
                     },
@@ -460,6 +472,12 @@
                     },
                     "maxTimeout": {
                         "type": "number"
+                    },
+                    "backupGatewaysAllowed": {
+                        "type":"array",
+                        "items":{
+                            "type":"string"
+                        }
                     }
                 },
                 "required": [
@@ -478,7 +496,8 @@
                     "sourceBasePath",
                     "recipientBasePath",
                     "maxRetries",
-                    "maxTimeout"
+                    "maxTimeout",
+                    "backupGatewaysAllowed"
                 ]
             },
             "TransferInitializationV1Response":{
@@ -518,6 +537,12 @@
                     },
                     "signature":{
                         "type":"string"
+                    },
+                    "backupGatewaysAllowed": {
+                        "type":"array",
+                        "items":{
+                            "type":"string"
+                        }
                     }
                 },
                 "required": [
@@ -528,7 +553,8 @@
                     "sessionID",
                     "serverIdentityPubkey",
                     "sequenceNumber",
-                    "signature"
+                    "signature",
+                    "backupGatewaysAllowed"
                 ]
             },
             "TransferCommenceV1Request":{
@@ -1057,6 +1083,15 @@
                     "lastLogEntryTimestamp": {
                         "type": "string"
                     },
+                    "isBackup": {
+                        "type": "boolean"
+                    },
+                    "newBasePath": {
+                        "type": "string"
+                    },
+                    "newGatewayPubKey": {
+                        "type": "string"
+                    },
                     "signature": {
                         "type": "string"
                     }
@@ -1066,6 +1101,8 @@
                     "odapPhase",
                     "sequenceNumber",
                     "lastLogEntryHash",
+                    "isBackup",
+                    "newBasePath",
                     "signature"
                 ]
             },

packages/cactus-plugin-odap-hermes/src/main/typescript/gateway/client/transfer-initialization.ts

@@ -35,7 +35,8 @@ export async function sendTransferInitializationRequest(
     sessionData.lastSequenceNumber == undefined ||
     sessionData.sourceGatewayDltSystem == undefined ||
     sessionData.recipientGatewayPubkey == undefined ||
-    sessionData.recipientGatewayDltSystem == undefined
+    sessionData.recipientGatewayDltSystem == undefined ||
+    sessionData.allowedSourceBackupGateways == undefined
   ) {
     throw new Error(`${fnTag}, session data is not correctly initialized`);
   }
@@ -71,6 +72,7 @@ export async function sendTransferInitializationRequest(
     // permissions
     maxRetries: sessionData.maxRetries,
     maxTimeout: sessionData.maxTimeout,
+    backupGatewaysAllowed: sessionData.allowedSourceBackupGateways,
   };
 
   const messageSignature = PluginOdapGateway.bufArray2HexStr(
@@ -184,6 +186,7 @@ function storeSessionData(
 
   sessionData.serverSignatureInitializationResponseMessage = response.signature;
 
+  sessionData.allowedRecipientBackupGateways = response.backupGatewaysAllowed;
   sessionData.fabricAssetSize = "1";
 
   sessionData.step = 3;

packages/cactus-plugin-odap-hermes/src/main/typescript/gateway/plugin-odap-gateway.ts

@@ -168,6 +168,7 @@ export interface IPluginOdapGatewayConstructorOptions {
   dltIDs: string[];
   instanceId: string;
   keyPair?: IOdapGatewayKeyPairs;
+  backupGatewaysAllowed?: string[];
 
   ipfsPath?: string;
   fabricPath?: string;
@@ -216,6 +217,8 @@ export class PluginOdapGateway implements ICactusPlugin, IPluginWebService {
   //map[]object, object refer to a state
   //of a specific comminications
   public supportedDltIDs: string[];
+  public backupGatewaysAllowed: string[];
+
   private odapSigner: JsObjectSigner;
   public fabricAssetLocked: boolean;
   public fabricAssetDeleted: boolean;
@@ -245,6 +248,8 @@ export class PluginOdapGateway implements ICactusPlugin, IPluginWebService {
     this.name = options.name;
     this.supportedDltIDs = options.dltIDs;
     this.sessions = new Map();
+
+    this.backupGatewaysAllowed = options.backupGatewaysAllowed || [];
     const keyPairs = options.keyPair
       ? options.keyPair
       : Secp256k1Keys.generateKeyPairsBuffer();
@@ -564,8 +569,19 @@ export class PluginOdapGateway implements ICactusPlugin, IPluginWebService {
       const sessionData: SessionData = JSON.parse(log.data);
 
       sessionData.lastLogEntryTimestamp = log.timestamp;
+
+      let amIBackup = false;
+      if (
+        this.pubKey != sessionData.sourceGatewayPubkey &&
+        this.pubKey != sessionData.recipientGatewayPubkey
+      ) {
+        // this is a backup gateway -> for now we assume backup gateways only on the client side
+        sessionData.sourceGatewayPubkey = this.pubKey;
+        amIBackup = true;
+      }
+
       this.sessions.set(sessionID, sessionData);
-      if (remote) await sendRecoverMessage(sessionID, this, true);
+      if (remote) await sendRecoverMessage(sessionID, this, amIBackup, true);
     }
   }
 
@@ -1117,9 +1133,13 @@ export class PluginOdapGateway implements ICactusPlugin, IPluginWebService {
     sessionData.step = 1;
     sessionData.version = request.version;
     sessionData.lastSequenceNumber = randomInt(4294967295);
+
     sessionData.sourceBasePath = request.clientGatewayConfiguration.apiHost;
     sessionData.recipientBasePath = request.serverGatewayConfiguration.apiHost;
 
+    sessionData.allowedSourceBackupGateways = this.backupGatewaysAllowed;
+    sessionData.allowedRecipientBackupGateways = [];
+
     sessionData.payloadProfile = request.payloadProfile;
     sessionData.loggingProfile = request.loggingProfile;
     sessionData.accessControlProfile = request.accessControlProfile;

packages/cactus-plugin-odap-hermes/src/main/typescript/gateway/recovery/recover-update.ts

@@ -127,7 +127,17 @@ export async function checkValidRecoverUpdateMessage(
 
     if (parseInt(recLog.timestamp) > parseInt(maxTimestamp)) {
       maxTimestamp = recLog.timestamp;
-      odap.sessions.set(sessionID, JSON.parse(recLog.data));
+
+      const data = JSON.parse(recLog.data);
+
+      // don't override new gateway public keys in case of being a backup gateway
+      if (odap.isClientGateway(sessionID)) {
+        data.sourceGatewayPubkey = odap.pubKey;
+      } else {
+        data.recipientGatewayPubkey = odap.pubKey;
+      }
+
+      odap.sessions.set(sessionID, data);
     }
   }
 

packages/cactus-plugin-odap-hermes/src/main/typescript/gateway/recovery/recover.ts

@@ -10,6 +10,7 @@ const log = LoggerProvider.getOrCreate({
 export async function sendRecoverMessage(
   sessionID: string,
   odap: PluginOdapGateway,
+  backup: boolean,
   remote: boolean,
 ): Promise<void | RecoverV1Message> {
   const fnTag = `${odap.className}#sendRecoverMessage()`;
@@ -33,6 +34,9 @@ export async function sendRecoverMessage(
     odapPhase: "sessionData.odapPhase",
     sequenceNumber: sessionData.lastSequenceNumber,
     lastLogEntryTimestamp: sessionData.lastLogEntryTimestamp,
+    isBackup: backup,
+    newBasePath: "",
+    newGatewayPubKey: sessionData.sourceGatewayPubkey,
     signature: "",
   };
 
@@ -71,9 +75,45 @@ export async function checkValidRecoverMessage(
     throw new Error(`${fnTag}, session data is undefined`);
   }
 
-  const pubKey = odap.isClientGateway(response.sessionID)
-    ? sessionData.recipientGatewayPubkey
-    : sessionData.sourceGatewayPubkey;
+  let pubKey = undefined;
+
+  if (odap.isClientGateway(response.sessionID)) {
+    if (
+      response.isBackup &&
+      sessionData.recipientGatewayPubkey != response.newGatewayPubKey
+    ) {
+      // this is a backup gateway
+      sessionData.recipientGatewayPubkey = response.newGatewayPubKey;
+
+      if (
+        !sessionData.recipientGatewayPubkey ||
+        !sessionData.allowedSourceBackupGateways?.includes(
+          sessionData.recipientGatewayPubkey,
+        )
+      ) {
+        throw new Error(`${fnTag}, backup gateway not allowed`);
+      }
+    }
+    pubKey = sessionData.recipientGatewayPubkey;
+  } else {
+    if (
+      response.isBackup &&
+      sessionData.sourceGatewayPubkey != response.newGatewayPubKey
+    ) {
+      // this is a backup gateway
+      sessionData.sourceGatewayPubkey = response.newGatewayPubKey;
+
+      if (
+        !sessionData.sourceGatewayPubkey ||
+        !sessionData.allowedSourceBackupGateways?.includes(
+          sessionData.sourceGatewayPubkey,
+        )
+      ) {
+        throw new Error(`${fnTag}, backup gateway not allowed`);
+      }
+    }
+    pubKey = sessionData.sourceGatewayPubkey;
+  }
 
   if (pubKey == undefined) {
     throw new Error(`${fnTag}, session data is undefined`);
@@ -95,5 +135,7 @@ export async function checkValidRecoverMessage(
 
   sessionData.lastLogEntryTimestamp = response.lastLogEntryTimestamp;
 
+  odap.sessions.set(sessionID, sessionData);
+
   log.info(`RecoverMessage passed all checks.`);
 }

packages/cactus-plugin-odap-hermes/src/main/typescript/gateway/server/transfer-initialization.ts

@@ -44,6 +44,7 @@ export async function sendTransferInitializationResponse(
     serverIdentityPubkey: odap.pubKey,
     sequenceNumber: sessionData.lastSequenceNumber,
     signature: "",
+    backupGatewaysAllowed: odap.backupGatewaysAllowed,
   };
 
   transferInitializationResponse.signature = PluginOdapGateway.bufArray2HexStr(
@@ -158,6 +159,10 @@ async function storeSessionData(
   sessionData.version = request.version;
   sessionData.maxRetries = request.maxRetries;
   sessionData.maxTimeout = request.maxTimeout;
+
+  sessionData.allowedSourceBackupGateways = request.backupGatewaysAllowed;
+  sessionData.allowedRecipientBackupGateways = odap.backupGatewaysAllowed;
+
   sessionData.sourceBasePath = request.sourceGatewayPath;
   sessionData.recipientBasePath = request.recipientBasePath;
   sessionData.lastSequenceNumber = request.sequenceNumber;