Skip to content

Commit 9fef336

Browse files
outSHpetermetz
authored andcommitted
feat(persistence-fabric): add sample setup scripts, improve documentation
- Fix error when empty `transaction.actions` was received from a block. - Copy schema SQL during build. - Remove unique constraint on fabric transactions hash column (there are transactions with empty hash that would break it). - Improve parsing of certificate subject/issuer attributes to handle more delimiters. - Use new `install-fabric.sh` script (that is recommended for Fabric 2.5) instead of old bootstrap to fix some runtime issues that I've encountered. - Add sample setup scripts. Simple can be used to run persistence against already running fabric ledger, complete will setup entire environment and run some basic operations to generate sample data. - Improve documentation to include these new scripts and how to use them, fix smaller issues. Signed-off-by: Michal Bajer <[email protected]>
1 parent 9ce9057 commit 9fef336

File tree

12 files changed

+552
-47
lines changed

12 files changed

+552
-47
lines changed

packages/cactus-plugin-ledger-connector-fabric/src/main/typescript/get-block/cacti-block-formatters.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -79,7 +79,7 @@ export function formatCactiFullBlockResponse(
7979
const transaction = payload.data;
8080

8181
const transactionActions: FullBlockTransactionActionV1[] = [];
82-
for (const action of transaction.actions) {
82+
for (const action of transaction.actions ?? []) {
8383
const actionPayload = action.payload;
8484
const proposalPayload = actionPayload.chaincode_proposal_payload;
8585
const invocationSpec = proposalPayload.input;

packages/cactus-plugin-persistence-fabric/README.md

+71-30
Original file line numberDiff line numberDiff line change
@@ -25,53 +25,94 @@ Clone the git repository on your local machine. Follow these instructions that w
2525

2626
### Prerequisites
2727

28+
#### Build
29+
2830
In the root of the project, execute the command to install and build the dependencies. It will also build this persistence plugin:
2931

3032
```sh
3133
yarn run configure
3234
```
3335

34-
### Usage
36+
#### Hyperledger Fabric Ledger and Connector
3537

36-
Instantiate a new `PluginPersistenceFabric` instance:
38+
This plugin requires a running Hyperledger Fabric ledger that you want to persist to a database. For testing purposes, you can use our [test fabric-all-in-one Docker image](../../tools/docker/fabric-all-in-one/README.md). To access the ledger you'll need your organization connection profile JSON and a wallet containing registered identity. If you are using our `fabric-all-in-one` image, you can run our [asset-transfer-basic-utils scripts](../../tools/docker/fabric-all-in-one/asset-transfer-basic-utils/README.md) to fetch Org1 connection profile from a docker and register new user to a localhost wallet.
3739

38-
```typescript
39-
import { PluginPersistenceFabric } from "@hyperledger/cactus-plugin-persistence-fabric";
40-
import { v4 as uuidv4 } from "uuid";
40+
```shell
41+
# Start the test ledger
42+
docker compose -f tools/docker/fabric-all-in-one/docker-compose-v2.x.yml up
43+
# Wait for it to start (status should become `healthy`)
4144

42-
const persistencePlugin = new PluginPersistenceFabric({
43-
apiClient,
44-
logLevel: "info",
45-
instanceId,
46-
connectionString: "postgresql://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres",,
47-
channelName: "mychannel",
48-
gatewayOptions: {
49-
identity: signingCredential.keychainRef,
50-
wallet: {
51-
keychain: signingCredential,
52-
},
53-
},
54-
});
45+
# Run asset-transfer-basic-utils scripts
46+
cd tools/docker/fabric-all-in-one/asset-transfer-basic-utils
47+
# Cleanup artifacts from previous runs
48+
rm -fr wallet/ connection.json
49+
# Fetch connection profile to `tools/docker/fabric-all-in-one/asset-transfer-basic-utils/connection.json`
50+
# Enroll user using wallet under `tools/docker/fabric-all-in-one/asset-transfer-basic-utils/wallet`
51+
npm install
52+
CACTUS_FABRIC_ALL_IN_ONE_CONTAINER_NAME=fabric_all_in_one_testnet_2x ./setup.sh
53+
```
5554

56-
// Initialize the connection to the DB
57-
await persistencePlugin.onPluginInit();
55+
Once you have an Fabric ledger ready, you need to start the [Ethereum Cacti Connector](../cactus-plugin-ledger-connector-fabric/README.md). We recommend running the connector on the same ApiServer instance as the persistence plugin for better performance and reduced network overhead. See the connector package README for more instructions, or check out the [setup sample scripts](./src/test/typescript/manual).
56+
57+
#### Supabase Instance
58+
59+
You need a running Supabase instance to serve as a database backend for this plugin.
60+
61+
### Setup Tutorials
62+
63+
We've created some sample scripts to help you get started quickly. All the steps have detailed comments on it so you can quickly understand the code.
64+
65+
#### Sample Setup
66+
67+
Location: [./src/test/typescript/manual/sample-setup.ts](./src/test/typescript/manual/sample-setup.ts)
68+
69+
This sample script can be used to set up `ApiServer` with the Fabric connector and persistence plugins to monitor and store ledger data in a database. You need to have a ledger running before executing this script.
70+
71+
To run the script you need to set the following environment variables:
72+
73+
- `FABRIC_CONNECTION_PROFILE_PATH`: Full path to fabric ledger connection profile JSON file.
74+
- `FABRIC_CHANNEL_NAME`: Name of the channel we want to connect to (to store it's data).
75+
- `FABRIC_WALLET_PATH` : Full path to wallet containing our identity (that can connect and observe specified channel).
76+
- `FABRIC_WALLET_LABEL`: Name (label) of our identity in a wallet provided in FABRIC_WALLET_PATH
77+
78+
By default, the script will try to use our `supabase-all-in-one` instance running on localhost. This can be adjusted by setting PostgreSQL connection string in `SUPABASE_CONNECTION_STRING` environment variable (optional).
79+
80+
```shell
81+
# Example assumes fabric-all-in-one was used. Adjust the variables accordingly.
82+
FABRIC_CONNECTION_PROFILE_PATH=/home/cactus/tools/docker/fabric-all-in-one/asset-transfer-basic-utils/connection.json FABRIC_CHANNEL_NAME=mychannel FABRIC_WALLET_PATH=/home/cactus/tools/docker/fabric-all-in-one/asset-transfer-basic-utils/wallet FABRIC_WALLET_LABEL=appUser
83+
node ./dist/lib/test/typescript/manual/sample-setup.js
5884
```
5985

60-
Alternatively, import `PluginFactoryLedgerPersistence` from the plugin package and use it to create a plugin.
86+
#### Complete Sample Scenario
87+
88+
Location: [./src/test/typescript/manual/common-setup-methods](./src/test/typescript/manual/common-setup-methods)
89+
90+
This script starts the test Hyperledger Fabric ledger for you and executes few transactions on a `basic` chaincode. Then, it synchronizes everything to a database and monitors for all new blocks. This script can also be used for manual, end-to-end tests of a plugin.
91+
92+
By default, the script will try to use our `supabase-all-in-one` instance running on localhost.
93+
94+
```shell
95+
npm run complete-sample-scenario
96+
```
97+
98+
Custom supabase can be set with environment variable `SUPABASE_CONNECTION_STRING`:
99+
100+
```shell
101+
SUPABASE_CONNECTION_STRING=postgresql://postgres:[email protected]:5432/postgres npm run complete-sample-scenario
102+
```
103+
104+
### Usage
105+
106+
Instantiate a new `PluginPersistenceFabric` instance:
61107

62108
```typescript
63-
import { PluginFactoryLedgerPersistence } from "@hyperledger/cactus-plugin-persistence-fabric";
64-
import { PluginImportType } from "@hyperledger/cactus-core-api";
109+
import { PluginPersistenceFabric } from "@hyperledger/cactus-plugin-persistence-fabric";
65110
import { v4 as uuidv4 } from "uuid";
66111

67-
const factory = new PluginFactoryLedgerPersistence({
68-
pluginImportType: PluginImportType.Local,
69-
});
70-
71-
const persistencePlugin = await factory.create({
72-
apiClient,
112+
const persistencePlugin = new PluginPersistenceFabric({
113+
apiClient: new FabricApiClient(apiConfigOptions),
73114
logLevel: "info",
74-
instanceId,
115+
instanceId: "my-instance",
75116
connectionString: "postgresql://postgres:your-super-secret-and-long-postgres-password@localhost:5432/postgres",,
76117
channelName: "mychannel",
77118
gatewayOptions: {

packages/cactus-plugin-persistence-fabric/package.json

+8-2
Original file line numberDiff line numberDiff line change
@@ -47,14 +47,18 @@
4747
"dist/*"
4848
],
4949
"scripts": {
50-
"build": "npm run build-ts",
50+
"build": "npm run build-ts && npm run build:dev:backend:postbuild",
5151
"build-ts": "tsc",
52+
"build:dev:backend:postbuild": "npm run copy-sql && npm run copy-yarn-lock",
5253
"codegen": "yarn run --top-level run-s 'codegen:*'",
5354
"codegen:openapi": "npm run generate-sdk",
54-
"copy-yarn-lock": "cp -af ../../yarn.lock ./dist/yarn.lock",
55+
"complete-sample-scenario": "npm run build && node ./dist/lib/test/typescript/manual/complete-sample-scenario.js",
56+
"copy-sql": "mkdir -p ./dist/lib/main/ && cp -Rfp ./src/main/sql ./dist/lib/main/",
57+
"copy-yarn-lock": "mkdir -p ./dist/lib/ && cp -rfp ../../yarn.lock ./dist/yarn.lock",
5558
"generate-sdk": "run-p 'generate-sdk:*'",
5659
"generate-sdk:go": "openapi-generator-cli generate -i ./src/main/json/openapi.json -g go -o ./src/main/go/generated/openapi/go-client/ --git-user-id hyperledger --git-repo-id $(echo $npm_package_name | replace @hyperledger/ \"\" -z)/src/main/go/generated/openapi/go-client --package-name $(echo $npm_package_name | replace @hyperledger/ \"\" -z) --reserved-words-mappings protected=protected --ignore-file-override ../../openapi-generator-ignore",
5760
"generate-sdk:typecript-axios": "openapi-generator-cli generate -i ./src/main/json/openapi.json -g typescript-axios -o ./src/main/typescript/generated/openapi/typescript-axios/ --reserved-words-mappings protected=protected --ignore-file-override ../../openapi-generator-ignore",
61+
"sample-setup": "npm run build && node ./dist/lib/test/typescript/manual/sample-setup.js",
5862
"watch": "npm-watch"
5963
},
6064
"dependencies": {
@@ -69,13 +73,15 @@
6973
"uuid": "10.0.0"
7074
},
7175
"devDependencies": {
76+
"@hyperledger/cactus-cmd-api-server": "2.0.0-rc.3",
7277
"@hyperledger/cactus-plugin-keychain-memory": "2.0.0-rc.3",
7378
"@hyperledger/cactus-test-tooling": "2.0.0-rc.3",
7479
"@openapitools/openapi-generator-cli": "2.7.0",
7580
"@types/express": "4.17.21",
7681
"@types/pg": "8.6.5",
7782
"body-parser": "1.20.2",
7883
"express": "4.19.2",
84+
"fabric-network": "2.5.0-snapshot.23",
7985
"jest-extended": "4.0.1",
8086
"rxjs": "7.8.1",
8187
"socket.io": "4.6.2"

packages/cactus-plugin-persistence-fabric/src/main/sql/schema.sql

+1-3
Original file line numberDiff line numberDiff line change
@@ -112,10 +112,8 @@ ALTER TABLE fabric.transaction OWNER TO postgres;
112112

113113
ALTER TABLE ONLY fabric.transaction
114114
ADD CONSTRAINT transaction_pkey PRIMARY KEY (id);
115-
ALTER TABLE ONLY fabric.transaction
116-
ADD CONSTRAINT transaction_hash_key UNIQUE (hash);
117115

118-
CREATE UNIQUE INDEX transaction_hash_unique_idx ON fabric.transaction USING btree (hash);
116+
CREATE INDEX transaction_hash_idx ON fabric.transaction (hash);
119117
--
120118
-- Name: transaction_action; Type: TABLE; Schema: fabric; Owner: postgres
121119
--

packages/cactus-plugin-persistence-fabric/src/main/typescript/db-client/db-client.ts

+5-2
Original file line numberDiff line numberDiff line change
@@ -200,15 +200,18 @@ export default class PostgresDatabaseClient {
200200
* @returns Map of cert attributes
201201
*/
202202
private certificateAttrsStringToMap(attrString: string): Map<string, string> {
203+
const separatorSplitRegex = new RegExp(`[/,+;\n]`);
204+
203205
return new Map(
204-
attrString.split("\n").map((a) => {
206+
attrString.split(separatorSplitRegex).map((a) => {
205207
const splitAttrs = a.split("=");
206208
if (splitAttrs.length !== 2) {
207209
throw new Error(
208210
`Invalid certificate attribute string: ${attrString}`,
209211
);
210212
}
211-
return splitAttrs as [string, string];
213+
const [key, value] = splitAttrs;
214+
return [key.trim(), value.trim()];
212215
}),
213216
);
214217
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
/**
2+
* Common setup code for the persistence plugin with detailed comments on each step.
3+
* Requires environment variable `SUPABASE_CONNECTION_STRING` to be set before running the script that includes this!
4+
* If not provided, a localhost instance of supabase will be assumed.
5+
*/
6+
7+
import process from "process";
8+
import { v4 as uuidV4 } from "uuid";
9+
import { PluginRegistry } from "@hyperledger/cactus-core";
10+
import { PluginKeychainMemory } from "@hyperledger/cactus-plugin-keychain-memory";
11+
import {
12+
LogLevelDesc,
13+
LoggerProvider,
14+
Logger,
15+
} from "@hyperledger/cactus-common";
16+
import { Configuration } from "@hyperledger/cactus-core-api";
17+
import {
18+
ApiServer,
19+
AuthorizationProtocol,
20+
ConfigService,
21+
} from "@hyperledger/cactus-cmd-api-server";
22+
import { DiscoveryOptions, X509Identity } from "fabric-network";
23+
import {
24+
DefaultEventHandlerStrategy,
25+
FabricApiClient,
26+
PluginLedgerConnectorFabric,
27+
} from "@hyperledger/cactus-plugin-ledger-connector-fabric";
28+
29+
import { PluginPersistenceFabric } from "../../../main/typescript";
30+
31+
//////////////////////////////////
32+
// Constants
33+
//////////////////////////////////
34+
35+
const SUPABASE_CONNECTION_STRING =
36+
process.env.SUPABASE_CONNECTION_STRING ??
37+
"postgresql://postgres:[email protected]:5432/postgres";
38+
39+
const testLogLevel: LogLevelDesc = "info";
40+
const sutLogLevel: LogLevelDesc = "info";
41+
42+
// Logger setup
43+
const log: Logger = LoggerProvider.getOrCreate({
44+
label: "common-setup-methods",
45+
level: testLogLevel,
46+
});
47+
48+
/**
49+
* Common ApiServer instance, can be empty if setup was not called yet!
50+
*/
51+
let apiServer: ApiServer;
52+
53+
//////////////////////////////////
54+
// Methods
55+
//////////////////////////////////
56+
57+
/**
58+
* Setup Cacti ApiServer instance containing Fabric Connector plugin (for accessing the fabric ledger)
59+
* and Fabric Persistence plugin (for storing data read from ledger to the database).
60+
*
61+
* @param port Port under which an ApiServer will be started. Can't be 0.
62+
* @param channelName Channel that we want to connect to.
63+
* @param connectionProfile Fabric connection profile (JSON object, not a string!)
64+
* @param userIdentity Signing identity to use to connect to the channel (object, not a string!)
65+
*
66+
* @returns `{ persistence, apiClient, signingCredential }`
67+
*/
68+
export async function setupApiServer(
69+
port: number,
70+
channelName: string,
71+
connectionProfile: any,
72+
userIdentity: X509Identity,
73+
) {
74+
// PluginLedgerConnectorFabric requires a keychain plugin to operate correctly, ensuring secure data storage.
75+
// We will store our userIdentity in it.
76+
// For testing and debugging purposes, we use PluginKeychainMemory, which stores all secrets in memory (remember: this is not secure!).
77+
const keychainId = uuidV4();
78+
const keychainEntryKey = "monitorUser";
79+
const keychainPlugin = new PluginKeychainMemory({
80+
instanceId: uuidV4(),
81+
keychainId,
82+
backend: new Map([[keychainEntryKey, JSON.stringify(userIdentity)]]),
83+
logLevel: testLogLevel,
84+
});
85+
const signingCredential = {
86+
keychainId,
87+
keychainRef: keychainEntryKey,
88+
};
89+
90+
// We create fabric connector instance with some default settings assumed.
91+
const discoveryOptions: DiscoveryOptions = {
92+
enabled: true,
93+
asLocalhost: true,
94+
};
95+
const connector = new PluginLedgerConnectorFabric({
96+
instanceId: uuidV4(),
97+
pluginRegistry: new PluginRegistry({ plugins: [keychainPlugin] }),
98+
sshConfig: {},
99+
cliContainerEnv: {},
100+
peerBinary: "/fabric-samples/bin/peer",
101+
logLevel: sutLogLevel,
102+
connectionProfile,
103+
discoveryOptions,
104+
eventHandlerOptions: {
105+
strategy: DefaultEventHandlerStrategy.NetworkScopeAnyfortx,
106+
commitTimeout: 300,
107+
},
108+
});
109+
110+
// Remember to initialize a plugin
111+
await connector.onPluginInit();
112+
113+
// We need an `FabricApiClient` to access `PluginLedgerConnectorFabric` methods from our `PluginPersistenceFabric`.
114+
const apiConfig = new Configuration({ basePath: `http://127.0.0.1:${port}` });
115+
const apiClient = new FabricApiClient(apiConfig);
116+
117+
// We create persistence plugin, it will read data from fabric ledger through `apiClient` we've just created,
118+
// and push it to PostgreSQL database accessed by it's SUPABASE_CONNECTION_STRING (read from the environment variable).
119+
const persistence = new PluginPersistenceFabric({
120+
channelName,
121+
gatewayOptions: {
122+
identity: signingCredential.keychainRef,
123+
wallet: {
124+
keychain: signingCredential,
125+
},
126+
},
127+
apiClient,
128+
logLevel: sutLogLevel,
129+
instanceId: uuidV4(),
130+
connectionString: SUPABASE_CONNECTION_STRING,
131+
});
132+
// Plugin initialization will check connection to the database and setup schema if needed.
133+
await persistence.onPluginInit();
134+
135+
// The API Server is a common "container" service that manages our plugins (connector and persistence).
136+
// We use a sample configuration with most security measures disabled for simplicity.
137+
log.info("Create ApiServer...");
138+
const configService = new ConfigService();
139+
const cactusApiServerOptions = await configService.newExampleConfig();
140+
cactusApiServerOptions.authorizationProtocol = AuthorizationProtocol.NONE;
141+
cactusApiServerOptions.configFile = "";
142+
cactusApiServerOptions.apiCorsDomainCsv = "*";
143+
cactusApiServerOptions.apiTlsEnabled = false;
144+
cactusApiServerOptions.apiPort = port;
145+
const config = await configService.newExampleConfigConvict(
146+
cactusApiServerOptions,
147+
);
148+
149+
apiServer = new ApiServer({
150+
config: config.getProperties(),
151+
pluginRegistry: new PluginRegistry({ plugins: [connector, persistence] }),
152+
});
153+
154+
const apiServerStartOut = await apiServer.start();
155+
log.debug(`apiServerStartOut:`, apiServerStartOut);
156+
// Our setup is operational now!
157+
158+
return { persistence, apiClient, signingCredential };
159+
}
160+
161+
/**
162+
* Cleanup all the resources allocated by our Api Server.
163+
* Remember to call it before exiting!
164+
*/
165+
export async function cleanupApiServer() {
166+
log.info("cleanupApiServer called.");
167+
168+
if (apiServer) {
169+
await apiServer.shutdown();
170+
}
171+
}

0 commit comments

Comments
 (0)