Support geo-replication

[prashanthpai]

[aravindavk]

Details of Geo-replication plugin implementation

Create command

Existing,

• Verify Slave Volume version, Space available in Slave Volume etc
• Record session creation information in VOLINFO file in all peer nodes
• Enables "changelog" and "index" features in Master Volume
• Creates config file in all Master nodes
• Creates session directory in /var/lib/glusterd/geo-replication/ and creates
  monitor.status file inside that directory.
• Pushes ssh keys of master nodes to Slave nodes and add to authorized_keys file
  in Slave nodes.

With Glusterd2,

• Verification yet to be decided(Since this involves reaching Slave Cluster)
• Record session creation information in etcd store.
• Enables "changelog" and "index" features using Volume Set APIs
• Session directory and monitor status file not required since the status can be
  maintained in etcd. Geo-rep monitor process can create session directory when
  started.
• Push ssh keys APIs yet to be decided.

Note: Verification, SSH Key generation and distribution steps are manual now.

Start Command

Existing,

• Start Geo-rep monitor process in all peer nodes and update monitor status as
  "Started"

With Glusterd2,

• Register Geo-rep monitor as enabled daemon in Glusterd2(with required args)
• Start the daemon in all master nodes(No need to start in all peer nodes)

Stop Command

Existing,

• Stop the Geo-rep session processes by sending Kill command in all peer nodes

With Glusterd2,

• Stop the Geo-rep session daemon using glusterd2 daemon management infra
• Unregister the session daemon from Glusterd2 store. (This is required to avoid
  automatically starting georep processes on glusterd2 restart)

Pause and Resume Command

Existing,

• Send SIGSTOP to Geo-rep processes running in all peer nodes
• Store the Status as Paused

With Glusterd2,

• Send SIGSTOP to Geo-rep daemons running in all Master volume nodes
• Store the status as Paused (This is required to get same state when geo-rep
  processes restarts or node reboot)

Note: SIGCONT for Resume

Delete Command

Existing,

• Delete session directory from all peer nodes
• Delete info from VOLINFO file

With Glusterd2,

• Delete session details from glusterd2 etcd store

Status Command

Existing,

• Collect Status from every master node, parse and show.

With Glusterd2, no change in behavior.

Config Set and Reset Command

Existing,

• Send Config change command to all peer nodes
• Restart/Reload Geo-rep processes in all peer nodes

With Glusterd2,

• Update config changes only in etcd
• Restart/Reload Geo-rep processes only in Master Volume nodes

Config Get Command

Existing,

• Get Config values from local Geo-rep process

With Glusterd2,

• Get config values from glusterd2 etcd store

Other challenges

Configuration file

Similar to volfile generation, config file generation is required in all Master
volume nodes when,

• A Geo-rep session is created
• A Geo-rep session config is set/reset

Also Geo-rep needs to expose the list of default options so that, Glusterd2
can consume them and does validation while config set/reset

SSH Keys distribution to Slave nodes

Option 1:

Existing Geo-rep is using hook script to push the SSH keys to Slave, with
Glusterd2 Slave can expose an API to receive the SSH Keys and distribute to its
nodes.

Note: Glusterd2 of Master Volume's cluster will not have access to this Slave
cluster's API. I think CLI can communicate with both the cluster it has to call
GET /geo-replication/ssh-keys from master cluster and call POST
/geo-replication/ssh-keys to distribute the keys.

Option 2:
Do this step manually using external tools like gdeploy which will have access
to both Master and Slave Clusters.

[aravindavk]

Initial patch(#349) talks about the APIs which look similar to existing CLI design. But changing the API design to use Volume IDs will help to avoid lot of
validations and assumptions during session create and recreate. Except Create command, all the other commands can support both the APIs(ID and name based)

Session Create API:

 PUT /geo-replication/{masterid}/{slaveid}

Request JSON:

    {
            "mastervol": MASTERVOL,
            "slavevol": SLAVEVOL,
            "slaveuser": SLAVEUSER",
            "slavehosts": []
    }

Session Update API:

 POST /geo-replication/{masterid}/{slaveid}

Request JSON:

    {
            "mastervol": MASTERVOL,
            "slavevol": SLAVEVOL,
            "slaveuser": SLAVEUSER",
            "slavehosts": []
    }

Session Start/Stop/Pause/Resume API:

POST   /geo-replication/{masterid}/{slaveid}/{action}

# Name based API, any one slavehost can be used to perform the action,
# also note Slave User name is not required.
POST   /geo-replication/{mastervol}/{slavehost}/{slavevol}/{action}

Session Delete API:

DELETE /geo-replication/{masterid}/{slaveid}

# Name based API, any one slavehost can be used to perform the action,
# also note Slave User name is not required.
DELETE /geo-replication/{mastervol}/{slavehost}/{slavevol}/{action}

Config Get, Set and Reset APIs:

GET    /geo-replication/{masterid}/{slaveid}/config
GET    /geo-replication/{mastervol}/{slavehost}/{slavevol}/config

POST   /geo-replication/{masterid}/{slaveid}/config
POST   /geo-replication/{mastervol}/{slavehost}/{slavevol}/config

DELETE /geo-replication/{masterid}/{slaveid}/config
DELETE /geo-replication/{mastervol}/{slavehost}/{slavevol}/config

Session Status API:

# Get All Sessions details
GET    /geo-replication

# Get all sessions of one Master Volume
GET    /geo-replication/{masterid}
GET    /geo-replication/{mastervol}

# Get details of one session
GET    /geo-replication/{masterid}/{slaveid}
GET    /geo-replication/{mastervol}/{slavehost}/{slavevol}

Checkpoint Get, Set and Reset APIs:

GET    /geo-replication/{masterid}/{slaveid}/checkpoint
GET    /geo-replication/{mastervol}/{slavehost}/{slavevol}/checkpoint

POST   /geo-replication/{masterid}/{slaveid}/checkpoint
POST   /geo-replication/{mastervol}/{slavehost}/{slavevol}/checkpoint

DELETE /geo-replication/{masterid}/{slaveid}/checkpoint
DELETE /geo-replication/{mastervol}/{slavehost}/{slavevol}/checkpoint

Existing CLI commands will not change, it will use the available APIs to make Geo-rep REST API call.

[aravindavk]

To implement push-pem functionality with glusterd2, we need to expose the following APIs and CLI will act as intermediary to communicate to both cluster.

API to generate SSH keys in all master volume nodes and return public keys.

POST /geo-replication/{volname}/ssh-keys
GET   /geo-replication/{volname}/ssh-keys

API to accept SSH public keys and to distribute to all all its nodes(Slave Volume nodes)

POST /geo-replication/{volname}/ssh-keys/push   # volname is slave volume name

CLI will run API in Master cluster to generate the keys and second API in Slave Cluster to upload and distribute the keys.

[atinmu]

Checklist

• ReST APIs availibility in wiki
• CLI commands
• Sufficient logging
• Availibility of metrics data if applicable
• Xlator options table upto date?
• Test cases & test coverage of the functionality