CNCI agent should be stateless

[markdryan]

Currently, the CNCI agent maintains some state in a boltdb database. Architecturally, it would be preferable if the CNCI agent maintained no state at all. Instead this information could be passed to the cnici agent by controller when it first starts. In the current implementation there is CNCI migration bug which needs to be addressed. The issue is as follows.

1. There is a cluster with one NN node.
2. The NN is evacuated.
3. Controller tries and fails to restart the CNCI so it is deleted.
4. The NN is restored.
5. Next time the user needs a CNCI, a new CNCI will be started on the network node, but the old state will have been lost.

Storing no state in the CNCI agent would fix this issue.

[markdryan]

@kaccardi, @mcastelino I'd like your opinion on this. We did discuss doing this but I don't remember whether we were all agreed that this was the best way forward. It's certainly my preference.

[mcastelino]

@markdryan it will not be very simple.

The controller has no idea about the placement of the subnets:nodes and the tunnels setup.

It would be better if we just created a persistent volume and attached it to the CNCI.
The CNCI code already allows for this. That would be consistent with the current design.

Also you cannot restart the CNCI with a different mac address/client ID. If you do, each and every tunnel setup will need to be torn down and rebuilt as the IP address of the CNCI would change.

If you want the CNCI to be perfectly stateless we would need to significantly change the entire design.

• Launcher should be capable of tearing down and setting up tunnels when a CNCI goes down
• Where there are multiple CNCI's, it should be able to do this based on the subnet (so it need to keep track of CNCI -> Subnet mappings)
• Controller should notify all launchers of the CNCI re-launch
• Also any public IP policies and firewall rules need to re-setup.

[mcastelino]

Some initial thoughts about this.

Current State:

• The CNCI maintains the following state
  o Topology: Location of all remote subnets (i.e. Subnet:CN IP)
  o Configuration: Public IP to Tenant IP Mappings

• In the future the CNCI will also maintain
  o CNCI Topology: CNCI IP:CNCI Subnet

• Of these three, the controller today is not aware of Topology.

Topology

• is generated when a new subnet is created on a CN
• The launcher (network library) sends a tenant add event to the CNCI IP
• The network library now has a Tenant Subnet -> CNCI mapping

Stateful part of Ciao Networking

• Ciao networking assumes that the CNCI IP does not change even if the CNCI is restarted
  o That is ensured by keeping the Client ID (i.e. the instance id) and mac address of the CNCI does not change
   It assumes that the DHCP server will allocate the same IP
  o This also assumes that the CNCI is restarted on the same physical network segment (as seen by the DHCP server)
• Key benefit of this design
  o If the CNCI is restarted it rebuilds the topology
   No changes to controller or to any of the remote nodes which have tunnels connecting to the CNCI
  • They will re-connect when the CNCI comes back up

Making the CNCI fully stateless

           This assumes that the CNCI loses its database

Approach 1: CNCI stateless but will retain the same client ID, mac and IP on restart

• The networking library detects when a CNCI is restarted and sends it the tenant added event on restart from all NNs

Advantages

• Controller already has the rest of the information, so all needs is to resend it
• Some work in the CNCI agent and CN Networking Library and Launcher
• Makes the current networking implementation a bit more resilient
• CNCI restart is faster, and workloads regain network connectivity instantly.

Disadvantages

• Introduces active threads in the Networking Library on the CNs

Approach 2: CNCI stateless but will retain the same client ID, mac. But the IP address is allowed to change

• The controller sends the CNCI it the Configuration and CNCI Topology when it sees a CNCI added event reported back
• The controller then sends the updated IP address to all CNs (or interested CNs)
• The networking library tears down existing tunnels, sets up new ones to the new CNCI IP and sends the new CNCI the tenant added events

Advantages

• CNCI is fully stateless.
• No active logic in the CN/Launcher/Networking library
• CNCI’s can finally migrate across physical network segments/DHCP servers (big advantage)
• Now the CNCI data base, disk etc can get corrupted, and we can fix it with a simple restart
  o If we do not want any data base in the CNCI, then even a agent restart should act like a CNCI restart

Disadvantages

• We need to add a broadcast method to the scheduler (may be simpler)
• Or
• the Controller will need some sort of directed multicast capability and controller needs to track network topology

[markdryan]

@mcastelino I'd like to clarify what exactly happens in scenario 1. Is the idea that the controller starts a new CNCI to replace the one that failed but that this CNCI would still be missing some information? You mention that the networking library will send Tenant Added events. Is this done by every compute node affected by the CNCI restart?

Introduces active threads in the Networking Library on the CNs

I think this is okay. We know how to do this in a clean way now.

Doesn't the advantage listed in Approach 2, about being able to recover from a corrupted database also apply to approach 1?