KEP: clusteradm operator

[TylerGillson]

Proposal for wrapping clusteradm with a Kubernetes controller that reconciles a MultiCluster custom resource, providing declarative lifecycle management for OCM multi-clusters.

Note: the controller already exists and I am proposing that Spectro Cloud donates it to the OCM ecosystem. I spoke to @mikeshng about this briefly in Slack. My team and I wrote and are currently maintaining this controller, as we're using it within a product that we're developing at Spectro Cloud. I would be happy to continue maintaining it if/when it is adopted by OCM.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: TylerGillson
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[qiujian16]

clusteradm is the client tool so it meant to be imperative. But you can use operator and API ClusterManager/Klusterlet without clusteradm at all.

[mikeshng]

Right. I think the proposed API and implementation outlined in this enhancement PR can still land in the clusteradm repo.

[qiujian16]

this is very similar as what we did with cluster-api https://github.com/open-cluster-management-io/ocm/tree/main/pkg/registration/hub/importer/providers. And in ACM, we have a way to config klusterlet https://github.com/stolostron/cluster-lifecycle-api/blob/main/klusterletconfig/v1alpha1/types.go on the hub.

[qiujian16]

what happens if a user wants to add or delete a managedcluster? Can they directly create/delete a managedcluster resource?

[TylerGillson]

Managed clusters can be added/removed by modifying the MultiCluster. The controller will execute the appropriate clusteradm commands to achieve the desired result, e.g. clusteradm join, clusteradm unjoin (against the spoke), clusteradm accept (against the hub), etc.

[qiujian16]

I think we also iterate on some alternatives. For example:

User can set a global or per cluster configs, so when the user create a managedcluster, the agent will be deployed to the target cluster automatically.

[TylerGillson]

The primary objective of the "clusteradm operator" is to bootstrap all required OCM components on both the hub and spokes. What you've mentioned sounds to me like it relies on OCM operators already being installed and running.

[mikeshng]

@qiujian16 is referring to the ACM pattern for importing an existing cluster. There's an import controller that watches for the creation of a ManagedCluster, along with its namespace and a way to connect to the spoke cluster (like a kubeconfig Secret). Once those are in place, it can connect to the managed cluster, deploy the klusterlet operator, and create the Klusterlet CR to initiate the registration process.

[qiujian16]

I mean you will still need to start "clusteradm operator" on the hub/spoke, and still need to handle the version and lifecycle of this operator. It is not clear to me why user cannot just install ocm operators directly. It would need some clarification.

[TylerGillson]

Everything you said is true. The value add that the proposed operator provides is around consolidation and minimizing manual steps. Users would have a single cluster with network connectivity to all other clusters that acts similarly to a CAPI management cluster. Say you have 20 clusters that you want to link together. With this approach you would only need to install the multi-cluster operator on one cluster and apply a single CR.

[qiujian16]

the goal is very similar as what we did on capi-importer work https://github.com/open-cluster-management-io/ocm/tree/main/pkg/registration/hub/importer. The difference is the join happens when a managedcluster is created on the hub. Would you also take a look at to see if the existing approach is good enough and can be extended to support your goal?

[TylerGillson]

Thanks for sharing that @qiujian16. A generic import provider that enqueues import requests when kubeconfig secrets with a certain annotation are created could be a valid alternative.

The biggest concern I have with that is that the registration controller image will become very large once support for arbitrary managed cluster kubeconfigs is added (due to aws-iam-authenticator, gcloud, python, etc.). The cluster API kubeconfigs all use a token, but requiring a long-lived token for all spoke cluster kubeconfigs is a major limitation. Many orgs will be unwilling to take that approach for security reasons.

By splitting the proposed functionality into a separate, optional operator, we can avoid blowing up the size of the registration controller. Also, by using clusteradm within the proposed operator, we avoid code duplication. I can see that the importer is already doing some of the same things that occur during clusteradm join.

[qiujian16]

is this necessary? I think user can just apply ClusterManager resource.

[TylerGillson]

That assumes that the OCM hub operators are already installed to reconcile the ClusterManager. How would the OCM hub components be initialized in the scenario you're describing?

By running clusteradm init, the OCM hub components and installed and the ClusterManager is created.

[mikeshng]

Users will need to install this clusteradm operator and API as the first step. Would you mind adding a bit more detail about how that installation would work in the KEP? For example, will it be via a Helm chart, a new clusteradm subcommand, or something else?

[TylerGillson]

The operator we've developed is currently installed via Helm. We could certainly extend clusteradm to support installing that Helm chart as well. Maybe, clusteradm fleet-operator-init? Any command name proposals?

[mikeshng]

Helm chart sounds good. We probably don't need the clusteradm sub-command for a while.