Kubeless deployment on separate namespace inside same cluster

[hwa123]

What happened:
I have kubeless-0.5.0 up and running on kubeless namespace, everything works as expected there. When I'm deploying the same kubeless suite (kafka, zookeeper, kafka-trigger-controller, kubeless-manager-controller) on a different namespace, some weird behaviors were observed.

Kafka and Zookeeper seem to sync all function triggers and topics from kubeless namespace. I tried to use a namespace level custom role for this deployment but same thing happens again.

How can I make this deployment work on namespace level and completely independent from other kubeless resources?

What you expected to happen:

Having kubeless suites running independently on separate namespaces.

Environment:

• Kubernetes version (use kubectl version):
  oc v3.9.0+191fece
  kubernetes v1.9.1+a0ce1bc657
  features: Basic-Auth
  Server <internal_server>
  openshift v3.9.25
  kubernetes v1.9.1+a0ce1bc657
• Kubeless version (use kubeless version): kubeless v0.5.0
• Cloud provider or physical cluster: OpenShift v3.9.25

[gkarthiks]

@andresmgot please let us know if you need any further information.

[andresmgot]

Hi, we have some docs regarding running kubeless in a different namespace https://kubeless.io/docs/function-controller-configuration/#install-kubeless-in-different-namespace (but I am not sure if that is 100% complete).

I am not sure I am fully understanding the issue. The triggers CRD definition are not namespaced and the function triggers are created in the namespace of the function (e.g. default) not in the namespace of Kubeless. Can you post the commands you used to install kubeless or the steps to reproduce the issue?

[gkarthiks]

@andresmgot Thanks Andres, but we don't want to bind our CRD to a namespace. Because we are intending to have independent kubeless frameworks for different namespaces in the same cluster. Binding the CRD will not help in this scenario. We just followed the normal deployment minus the namespace=kubeless for deploying in our custom namespace.

[prydonius]

It sounds like what @gkarthiks is looking for is a way to run multiple instances of the kubeless and kafka controllers in multiple namespaces, and only have them watch Function and KafkaTrigger kinds in the namespace they are deployed in (i.e. similar to the Tiller-per-namespace model).

In order for Kubeless to support this, we could have a --watch-namespace flag (similar to the Nginx Ingress Controller) and if specified use that namespace when configuring the shared informer (https://github.com/kubeless/kubeless/blob/master/pkg/controller/function_controller.go#L79).

[andresmgot]

@gkarthiks we have added a new configuration property (functions-namespace) for that purpose. I've also written a piece of documentation that you can follow to achieve what you need:

https://github.com/kubeless/kubeless/blob/master/docs/function-controller-configuration.md#install-several-instances-of-kubeless-multi-tenancy

Would you mind to give it a try following the guide and let us know if you find any trouble? To test it you need to use the latest Kubeless manifest (not released yet). You can download it from here:

RBAC: https://1714-73902337-gh.circle-artifacts.com/0/home/circleci/.go_workspace/src/github.com/kubeless/kubeless/build-manifests/kubeless.yaml
Openshift:
https://1714-73902337-gh.circle-artifacts.com/0/home/circleci/.go_workspace/src/github.com/kubeless/kubeless/build-manifests/kubeless-openshift.yaml

[gkarthiks]

Thanks @andresmgot sure, let me try this.

[prydonius]

@andresmgot nice! FYI I think the other controllers (kafka, etc.) should be updated too.

[andresmgot]

FYI I think the other controllers (kafka, etc.) should be updated too.

Yes, they have been already updated :) (controllers with the tag v1.0.0-alpha.9 contains that feature)

[vladimirtiukhtin]

@andresmgot but it still requires clusterrole to be bind to kubeless SA, doesn't it? I guess the right approach is to bind to a role which automatically scopes SA to a specific namespace.
At the end of day what can stop me from altering configmap and extending kubeless to other namespaces if it is bind to a clusterrole