Configure Envoy Proxy fleet in different namespaces

[arkodg]

          thinking out loud, potential API could look like

deploymentMode:
  watch:
    namespaces: .........
  deploy:
    envoy:
      namespaceType: GatewayNamespace  

• By default (when this field is unset), EG watches all resources in all namespaces, and depoys infra in the namespace where EG is running (ControllerNamespace)
• watch.namespaces can be used to set the specific namespaces to watch resources (Gateway, xRoute, Service etc)
• deploy.envoy.namespaceType can be used to specify to EG to deploy proxy infra to deploy envoy deployments in the namespace where the Gateway is created in. This option will need a helm knob too, to configure added permissions to be able to create deployments and services in other namespaces

Originally posted by @arkodg in #1117 (comment)

[cnvergence]

I would like to tackle this one

[cnvergence]

It will also be necessary to move away from mTLS for control plane auth to sTLS + JWT Authn to validate the envoy.
cc @arkodg

[arkodg]

yah @cnvergence, we can selectively do it for the case when the Gateway is running in a user namespace

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[kamalmarhubi]

I believe this would be solved once GEP-1762: In Cluster Gateway Deployments is implemented, which is tracked in #4330 as referenced just above. Or I know at least the use case that led me to this issue would be solved by it.

[arkodg]

@kamalmarhubi can you share your use case ?
we already support customizing gateway infrastructure https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/, the generated resources live in the envoy-gateway-system namespace, instead of the Gateway namespace

[kamalmarhubi]

@arkodg thanks yes, I'm already customizing it!

The reason I'd like the resources to be created in the Gateway namespace is that it allows cleaner RBAC policies. I'd like owners to be able to inspect their gateways' generated resources without necessarily giving them access to the gateway controller itself.

It also makes setting up network policies easier: with a default-deny setup, we need to add allow-egress policies in the envoy-gateway-system which seems messy. It also makes RBAC less clean: gateway owners either have to ask someone with appropriate permissions in envoy-gateway-system to do this for them, or we need to grant all gateway owners netpol management permissions in envoy-gateway-system. Neither of those options are ideal.

Do you expect that Envoy Gateway will implement GEP-1762?

[arkodg]

the inability to target the generated resources when creating a resource like network policy, is a fair point @kamalmarhubi
Envoy Gateway already implements most of GEP-1762, we would need to implement this issue to completely implement the GEP

[kamalmarhubi]

@arkodg while continuing to deploy envoy-gateway, I came across another reason to want to run envoy in the Gateway's namespace: cert-manager has an integration with the Gateway API and can create certificates for annotated Gateway objects.

However, the certificates it creates (and so the TLS secrets) live in the Gateway's namespace, and that's a limitation they're unlikely to attempt to lift because ownerReferences don't work across namespaces (cert-manager/cert-manager#5610). Since workloads can't reference secrets from other namespaces, the envoy deployment or daemonset wouldn't be able to read those secrets.

It's easy enough to manually create the required certificates, but the cert-manager integration would be a handy shorthand, and would also make it harder to mess up by letting the certificate's DNS names and Gateway's TLS listeners diverge.

edit: Never mind I think I was incorrect; this is documented on the Envoy gateway website.

[cnvergence]

Hey all, slowly returning back to speed, I should have some cycles to work again on EG
Looking to move this one forward

[arkodg]

thanks @cnvergence, thinking out loud you should be able to use the k8s service account token and append it the request, may need changes to the xds cluster , and perform sTLS + JWT Validation in the xds server for this case / if token exists

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[illrill]

Enthusiastic about this and GEP-1762 / #4330. In addition to the already mentioned use cases, running the Proxy in the same namespace as the Gateway would allow for a really clean a microgateway pattern type of architecture to support east-west communication between applications internally in a cluster. Here's my use case:

As cluster admin, I could bundle a Gateway in each tenant namespace, configured to have Envoy Gateway Controller spawn an Envoy Proxy exposed by a ClusterIP Service with a predefined name like gateway. The different tenants in the cluster would then be able to talk to each other's Envoy Proxies using gateway.tenant-a.svc.cluster.local and gateway.tenant-b.svc.cluster.local, instead of talking directly to each other's application's services as they otherwise tend to do (introducing the need for a service mesh for security and traffic management). With the variety of traffic management and authentication capabilities that Envoy Gateway provides, we could effectively remove the need for a bulky service mesh.

Bonus: Tenants don't need to step outside their namespace boundary to view their proxy access logs.

WDYT @zhaohuabing @cnvergence, can we expect this in the near future (v1.3.0)? 🙌

[zhaohuabing]

Hi @illrill Thanks for sharing your use case! This issue has alredy been added to v1.3.0 milestone. Hopeflly, someone from the community will pick it up soon.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.