Deploying OPA in Kubernetes with a PersistentVolume and multiple Pods (or even a HPA)

[HenriBlacksmith]

Hello,

I am looking for information about deploying OPA in Kubernetes with a failure. (and update)-tolerant deployment.

For now we are using OPA in a deployment which looks like the official one: https://www.openpolicyagent.org/docs/latest/deployments/#kicking-the-tires

The issue is that we write data to OPA and evaluate policies which means that our OPA instance has dynamic data which must be shared between all OPA instances.

For now we use only one replica as documented but this is not failure-tolerant and adding more replicas cannot work as OPA does not seem to support a shared storage (with transactions or locks to allow a consistant update of the storage by multiple instances).

Is there a way to have a failure-tolerant deployment of OPA in Kubernetes (i.e. a shared storage and more than one Pod)?

[charlieegan3]

Generally we see users distributing data to OPA in an eventually consistent manner using the bundle service API. This is generally the most simple and robust option.

Is there a reason that eventually consistent updates for data won't work for you?

[HenriBlacksmith]

Hmm ok, I see, I will explore that option with my team

[HenriBlacksmith]

I mean, our goal is to rely on OPA only and not to create a dependency on another service to read data and we also need to be able to update data (so static data cannot be used either)

[anderseknert]

Can only +1 the recommendation of using a bundle server for this purpose. Running OPA in production and at scale almost always entails running one distributed component (OPA) and a centralized one (a bundle server or control plane) for management.

Things like policy and data updates are all accounted for in the bundle service API, and additionally you'll most likely want to use a centralized service for decision logging as well.

[ritazh]

I’m curious if you have tried Gatekeeper (multiple pod deployment of opa in k8s) and the external data feature for your data? https://open-policy-agent.github.io/gatekeeper/website/docs/externaldata

[HenriBlacksmith]

Is'nt Gatekeeper only to apply policies to Kubernetes resources?

My use-case is to use OPA for application permissions

[HenriBlacksmith]

Sorry for replying late, i seems that my team used a Bundle API in the past and we switched to a push mode (as opposed to a mode where OPA pulls the bundle) for performance/speed issues as there was a significant delay (a few seconds) between the bundle publication and its pull by OPA.

Another option I imagine is to push data to multiple instances synchronously but it does not look like the official way to go (that's what we do today but on a single instance/Pod)

[ashutosh-narkar]

Sorry for replying late, i seems that my team used a Bundle API in the past and we switched to a push mode (as opposed to a mode where OPA pulls the bundle) for performance/speed issues as there was a significant delay (a few seconds) between the bundle publication and its pull by OPA.

Specifically on this, have you tried a combination of Delta bundles with HTTP long polling? It may help with faster data propagation.

[Duanxinxin]

I also encountered the same problem, may I ask if there is any good solution in the end

[HenriBlacksmith]

I still think it could be nice to have a way to have standalone deployments of OPA where data updated via the HTTP API are persisted on a volume.

This would require to have an active/standby pattern as any stateful app in Kubernetes.

This would significantly reduce the delays due to polling the bundle.

[HenriBlacksmith]

Sorry for replying late, i seems that my team used a Bundle API in the past and we switched to a push mode (as opposed to a mode where OPA pulls the bundle) for performance/speed issues as there was a significant delay (a few seconds) between the bundle publication and its pull by OPA.

Another option I imagine is to push data to multiple instances synchronously but it does not look like the official way to go (that's what we do today but on a single instance/Pod)

I am wondering if there could be a way to have the best of push and pull worlds and have some kind of trigger system to update bundle in all pods when the remote bundle is updated?

We will continue looking into that but we still do not have any satisfying solution for now (except the single replica OPA with local data)

[ashutosh-narkar]

I am wondering if there could be a way to have the best of push and pull worlds and have some kind of trigger system to update bundle in all pods when the remote bundle is updated?

OPA can download bundles periodically or when manually triggered. The later however is only available when using OPA as a Go package atm. On the former have y'all tried a combination of long polling plus shorter pull frequency?

[hpvd]

Are there already thoughts on or a POC of using Kubernetes upcoming native support for "Read Only Volumes Based On OCI Artifacts" for delivering OPA policies?
see https://kubernetes.io/blog/2024/08/16/kubernetes-1-31-image-volume-source/

=> OPA policies look to me like a perfect usecase for this new feature.

edit:

the container runtime used also needs support for this.

• CRI-O support implemented Add OCI/Image Volume Source support cri-o/cri-o#8317
• Containerd support not implemented yet Add OCI Volume Source support containerd/containerd#10496

[charlieegan3]

Hey, this is not a feature I've tried but I think it should be supported out of the box in OPA. However, I would advise against going this route for most users as it requires a rollout to update the policy loaded into OPA (as it'd be a change in the k8s spec).

We already have OCI bundle support so I think this would be a better way to source policy if you're looking to use OCI as a mechanism to distribute bundles.

[hpvd]

many thanks for your opinion and advice!