Adds NetworkPolicies to all components of Kube-prometheus

[ArthurSens]

Description

This is continuing the work started at #1470.

It adds NetworkPolicies to all components of the stack:

• Egress is allowed for all just for getting NetworkPolicies in quickly - We might want to revisit and be more restrict here
• Ingress only allowed by Prometheus for all components, except Prometheus-adapter (HPAs need to send requests for metrics)

Extra changes to make sure we keep supporting NetworkPolicies without breaking anything:

• CI was updated to disable default CNI and is installing Cilium to enable NetworkPolicies
• Scripts used for development were also updated to enable NetworkPolicies with Cilium

Fixes #1596

Type of change

What type of changes does your code introduce to the kube-prometheus? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. Later this will be copied to the changelog file.

Add NetworkPolicies to follow security best-practices

[ArthurSens]

Hmmm strange, this same setup worked when running locally 🤔
I'll try to take a look another day

[ArthurSens]

I'm not really sure what is wrong here, testing locally works...

@prometheus-operator/kube-prometheus-reviewers any idea why e2e tests doesn't work with NetworkPolicies during CI?

[paulfantom]

Does it work locally?

[ArthurSens]

yes it does 🤔

[ArthurSens]

Alright, after @simonpasquier's suggestions, I've created another branch that only adds Calico to the CI just to make sure that it wasn't the problem here.

Turns out that only by adding Calico, and not adding any network policy, our CI already fails. I'm not that knowledgeable about Kubernetes Networking, so I still haven't figured out why that breaks the CI.

[philipgough]

@ArthurSens - I did take a look. Also no clue right now why it fails on CI with Calico installed. FWIW I checked out your branch and ran the tests locally on OS X and they passed too :/

[ArthurSens]

OKEY! Replacing Calico with Cilium did the trick 🚀

I believe this PR is ready for review now :)

[philipgough]

@ArthurSens - great job, looks good to me overall but I wonder if we should update the quickstart guide also?

[ArthurSens]

@ArthurSens - great job, looks good to me overall but I wonder if we should update the quickstart guide also?

Good point! I'm not sure if that's needed though...

Network policies should be a no-op on Kubernetes without a plugin with support for it, so this should work out-of-the-box. I'm not entirely sure we want to add another step to the quickstart guide to enable this. I think quickstart should have the less amount of friction possible 🤔

[philipgough]

@ArthurSens - thanks for the link, I wasn't aware of the no-op situation and in that case it make perfect sense to keep the quickstart as simple as possible.

All good my side then. Will wait for some eyes from @paulfantom /others

[ArthurSens]

I'm merging since there were no oppositions so far 🙂

[werdnum]

FWIW, this seems to have completely broken my relatively ordinary setup on k3s (specifically, it broke my ingress rules) and I had to go digging to find out why. There doesn't seem to be a clear way to enable ingress (even to Grafana) without completely deleting the network policies. I'd hope you could add at least an escape hatch in 'values' to allow adding the ingress controller to the network policy.

[SISheogorath]

@werdnum you have to do the opposite: You have to create a NetworkPolicy that allows access from your ingress (controller) Pods to the Grafana Pod, deleting all NetworkPolicies removes any access restrictions for this namespace.

[werdnum]

@SISheogorath:

@werdnum you have to do the opposite: You have to create a NetworkPolicy that allows access from your ingress (controller) Pods to the Grafana Pod, deleting all NetworkPolicies removes any access restrictions for this namespace.

Is there some kind of affordance in the templates to do so? Ideally there's something in values that allows me to put in a PodSelector value that matches the ingress controller, instead of me having to learn what a NetworkPolicy is.

Seems like this should have been marked as a breaking change.

[roelandvanbatenburg]

@SISheogorath:

@werdnum you have to do the opposite: You have to create a NetworkPolicy that allows access from your ingress (controller) Pods to the Grafana Pod, deleting all NetworkPolicies removes any access restrictions for this namespace.

Is there some kind of affordance in the templates to do so? Ideally there's something in values that allows me to put in a PodSelector value that matches the ingress controller, instead of me having to learn what a NetworkPolicy is.

Seems like this should have been marked as a breaking change.

I did some digging and provided an example of how to modify the NetworkPolicies in #1960. Not quite what you were asking for, but I have little experience with Helm.