test(e2e): wait for CRDs

[jakubdyszkiewicz]

Motivation

Very rarely we see a flake on the CI that looks like this

2024-10-03T13:54:30.911Z	ERROR	kuma-cp.run	problem running Control Plane	{"error": "error running Kubernetes Manager: no matches for kind \"MeshCircuitBreaker\" in version \"kuma.io/v1alpha1\"", "errorVerbose": "no matches for kind \"MeshCircuitBreaker\" in version \"kuma.io/v1alpha1\"\nerror running Kubernetes Manager\ngithub.jpy.wang/kumahq/kuma/pkg/plugins/bootstrap/k8s.(*kubeComponentManager).Start\n\tgithub.jpy.wang/kumahq/kuma/pkg/plugins/bootstrap/k8s/plugin.go:208\ngithub.jpy.wang/kumahq/kuma/app/kuma-cp/cmd.newRunCmdWithOpts.func1\n\tgithub.jpy.wang/kumahq/kuma/app/kuma-cp/cmd/run.go:170\ngithub.jpy.wang/spf13/cobra.(*Command).execute\n\tgithub.jpy.wang/spf13/[email protected]/command.go:985\ngithub.jpy.wang/spf13/cobra.(*Command).ExecuteC\n\tgithub.jpy.wang/spf13/[email protected]/command.go:1117\ngithub.jpy.wang/spf13/cobra.(*Command).Execute\n\tgithub.jpy.wang/spf13/[email protected]/command.go:1041\ngithub.jpy.wang/kumahq/kuma/app/kuma-cp/cmd.Execute\n\tgithub.jpy.wang/kumahq/kuma/app/kuma-cp/cmd/root.go:83\nmain.main\n\tgithub.jpy.wang/kumahq/kuma/app/kuma-cp/main.go:6\nruntime.main\n\truntime/proc.go:272\nruntime.goexit\n\truntime/asm_arm64.s:1223"}

While it does not mess with any test, it causes CP to restart on which we have assertion after all tests are run.

Here is an example https://github.com/Kong/kong-mesh/actions/runs/11144879402/job/30973956156

We may be starting CP too soon. When we deploy using kumactl we install CRDs and CP deployment at the same time.

Implementation information

The solution is to first install CRD, wait for them using kumactl and only then install the CP.

I did not fix it for HELM because HELM installs CRDs via hook so there is much more time between CP installation and CP start, I don't think we ever saw this problem with HELM.

I checked that it takes about ~3s to wait for all CRDs so it's not a big overhead.

Supporting documentation

No issue, just a flake that we noticed