kubectl apply is slow due to client side throttling

[Rick-van-Dam]

I constantly see these kind of messages when doing kubectl apply -f - --server-side --force-conflicts --prune --applyset=<applyset>:

Waited for 14.1888454s due to client-side throttling, not priority and fairness, request: GET:https:<url>

Likely this is due to having a decent amount of custom CRD's which causes me to hit the client side limit.

I tried to find a way to increase the client limits but these seem to be not configurable. My server can easily handle more (and if not I could simply add more powerful hardware). How to disable or increase this limit when using the cli? Having to wait is annoying and costly. I think its fine to have somewhat conservative limits by default but they should be configurable in a way.

[k8s-ci-robot]

This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ardaguclu]

Thanks for filing an issue. Limits are hardcoded in

kubectl/pkg/cmd/cmd.go

Line 97 in 1fe7110

return genericclioptions.NewConfigFlags(true).WithDeprecatedPasswordFlag().WithDiscoveryBurst(300).WithDiscoveryQPS(50.0)

. Incorrect configurations may overload the API server. So that these values are not configurable and I don't think, we'd want to change this behavior.

[Rick-van-Dam]

Everyone's situation is different so having this configurable would better fit everyones situation.

I don't think preventing ppl to overload the api server is a good reason to hardcode this. If someone really wants to he could just make a custom client to do so. A warning in the documentation with some recommended values should probably be enough.

Even better would be to apply some server side control of this actually by sending back a 429.

[mpuckett159]

I'll bring it up that the next sig-cli meeting, but because it is really easy to brick your cluster due to over loading the API server I don't think that we will move forward with making these options configurable.