Description
Is your feature request related to a problem? Please describe.
The ingester_client_inflight_push_requests metric measures the number of requests a distributor client is sending to a single ingester, but this is a per-ingester local limit that does not measure how many total inflight requests a distributor is waiting on callback for. This limit can help buildup of inflight requests when there is more than 1 bad ingester that is not responding and causes distributor to hold onto the request.
Describe the solution you'd like
My proposal is to introduce a global limit on the number of inflight requests for distributor (among all the ingester clients) and increment that metric in send.
Describe alternatives you've considered
The cortex_distributor_inflight_push_requests measures the number of Push requests but those are API calls rather than the actual number of requests waiting on callback from ingester (which could be larger).
Additional context