Skip to content

Add docs for accept multiple ha pairs #6364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions docs/guides/ha-pair-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,73 @@ Cortex will not return any data.

Therefore, the `__replica__` label should only be added for remote write.

## Accept multiple HA pairs in single request
Let's assume there are two teams (T1 and T2), and each team operates two Prometheus for the HA (T1.a, T1.b for T1 and
T2.a, T2.b for T2).
They want to operate another Prometheus, receiving whole Prometheus requests and sending write request to the
Distributor.

The write request flow is as follows: T1.a, T1.b, T2.a, T2.b -> Prometheus -> Distributor which means the Distributor's
incoming write request contains time series of T1.a, T1.b, T2.a, and T2.b.
In other words, there are two HA pairs in a single write request, and the expected push result is to accept each
Prometheus leader replicas (example: T1.a, T2.b for each team).

## Config
### Client side
The client setting is the same as a single HA pair.
For example:

For T1.a
```
cluster: prom-team1
__replica__: replica1 (or pod-name)
```

For T1.b

```
cluster: prom-team1
__replica__: replica2 (or pod-name)
```

For T2.a

```
cluster: prom-team2
__replica__: replica1 (or pod-name)
```

For T2.b

```
cluster: prom-team2
__replica__: replica2 (or pod-name)
```

### Server side

One additional setting is needed to accept multiple HA pairs; it is enabled via
`--experimental.distributor.ha-tracker.mixed-ha-samples=true` (or its YAML config option).

The following configuration snippet shows an example of accepting multiple HA pairs config via the YAML config file:

```yaml
limits:
...
accept_ha_samples: true
accept_mixed_ha_samples: true
...
distributor:
...
ha_tracker:
enable_ha_tracker: true
...
kvstore:
[ store: <string> | default = "consul" ]
[ consul | etcd: <config> ]
...
...
```

For further configuration file documentation, see
the [limits section](../configuration/config-file-reference.md#limits_config).
Loading