chore(docs): generate docs/monitoring/metrics.md file

[ivankatliarchuk]

Description

No issue. On one of the proposals, there were mentioning about metrics #5079 (comment). So I've tried to understand how metrics/observability is configured. Turns out, that there were no page about that, only FAQ with outdated data.

This pull request is very similar to #4983

It adds

• generation of all available metrics to a document
• monitoring documentation
• added debug logs for metric instrumentation

This PR is massive, I'm could split it if required, just need a guidance how to slice it.

Checklist

• Unit tests updated
• End user documentation updated

[ivankatliarchuk]

/label tide/merge-method-squash

[mloiseleur]

Overall PR LGTM.
Wdyt of moving the template in a separate file ?

It's the case for the chart and it's clearly easier to review & update.

[ivankatliarchuk]

Changed to a template file.

[mloiseleur]

Would you please add CI on this ? (like for flags)
On my system, it generates two additional metrics:

diff --git a/docs/monitoring/metrics.md b/docs/monitoring/metrics.md
index 6c7782af..effbd242 100644
--- a/docs/monitoring/metrics.md
+++ b/docs/monitoring/metrics.md
@@ -80,6 +80,8 @@ curl https://localhost:7979/metrics
 | http_request_duration_seconds |
 | process_cpu_seconds_total |
 | process_max_fds |
+| process_network_receive_bytes_total |
+| process_network_transmit_bytes_total |
 | process_open_fds |
 | process_resident_memory_bytes |
 | process_start_time_seconds |

[ivankatliarchuk]

The Go runtime's behavior differs considerably across systems due to factors like Docker containerization, the host operating system, and architecture (ARM, AMD64, etc.). This inconsistency is why its tests are disabled. We can either add them manually or remove them entirely.

The following Go runtime metrics are available for scraping. Please note that they may change over time and they are OS dependent.

[ivankatliarchuk]

process_network_receive_bytes_total

+| process_network_receive_bytes_total |
+| process_network_transmit_bytes_total |

I'm unsure how to catch them from within a code, they are only available when promhttp.Handler() is activated.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from mloiseleur. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ivankatliarchuk]

I've added manually this two metrics. It's a best effort at the moment.

[ivankatliarchuk]

Current behaviour for CI tests

• Will fail if new custom metric is added to a code but not to a file
• Will ingore if Go adds/removes runtime metrics. So no CI tests to fail in such case

[ivankatliarchuk]

Need guidance here.