CNDB-13583: Add vector ann and brute force metrics

[michaeljmarshall]

What is the issue

https://github.com/riptano/cndb/issues/13583

What does this PR fix and why was it fixed

This PR adds comprehensive metrics for Storage Attached Indexes (SAI) vector search operations, providing crucial insights into both ANN (Approximate Nearest Neighbor) graph searches and brute force operations.

New Vector Search Metrics:

Search Operation Counters:

• ANNNodesVisited: Total number of nodes visited during ANN searches (this is equivalent to approximate similarity score computations)
• ANNNodesReranked: Number of nodes that underwent exact distance computation for reranking (this is equivalent to exact similarity score computations)
• ANNNodesExpanded: Total number of nodes whose edges were explored
• ANNNodesExpandedBaseLayer: Number of nodes expanded in the base layer of the graph
• ANNGraphSearches: Count of new graph searches initiated
• ANNGraphResumes: Count of resumed graph searches (when a search continues from previous results)
• ANNGraphSearchLatency: Timer measuring individual graph search latency (Note: This measures per-graph search time, not total query time which may involve multiple graphs)

Brute Force Operation Counters:

• BruteForceNodesVisited: Number of nodes visited during brute force searches (approximate similarity comparisons)
• BruteForceNodesReranked: Number of nodes that underwent exact similarity computation during brute force searches

Memory Usage Tracking:

• quantizationMemoryBytes: Current memory usage by the quantization (PQ or BQ) data structures
• ordinalsMapMemoryBytes: Current memory usage by ordinals mapping structures (only matters in some cases)
• onDiskGraphsCount: Number of currently loaded graph segments
• onDiskGraphVectorsCount: Total number of vectors in currently loaded graphs

These metrics will help us:

1. Understand if we are performing more comparisons than expected
2. Get insight into number of graphs queried
3. Get insight into the brute force vs graph query path
4. Understand current memory utilization

The counters provide operations/second metrics, allowing calculation of per-query averages by dividing by the number of queries. The memory tracking metrics help monitor resource usage across graph segments as they are loaded and unloaded.

[github-actions]

Checklist before you submit for review

• Make sure there is a PR in the CNDB project updating the Converged Cassandra version
• Use NoSpamLogger for log lines that may appear frequently in the logs
• Verify test results on Butler
• Test coverage for new/modified code is > 80%
• Proper code formatting
• Proper title for each commit staring with the project-issue number, like CNDB-1234
• Each commit has a meaningful description
• Each commit is not very long and contains related changes
• Renames, moves and reformatting are in distinct commits
• All new files should contain the DataStax copyright header instead of the Apache License one

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
95.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[cassci-bot]

✔️ Build ds-cassandra-pr-gate/PR-1683 approved by Butler

---

Approved by Butler
See build details here

[eolivelli]

+1