Update matrix-stats aggregation. #9434

dwelsch-esi · 2025-03-14T19:25:23Z

Description

Updated descriptions. Added explanation of result statistics.

This PR will probably require some major revisions before it's published. I have several questions. See line comments.

Issues Resolved

Version

Frontend features

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Dave Welsch <[email protected]>

github-actions · 2025-03-14T19:25:35Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

dwelsch-esi · 2025-03-14T19:27:40Z

_aggregations/metric/matrix-stats.md

+| Parameter | Required/Optional | Data type      | Description |
+| :--       | :--               | :--            | :--         |
+| `fields`  | Required          | String         | An array of fields for which the matrix stats are computed. |
+| `mode`    | Optional          | String         | Which value to use as a sample from a multi-valued or array field. Allowed values are `avg`, `min`, `max`, `sum`, and `median`.  |


mode did not seem to affect the result when I tested it with array data. Maybe it doesn't work when one field is a single value and another is an array? But that doesn't make much sense.

What is the default mode?

@jainankitk Could you please answer this quiestion?

@dwelsch-esi - Can you share the query that you ran?

@jainankitk Try the following:

DELETE /students POST _bulk { "create": { "_index": "students", "_id": "1" } } { "name": "John Doe", "gpa": 3.89, "class_grades": [3.0, 3.9, 4.0], "grad_year": 2022} { "create": { "_index": "students", "_id": "2" } } { "name": "Jonathan Powers", "gpa": 3.85, "class_grades": [4.1, 3.0, 4.0], "grad_year": 2025 } { "create": { "_index": "students", "_id": "3" } } { "name": "Jane Doe", "gpa": 3.52, "class_grades": [3.2, 2.1, 3.8], "grad_year": 2024 } GET students/_search { "size": 0, "aggs": { "matrix_stats_taxful_total_price": { "matrix_stats": { "fields": ["gpa", "class_grades"], "mode": "avg" } } } }

I tried different modes (min, max) instead of avg. In all cases, the correlation between gpa and class_grades is the same, 0.9820867239098596. Surely it should be different if the value used for class_grades is computed differently for different mode values?

Thanks @dwelsch-esi for close observation. I noticed that the first response is cached resulting in this discrepancy. You can try using the GET students/_search?request_cache=false parameter to avoid this issue. Or alternatively, POST /_cache/clear before each search request

we have PR for fixing this discrepancy - opensearch-project/OpenSearch#18254, in case anyone is curious

_aggregations/metric/matrix-stats.md

dwelsch-esi · 2025-03-14T19:34:39Z

_aggregations/metric/matrix-stats.md

@@ -9,8 +9,23 @@ redirect_from:

 # Matrix stats aggregations

-The `matrix_stats` aggregation generates advanced stats for multiple fields in a matrix form.
-The following example returns advanced stats in a matrix form for the `taxful_total_price` and `products.base_price` fields:
+The `matrix_stats` metric is a multi-value metric that generates covariance statistics for two or more fields in matrix form. 


Does matrix-stats always sample every available document? If not, how is sampling done?

What assumptions does matrix-stats make about independence of data samples? Is it sufficent to say that it assumes documents are independent of each other?

Again will probably keep it as multi-value metric aggregation and maybe link to the metric aggregation page

@jainankitk Please see my technical questions throughout this file. Can you provide any insights?

@jainankitk Could you please answer this quiestion?

Does matrix-stats always sample every available document? If not, how is sampling done?

I don't think there is any sampling, all the documents are processed

What assumptions does matrix-stats make about independence of data samples? Is it sufficent to say that it assumes documents are independent of each other?

Matrix-stats does not make any assumptions about the data. I am wondering why does it need to make any assumption?

@jainankitk There are several assumptions that have to be met for a correlation to be valid: https://www.statology.org/pearson-correlation-assumptions/. However, that level of detail is probably beyond the scope of the documentation here. At some level, the user is assumed to know what they're doing if they use an aggregation.

kolchfa-aws · 2025-03-14T20:06:39Z

@jainankitk Could you please review this PR? Thanks!

jainankitk · 2025-03-18T01:25:56Z

_aggregations/metric/matrix-stats.md

@@ -9,8 +9,23 @@ redirect_from:

 # Matrix stats aggregations

-The `matrix_stats` aggregation generates advanced stats for multiple fields in a matrix form.
-The following example returns advanced stats in a matrix form for the `taxful_total_price` and `products.base_price` fields:
+The `matrix_stats` metric is a multi-value metric that generates covariance statistics for two or more fields in matrix form. 


Again will probably keep it as multi-value metric aggregation and maybe link to the metric aggregation page

_aggregations/metric/matrix-stats.md

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Dave Welsch <[email protected]>

Signed-off-by: Fanit Kolchina <[email protected]>

kolchfa-aws · 2025-05-21T14:58:23Z

@jainankitk Thanks for your feedback. I addressed your comments and added a section about the missing parameter.

natebower

@kolchfa-aws One comment and a couple of changes. Thanks!

_aggregations/metric/matrix-stats.md

Signed-off-by: kolchfa-aws <[email protected]>

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

* Update matrix-stats aggregation Signed-off-by: Dave Welsch <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Add missing parameter section Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Update _aggregations/metric/matrix-stats.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: Nathan Bower <[email protected]> (cherry picked from commit c8abc5e) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Update matrix-stats aggregation Signed-off-by: Dave Welsch <[email protected]> * Apply suggestions from code review Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Dave Welsch <[email protected]> * Doc review Signed-off-by: Fanit Kolchina <[email protected]> * Add missing parameter section Signed-off-by: Fanit Kolchina <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> * Update _aggregations/metric/matrix-stats.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Dave Welsch <[email protected]> Signed-off-by: Fanit Kolchina <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]> Co-authored-by: Fanit Kolchina <[email protected]> Co-authored-by: Nathan Bower <[email protected]>

Update matrix-stats aggregation

a9ee74d

Signed-off-by: Dave Welsch <[email protected]>

dwelsch-esi requested review from kolchfa-aws, Naarcha-AWS, AMoo-Miki, natebower, dlvenable and epugh as code owners March 14, 2025 19:25

github-actions bot assigned kolchfa-aws Mar 14, 2025

dwelsch-esi commented Mar 14, 2025

View reviewed changes

_aggregations/metric/matrix-stats.md Show resolved Hide resolved

dwelsch-esi commented Mar 14, 2025

View reviewed changes

kolchfa-aws added 3 - Tech review PR: Tech review in progress Content gap backport 2.19 labels Mar 14, 2025

jainankitk reviewed Mar 18, 2025

View reviewed changes

kolchfa-aws reviewed Apr 21, 2025

View reviewed changes

_aggregations/metric/matrix-stats.md Outdated Show resolved Hide resolved

Apply suggestions from code review

24dfaa7

Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Dave Welsch <[email protected]>

kolchfa-aws added backport 3.0 and removed backport 2.19 labels May 6, 2025

jainankitk mentioned this pull request May 8, 2025

[BUG] Caching does not consider mode parameter for matrix-stats aggregation opensearch-project/OpenSearch#18242

Closed

kolchfa-aws and others added 3 commits May 21, 2025 09:37

Merge branch 'main' into aggs-matrix-1

3d686c2

Doc review

d6c59a6

Signed-off-by: Fanit Kolchina <[email protected]>

Add missing parameter section

a7bbee7

Signed-off-by: Fanit Kolchina <[email protected]>

natebower reviewed May 21, 2025

View reviewed changes

_aggregations/metric/matrix-stats.md Outdated Show resolved Hide resolved

_aggregations/metric/matrix-stats.md Outdated Show resolved Hide resolved

_aggregations/metric/matrix-stats.md Show resolved Hide resolved

kolchfa-aws reviewed May 21, 2025

View reviewed changes

_aggregations/metric/matrix-stats.md Outdated Show resolved Hide resolved

kolchfa-aws and others added 2 commits May 21, 2025 11:36

Apply suggestions from code review

7791ad2

Signed-off-by: kolchfa-aws <[email protected]>

Update _aggregations/metric/matrix-stats.md

931d14b

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: kolchfa-aws <[email protected]>

kolchfa-aws approved these changes May 21, 2025

View reviewed changes

jainankitk approved these changes May 21, 2025

View reviewed changes

kolchfa-aws merged commit c8abc5e into opensearch-project:main May 22, 2025
8 checks passed

opensearch-trigger-bot bot mentioned this pull request May 22, 2025

[Backport 3.0] Update matrix-stats aggregation. #9964

Merged

github-actions bot pushed a commit that referenced this pull request May 22, 2025

Update matrix-stats aggregation. (#9434) (#9964)

0dba84d

Update matrix-stats aggregation. #9434

Update matrix-stats aggregation. #9434

Uh oh!

Conversation

dwelsch-esi commented Mar 14, 2025

Description

Issues Resolved

Version

Frontend features

Checklist

Uh oh!

github-actions bot commented Mar 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dwelsch-esi Apr 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dwelsch-esi Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented Mar 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kolchfa-aws commented May 21, 2025

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dwelsch-esi Apr 11, 2025 •

edited

Loading

dwelsch-esi Mar 18, 2025 •

edited

Loading