[BUG] Field-level `ignore_malformed` should override index-level setting

[andrross]

Describe the bug

The ignore_malformed property can be set at the index level and at the individual field level. I believe the field-level setting should override the index-level setting if both are present, but that is not the observed behavior for OpenSearch 2.4 and later.

Related component

Indexing

To Reproduce

Create an index like the following:

curl -X PUT -u admin:$PWD --insecure "https://localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.mapping.ignore_malformed": true
  },
  "mappings": {
    "properties": {
      "number_one": {
        "type": "byte"
      },
      "number_two": {
        "type": "integer",
        "ignore_malformed": false
      }
    }
  }
}
'

Index a document with a malformed value for the number_two field:

curl -X PUT -u admin:$PWD --insecure "https://localhost:9200/my-index-000001/_doc/2?pretty" -H 'Content-Type: application/json' -d'
{
  "text":       "Some text value",
  "number_two": "foo"
}
'

This succeeds, but it should fail because the ignore_malformed value has been set to false for that field.

Expected behavior

Field-level setting should override index-level setting.

Additional Details

This behaved as expected for versions of OpenSearch prior to 2.4.

Slack thread where originally reported: https://opensearch.slack.com/archives/C0539F41Z5X/p1730959987164549

[RS146BIJAY]

Thanks @andrross filing this bug. Looking forward for the fix of this issue.

[Mishail]

It may be related to #4494, which I believe makes the index-level ignore_malformed setting override field level

[afshirazi]

Hello, I'd like to try and fix this bug. Could it be assigned to me?

[andrross]

Hello, I'd like to try and fix this bug. Could it be assigned to me?

Done! Thanks @afshirazi!

[afshirazi]

I've been peeking around the code a little bit, and it does seem like #4494 is the culprit.
@andrross I'm not completely certain how to tackle this. After reverting the code from #4494, I tried testing the issue with a few cases; when setting ignore_malformed at either index or field level, documents with malformed data are successfully ignored, except when the data contains an object.

curl -X PUT -u admin:$PWD --insecure "localhost:9200/my-index-000001?pretty" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.mapping.ignore_malformed": true
  },
  "mappings": {
    "properties": {
      "number_one": {
        "type": "long"
      }
    }
  }
}
'

This succeeds, field number_one is ignored:

curl -X PUT -u admin:$PWD --insecure "localhost:9200/my-index-000006/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
  "number_one": "some value"      
}
'

However, this fails:

curl -X PUT -u admin:$PWD --insecure "localhost:9200/my-index-000006/_doc/2?pretty" -H 'Content-Type: application/json' -d'
{
  "number_one": {"some": "value"}
}
'

It seems like the parser fails on objects and embedded objects, regardless of whether ignore_malformed is set or not. Any ideas on where to start looking?

[andrross]

It seems like the parser fails on objects and embedded objects, regardless of whether ignore_malformed is set or not. Any ideas on where to start looking?

@afshirazi Hmm, I'm not sure. Can you see if this failed in versions prior to 2.4? That will at least tell you if it is related to changes since then.

[afshirazi]

Sorry for the delay. I checked with version 2.3.0, and the same issue seems to persist. Indexing a document with an object fails outright, while other types (array, string, etc.) correctly get ignored. I'll keep looking into it.

[afshirazi]

I've been looking into it in my free time. It seems that the text() method inside JsonXContentParser is contributing to some of this. Any time a START_OBJECT token is encountered by the parser, calling text() on it causes an IllegalStateException because the isValue() method inside START_OBJECT returns false. The exception does not get caught and causes the failure. However, START_ARRAY also has isValue() return false, but it does not cause the same issue.

When testing it out, it seemed like the parser would mysteriously skip over the START_ARRAY token and instantly jump into whatever was inside the array instead, so text() never gets called on START_ARRAY, and an exception is never thrown. I tried to look into it but still can't understand why the parser would just jump over the START_ARRAY token.

Curiously enough, the parse() method inside AbstractGeometryFieldMapper has a generic catch statement (near line 440) so when creating an index with some geo type, indexing a document that contains an object actually succeeds.

I'm going to try and see if I can figure out what is going on with the parser. The only other solution that I can think of would be to amend every FieldMapper to check for this case, which does not seem practical.