Skip to content

Add highlighting for match_only_text type #17101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Jan 30, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e038c68
Add highlighting for match_only_text type
etolbakov Jan 23, 2025
914ef1c
Add intergration tests, address CR feedback
etolbakov Jan 24, 2025
8d342d5
resolve conflicts
etolbakov Jan 29, 2025
25f439f
Updates version to fix BWC for SearchSourceBuilder (#17098)
junweid62 Jan 23, 2025
c6aa24e
Introduce Template query (#16818)
mingshl Jan 23, 2025
793cb6b
Propagate includes and excludes from fetchSourceContext to FieldsVisi…
Gankris96 Jan 24, 2025
42576e0
Update version in integration test
etolbakov Jan 24, 2025
65694b4
Fix compilation error
etolbakov Jan 24, 2025
0553b3c
Fix exists query for flat object (#17108)
bugmakerrrrrr Jan 24, 2025
ee8bd11
Added new Setting property `UnmodifiableOnRestore` to prevent updatin…
anntians Jan 24, 2025
740e884
Fix Binlong's name in MAINTAINERS.md (#17113)
andrross Jan 24, 2025
58fc9a9
Move o.o.action.support.master classes (#17104)
andrross Jan 24, 2025
ad013e8
Stop processing search requests when _msearch is canceled (#17005)
msfroh Jan 24, 2025
52ed338
Refactor bootstrap for JPMS support (#17117)
prudhvigodithi Jan 25, 2025
fa12e66
Fix flaky test RemoteSegmentTransferTrackerTests.testGetInflightUploa…
sachinpkale Jan 27, 2025
dec1f32
resolve conflicts
etolbakov Jan 29, 2025
432359f
Remove package `org.opensearch.action.support.master` (#17118)
andrross Jan 27, 2025
9594344
Update Netty to 4.1.117.Final (#17041)
reta Jan 27, 2025
6652fe5
[Star Tree] [Search] Resolve Date histogram with metric aggregation u…
sandeshkr419 Jan 27, 2025
ccf8ff3
Update Gradle to 8.12.1 (#17145)
reta Jan 27, 2025
c976924
OpenJDK Update (January 2025 Patch releases) (#17139)
reta Jan 27, 2025
e28df41
Adding support for append only indices (#17039)
RS146BIJAY Jan 28, 2025
0618921
Add Star Tree unsigned-long indexing changes (#17156)
Shailesh-Kumar-Singh Jan 28, 2025
f177a31
Remove MasterService class (#17151)
andrross Jan 28, 2025
be9467e
resolve conflicts
etolbakov Jan 29, 2025
bd8141d
[WLM] Add wlm support for scroll API (#16981)
kaushalmahi12 Jan 28, 2025
5caaed5
Remove more deprecated master classes (#17166)
andrross Jan 28, 2025
d586e22
Fix failing CI's (#17172)
prudhvigodithi Jan 29, 2025
07a79ca
Extensible design to add new query and field type support for Star Tr…
expani Jan 29, 2025
af1b5af
Address CR feedback(fix integration tests)
etolbakov Jan 29, 2025
9f00822
Merge branch 'main' into match_only_text-highlighting
etolbakov Jan 29, 2025
432a6bd
Fix minor typo
etolbakov Jan 29, 2025
71ca068
Fix yaml indentation
etolbakov Jan 30, 2025
6e908cc
Fix test assertion
etolbakov Jan 30, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Fix exists queries on nested flat_object fields throws exception ([#16803](https://github.com/opensearch-project/OpenSearch/pull/16803))
- Use OpenSearch version to deserialize remote custom metadata([#16494](https://github.com/opensearch-project/OpenSearch/pull/16494))
- Fix AutoDateHistogramAggregator rounding assertion failure ([#17023](https://github.com/opensearch-project/OpenSearch/pull/17023))
- Add highlighting for wildcard search on `match_only_text` field ([#17101](https://github.com/opensearch-project/OpenSearch/pull/17101))
- Fix the failing CI's with `Failed to load eclipse jdt formatter` error ([#17172](https://github.com/opensearch-project/OpenSearch/pull/17172))

### Security
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# integration tests for queries with specific analysis chains

---
"match query with stacked stems":
- skip:
version: " - 2.11.99"
Expand Down Expand Up @@ -68,3 +68,80 @@
query: fox runs
operator: AND
- match: {hits.total: 2}

---
"wildcard highlighting on match_only_text":
- skip:
version: " - 2.99.99"
reason: "wildcard highlighting on match_only_text type was added in 2.19"
- do:
indices.create:
index: test
body:
settings:
number_of_shards: 1
number_of_replicas: 1
analysis:
analyzer:
index:
tokenizer: standard
filter: [lowercase]
search:
rest_total_hits_as_int: true
tokenizer: standard
filter: [lowercase, keyword_repeat, porter_stem, unique_stem]
filter:
unique_stem:
type: unique
only_on_same_position: true
mappings:
properties:
text:
type: match_only_text
analyzer: index
search_analyzer: search

- do:
index:
index: test
id: 1
body: { "text": "the fox runs across the street" }
refresh: true

- do:
search:
rest_total_hits_as_int: true
body:
query:
match:
text:
query: fox runs
operator: AND
highlight:
fields:
- text: {}
- match: {hits.total: 1}
- match: {hits.hits.0.highlight.text.0: "the <em>fox</em> <em>runs</em> across the street"}

- do:
index:
index: test
id: 2
body: { "text": "run fox run" }
refresh: true

- do:
search:
rest_total_hits_as_int: true
body:
query:
match:
text:
query: fox runs
operator: AND
highlight:
fields:
- text: {}
- match: {hits.total: 2}
- match: {hits.hits.0.highlight.text.0: "the <em>fox</em> <em>runs</em> across the street"}
- match: {hits.hits.1.highlight.text.0: "<em>run</em> <em>fox</em> <em>run</em>"}
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@
import org.opensearch.common.regex.Regex;
import org.opensearch.index.mapper.KeywordFieldMapper;
import org.opensearch.index.mapper.MappedFieldType;
import org.opensearch.index.mapper.MatchOnlyTextFieldMapper;
import org.opensearch.index.mapper.SourceFieldMapper;
import org.opensearch.index.mapper.TextFieldMapper;
import org.opensearch.search.fetch.FetchContext;
Expand Down Expand Up @@ -152,7 +153,8 @@ private Map<String, Function<HitContext, FieldHighlightContext>> contextBuilders
continue;
}

// We should prevent highlighting if a field is anything but a text or keyword field.
// We should prevent highlighting if a field is anything but a text, match_only_text
// or keyword field.
// However, someone might implement a custom field type that has text and still want to
// highlight on that. We cannot know in advance if the highlighter will be able to
// highlight such a field and so we do the following:
Expand All @@ -162,7 +164,8 @@ private Map<String, Function<HitContext, FieldHighlightContext>> contextBuilders
// what they were doing and try to highlight anyway.
if (fieldNameContainsWildcards) {
if (fieldType.typeName().equals(TextFieldMapper.CONTENT_TYPE) == false
&& fieldType.typeName().equals(KeywordFieldMapper.CONTENT_TYPE) == false) {
&& fieldType.typeName().equals(KeywordFieldMapper.CONTENT_TYPE) == false
&& fieldType.typeName().equals(MatchOnlyTextFieldMapper.CONTENT_TYPE) == false) {
continue;
}
if (highlighter.canHighlight(fieldType) == false) {
Expand Down
Loading