Skip to content

Commit 5ce679e

Browse files
jillesvangurpjillesvangurp
authored
Merge pull request #171 from jillesvangurp/es9-os3
2 parents 2a84453 + 76ab618 commit 5ce679e

File tree

22 files changed

+227
-112
lines changed

22 files changed

+227
-112
lines changed

‎.github/workflows/deploy-docs-and-test.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ jobs:
4242
runs-on: ubuntu-latest
4343
strategy:
4444
matrix:
45-
version: ["es-7","es-8","os-1", "os-2"]
45+
version: ["es-7","es-8","es-9","os-1", "os-2","os-3"]
4646
steps:
4747
- name: Checkout the code
4848
uses: actions/checkout@master

‎README.md

+25-22
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,8 @@ Kt-search is extensible and modular. You can easily add your own custom DSLs for
4747

4848
## Gradle
4949

50+
Kt-search is published to the FORMATION maven repository.
51+
5052
Add the `maven.tryformation.com` repository:
5153

5254
```kotlin
@@ -59,21 +61,16 @@ repositories {
5961
}
6062
}
6163
```
62-
63-
And then the dependency to commonsMain or main:
64+
And then add the dependency like this:
6465

6566
```kotlin
6667
// check the latest release tag for the latest version
6768
implementation("com.jillesvangurp:search-client:2.x.y")
6869
```
6970

70-
**About maven central ...** I've switched maven repositories a couple of times now. Jitpack and multiplatform just doesn't work. Of course I would have liked to get this on maven central. However, after repeated attempts to get that done, I've decided to not sacrifice more time on this. The (lack of) documentation, the Jira bureaucracy, the uninformative errors, the gradle plugin, etc. just doesn't add up to something that works for a multi module, multi platform project. I'm sure it can be done but I'm not taking more time out my schedule to find out.
71-
72-
If somebody decides to fix a proper, modern solution for hosting packages, I'll consider using it but I'm done with maven central for now. Google buckets work fine for hosting. So does ssh or any old web server. So does aws. It's just maven central that's a huge PITA.
73-
7471
## Maven
7572

76-
If you have maven based kotlin project targeting jvm and can't use kotlin multiplatform dependency, you would need to add jvm targeting artifacts.
73+
If you have maven based kotlin project targeting jvm and can't use kotlin multiplatform dependency, you will need to **append '-jvm' to the artifacts**.
7774

7875
Add the `maven.tryformation.com` repository:
7976

@@ -87,24 +84,24 @@ Add the `maven.tryformation.com` repository:
8784
</repositories>
8885
```
8986

90-
And then add dependencies to jvm targets:
87+
And then add dependencies for jvm targets:
9188

9289
```xml
9390
<dependencies>
9491
<dependency>
9592
<groupId>com.jillesvangurp</groupId>
9693
<artifactId>search-client-jvm</artifactId>
97-
<version>2.1.25</version>
94+
<version>2.x.y</version>
9895
</dependency>
9996
<dependency>
10097
<groupId>com.jillesvangurp</groupId>
10198
<artifactId>search-dsls-jvm</artifactId>
102-
<version>2.1.25</version>
99+
<version>2.x.y</version>
103100
</dependency>
104101
<dependency>
105102
<groupId>com.jillesvangurp</groupId>
106103
<artifactId>json-dsl-jvm</artifactId>
107-
<version>3.0.0</version>
104+
<version>3.x.y</version>
108105
</dependency>
109106
</dependencies>
110107
```
@@ -427,30 +424,36 @@ For additional details, refer to the build file.
427424

428425
## Compatibility
429426

430-
The integration tests on GitHub Actions use a **matrix build** that tests everything against Elasticsearch 7 & 8 and Opensearch 1 & 2.
427+
The integration tests on GitHub Actions use a **matrix build** that tests everything against Elasticsearch 7.x, 8.x & 9.x and Opensearch 1.x, 2.x & 3.x.
431428

432429
It may work fine with earlier Elasticsearch versions as well. But we don't actively test this and the tests are known to not pass with Elasticsearch 6 due to some changes in the mapping dsl. You may be able to work around some of this, however.
433430

434-
There is an annotation that is used to restrict APIs when needed. E.g. `search-after` only works with Elasticsearch and and has the following annotation to indicate that:
431+
There is an annotation that is used to restrict APIs when needed. E.g. `search-after` support was added in Opensearch 2.x but is missing in 1.x:
435432

436433
```kotlin
437434
@VariantRestriction(SearchEngineVariant.ES7,SearchEngineVariant.ES8)
438435
suspend fun SearchClient.searchAfter(target: String, keepAlive: Duration, query: SearchDSL): Pair<SearchResponse,Flow<SearchResponse.Hit>> {
439436
validateEngine("search_after does not work on OS1",
440437
SearchEngineVariant.ES7,
441-
SearchEngineVariant.ES8)
438+
SearchEngineVariant.ES8,
439+
SearchEngineVariant.ES9,
440+
SearchEngineVariant.OS2,
441+
SearchEngineVariant.OS3,
442+
)
442443

443444
// ...
444445
}
445446
```
446447

447448
The annotation is informational only for now. In our tests, we use `onlyon` to prevent tests from
448-
failing on unsupported engines For example, this is added to the test for `search_after`:
449+
failing on unsupported engines:
449450

450451
```kotlin
451-
onlyOn("opensearch has search_after but it works a bit different",
452+
onlyOn("doesn't work on Opensearch",
452453
SearchEngineVariant.ES7,
453-
SearchEngineVariant.ES8)
454+
SearchEngineVariant.ES8,
455+
SearchEngineVariant.ES9,
456+
)
454457
```
455458

456459
## Kotlin & Multiplatform compatibility
@@ -465,11 +468,11 @@ If you try and find issues, use the issue tracker please.
465468

466469
This repository contains several kotlin modules that each may be used independently.
467470

468-
| Module | Description |
469-
|-----------------|--------------------------------------------------------------------------------------------------------------------------|
470-
| `search-dsls` | DSLs for search and mappings based on `json-dsl`. |
471-
| `search-client` | Multiplatform REST client for Elasticsearch 7 & 8 and Opensearch 1. This is what you would want to use in your projects. |
472-
| `docs` | Contains the code that generates the [manual](https://jillesvangurp.github.io/kt-search/manual/) and this readme.. |
471+
| Module | Description |
472+
|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------|
473+
| `search-dsls` | DSLs for search and mappings based on `json-dsl`. |
474+
| `search-client` | Multiplatform REST client for Elasticsearch 7.x, 8.x & 9.x and Opensearch 1.x, 2.x & 3.x. This is what you would want to use in your projects. |
475+
| `docs` | Contains the code that generates the [manual](https://jillesvangurp.github.io/kt-search/manual/) and this readme.. |
473476

474477
The search client module is the main module of this library. I extracted the json-dsl module and `search-dsls` module with the intention of eventually moving these to separate libraries. Json-dsl is actually useful for pretty much any kind of json dialect and I have a few APIs in mind where I might like to use it. The choice to not impose kotlinx.serialization on json dsl also means that both that and the search dsl are very portable and only depend on the Kotlin standard library.
475478

‎docker-compose-es-9.yml

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
services:
2+
es9:
3+
image: docker.elastic.co/elasticsearch/elasticsearch:9.0.0-rc1
4+
environment:
5+
- cluster.name=docker-test-cluster
6+
- bootstrap.memory_lock=true
7+
- discovery.type=single-node
8+
- "ES_JAVA_OPTS=-Xms2048m -Xmx2048m"
9+
- cluster.routing.allocation.disk.threshold_enabled=true
10+
# make sure it works on nearly full disk
11+
- cluster.routing.allocation.disk.watermark.low=3gb
12+
- cluster.routing.allocation.disk.watermark.high=2gb
13+
- cluster.routing.allocation.disk.watermark.flood_stage=1gb
14+
- cluster.routing.allocation.disk.threshold_enabled=false
15+
- xpack.security.enabled=false
16+
- network.host=127.0.0.1
17+
- http.host=0.0.0.0
18+
19+
ulimits:
20+
memlock:
21+
soft: -1
22+
hard: -1
23+
ports:
24+
# give our test instance a port number that is for sure not going to write to some poor cluster listening on 9200
25+
- "9999:9200"

‎docker-compose-os-3.yml

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
services:
2+
os2:
3+
image: opensearchproject/opensearch:3
4+
environment:
5+
- cluster.name=docker-test-cluster
6+
- bootstrap.memory_lock=true
7+
- discovery.type=single-node
8+
- "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
9+
- cluster.routing.allocation.disk.threshold_enabled=true
10+
# make sure it works on nearly full disk
11+
- cluster.routing.allocation.disk.watermark.low=3gb
12+
- cluster.routing.allocation.disk.watermark.high=2gb
13+
- cluster.routing.allocation.disk.watermark.flood_stage=1gb
14+
- cluster.routing.allocation.disk.threshold_enabled=false
15+
- DISABLE_SECURITY_PLUGIN=true
16+
ulimits:
17+
memlock:
18+
soft: -1
19+
hard: -1
20+
ports:
21+
# give our test instance a port number that is for sure not going to write to some poor cluster listening on 9200
22+
- "9999:9200"
23+
# os2-dashboards:
24+
# image: opensearchproject/opensearch-dashboards:2.18.0
25+
# environment:
26+
# - OPENSEARCH_HOSTS=http://os2:9200
27+
# - DISABLE_SECURITY_DASHBOARDS_PLUGIN=true
28+
# ports:
29+
# - "5601:5601"
30+
# depends_on:
31+
# - os2

‎docs/src/test/kotlin/documentation/manual/gettingstarted/getting-started.kt

+1-1
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ val gettingStartedMd = sourceGitRepository.md {
2626
To get started, simply add the dependency to your project and create a client.
2727
The process is the same for both jvm and kotlin-js.
2828
""".trimIndent()
29-
includeMdFile("../../projectreadme/gradle.md")
29+
includeMdFile("../../projectreadme/gradle-maven.md")
3030
// test server runs on 9999, so we need to override
3131
val client = SearchClient(KtorRestClient(Node("localhost", 9999)))
3232

‎docs/src/test/kotlin/documentation/manual/search/deep-paging.kt

+1-4
Original file line numberDiff line numberDiff line change
@@ -108,10 +108,7 @@ val deepPagingMd = sourceGitRepository.md {
108108
is generally used in combination with the Point in Time API. Kt-search adds a sort on "_shard_doc" for you.
109109
110110
If you want to add your own sort, you have to opt-in to this and deal with any issues yourself as you can
111-
easily break search_after this way.
112-
113-
Note. Currently only works with Elasticsearch. Opensearch has a very similar implementation but it works
114-
slightly differently and currently kt-search only supports the elastic variant for this.
111+
easily break search_after this way.
115112
""".trimIndent()
116113

117114
example {

‎docs/src/test/kotlin/documentation/projectreadme/maven.md renamed to ‎docs/src/test/kotlin/documentation/projectreadme/gradle-maven.md

+28-5
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,29 @@
1+
## Gradle
2+
3+
Kt-search is published to the FORMATION maven repository.
4+
5+
Add the `maven.tryformation.com` repository:
6+
7+
```kotlin
8+
repositories {
9+
mavenCentral()
10+
maven("https://maven.tryformation.com/releases") {
11+
content {
12+
includeGroup("com.jillesvangurp")
13+
}
14+
}
15+
}
16+
```
17+
And then add the dependency like this:
18+
19+
```kotlin
20+
// check the latest release tag for the latest version
21+
implementation("com.jillesvangurp:search-client:2.x.y")
22+
```
23+
124
## Maven
225

3-
If you have maven based kotlin project targeting jvm and can't use kotlin multiplatform dependency, you would need to add jvm targeting artifacts.
26+
If you have maven based kotlin project targeting jvm and can't use kotlin multiplatform dependency, you will need to **append '-jvm' to the artifacts**.
427

528
Add the `maven.tryformation.com` repository:
629

@@ -14,24 +37,24 @@ Add the `maven.tryformation.com` repository:
1437
</repositories>
1538
```
1639

17-
And then add dependencies to jvm targets:
40+
And then add dependencies for jvm targets:
1841

1942
```xml
2043
<dependencies>
2144
<dependency>
2245
<groupId>com.jillesvangurp</groupId>
2346
<artifactId>search-client-jvm</artifactId>
24-
<version>2.1.25</version>
47+
<version>2.x.y</version>
2548
</dependency>
2649
<dependency>
2750
<groupId>com.jillesvangurp</groupId>
2851
<artifactId>search-dsls-jvm</artifactId>
29-
<version>2.1.25</version>
52+
<version>2.x.y</version>
3053
</dependency>
3154
<dependency>
3255
<groupId>com.jillesvangurp</groupId>
3356
<artifactId>json-dsl-jvm</artifactId>
34-
<version>3.0.0</version>
57+
<version>3.x.y</version>
3558
</dependency>
3659
</dependencies>
3760
```

‎docs/src/test/kotlin/documentation/projectreadme/gradle.md

-25
This file was deleted.

‎docs/src/test/kotlin/documentation/projectreadme/readme-outro.md

+17-11
Original file line numberDiff line numberDiff line change
@@ -15,30 +15,36 @@ For additional details, refer to the build file.
1515

1616
## Compatibility
1717

18-
The integration tests on GitHub Actions use a **matrix build** that tests everything against Elasticsearch 7 & 8 and Opensearch 1 & 2.
18+
The integration tests on GitHub Actions use a **matrix build** that tests everything against Elasticsearch 7.x, 8.x & 9.x and Opensearch 1.x, 2.x & 3.x.
1919

2020
It may work fine with earlier Elasticsearch versions as well. But we don't actively test this and the tests are known to not pass with Elasticsearch 6 due to some changes in the mapping dsl. You may be able to work around some of this, however.
2121

22-
There is an annotation that is used to restrict APIs when needed. E.g. `search-after` only works with Elasticsearch and and has the following annotation to indicate that:
22+
There is an annotation that is used to restrict APIs when needed. E.g. `search-after` support was added in Opensearch 2.x but is missing in 1.x:
2323

2424
```kotlin
2525
@VariantRestriction(SearchEngineVariant.ES7,SearchEngineVariant.ES8)
2626
suspend fun SearchClient.searchAfter(target: String, keepAlive: Duration, query: SearchDSL): Pair<SearchResponse,Flow<SearchResponse.Hit>> {
2727
validateEngine("search_after does not work on OS1",
2828
SearchEngineVariant.ES7,
29-
SearchEngineVariant.ES8)
29+
SearchEngineVariant.ES8,
30+
SearchEngineVariant.ES9,
31+
SearchEngineVariant.OS2,
32+
SearchEngineVariant.OS3,
33+
)
3034

3135
// ...
3236
}
3337
```
3438

3539
The annotation is informational only for now. In our tests, we use `onlyon` to prevent tests from
36-
failing on unsupported engines For example, this is added to the test for `search_after`:
40+
failing on unsupported engines:
3741

3842
```kotlin
39-
onlyOn("opensearch has search_after but it works a bit different",
43+
onlyOn("doesn't work on Opensearch",
4044
SearchEngineVariant.ES7,
41-
SearchEngineVariant.ES8)
45+
SearchEngineVariant.ES8,
46+
SearchEngineVariant.ES9,
47+
)
4248
```
4349

4450
## Kotlin & Multiplatform compatibility
@@ -53,11 +59,11 @@ If you try and find issues, use the issue tracker please.
5359

5460
This repository contains several kotlin modules that each may be used independently.
5561

56-
| Module | Description |
57-
|-----------------|--------------------------------------------------------------------------------------------------------------------------|
58-
| `search-dsls` | DSLs for search and mappings based on `json-dsl`. |
59-
| `search-client` | Multiplatform REST client for Elasticsearch 7 & 8 and Opensearch 1. This is what you would want to use in your projects. |
60-
| `docs` | Contains the code that generates the [manual](https://jillesvangurp.github.io/kt-search/manual/) and this readme.. |
62+
| Module | Description |
63+
|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------|
64+
| `search-dsls` | DSLs for search and mappings based on `json-dsl`. |
65+
| `search-client` | Multiplatform REST client for Elasticsearch 7.x, 8.x & 9.x and Opensearch 1.x, 2.x & 3.x. This is what you would want to use in your projects. |
66+
| `docs` | Contains the code that generates the [manual](https://jillesvangurp.github.io/kt-search/manual/) and this readme.. |
6167

6268
The search client module is the main module of this library. I extracted the json-dsl module and `search-dsls` module with the intention of eventually moving these to separate libraries. Json-dsl is actually useful for pretty much any kind of json dialect and I have a few APIs in mind where I might like to use it. The choice to not impose kotlinx.serialization on json dsl also means that both that and the search dsl are very portable and only depend on the Kotlin standard library.
6369

‎docs/src/test/kotlin/documentation/projectreadme/readme.kt

+1-3
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,7 @@ val projectReadme = sourceGitRepository.md {
2121

2222
includeMdFile("readme-intro.md")
2323

24-
includeMdFile("gradle.md")
25-
26-
includeMdFile("maven.md")
24+
includeMdFile("gradle-maven.md")
2725

2826
section("Usage") {
2927
// our test server runs on port 9999

‎search-client/src/commonMain/kotlin/com/jillesvangurp/ktsearch/SearchResponse.kt

+2
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ data class SearchResponse(
3030
val scrollId: String?,
3131
@SerialName("pit_id")
3232
val pitId: String?,
33+
@SerialName("point_in_time_id")
34+
val pointInTimeId: String?,
3335
val suggest: Map<String, List<Suggest>>?
3436
) {
3537
@Serializable

‎search-client/src/commonMain/kotlin/com/jillesvangurp/ktsearch/ilm.kt

+2-1
Original file line numberDiff line numberDiff line change
@@ -87,7 +87,8 @@ suspend fun SearchClient.setIlmPolicy(policyId: String, block: IMLPhases.()->Uni
8787
validateEngine(
8888
"ilm only works on Elasticsearch",
8989
SearchEngineVariant.ES7,
90-
SearchEngineVariant.ES8
90+
SearchEngineVariant.ES8,
91+
SearchEngineVariant.ES9
9192
)
9293

9394
val config = ILMConfiguration()

0 commit comments

Comments
 (0)