Support join field list and join options #3803

LantaoJin · 2025-06-20T03:30:11Z

Description

Support join field list and join options:

Add a setting to enable SPL-compatible syntax: plugins.ppl.spl_compatible.enabled
Add new join syntax: {SPL_compatible_grammar_enabled}? JOIN (joinOption)* (fieldList)? right = tableOrSubqueryClause

SPL Compatible Syntax

(Experimental)
(Since 3.2.0)
(prerequisite: plugins.ppl.spl_compatible.enabled=true)
join [type=<joinType>] [overwrite=<bool>] <join-field-list> <right-dataset>

type=<joinType>: optional. The type of join to perform. The default is INNER if not specified. Other option is LEFT, RIGHT, FULL, CROSS, SEMI, ANTI.
overwrite=<bool>: optional. Specifies whether duplicate-named fields from <right-dataset> (subsearch results) should replace corresponding fields in the main search results. The default value is true.
join-field-list: optional. The fields to use to build join criteria. The join-field-list must be present in both sides. If no <join-field-list> is present, all fields that are common to both sides are used.
right-dataset: mandatory. Right dataset could be either an index or a subsearch with/without alias.

Related Issues

Resolves #3775

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin · 2025-06-20T10:28:13Z

common/src/main/java/org/opensearch/sql/common/setting/Settings.java

@@ -29,6 +29,7 @@ public enum Key {
    PATTERN_MODE("plugins.ppl.pattern.mode"),
    PATTERN_MAX_SAMPLE_COUNT("plugins.ppl.pattern.max.sample.count"),
    PATTERN_BUFFER_LIMIT("plugins.ppl.pattern.buffer.limit"),
+    SPL_COMPATIBLE_GRAMMAR_ENABLED("plugins.ppl.spl_compatible.enabled"),


any other options? such as plugins.ppl.splunk_compatible.enabled to highlight splunk instead of spl

LantaoJin · 2025-06-20T10:30:52Z

docs/user/ppl/cmd/join.rst

+SPL Compatible Syntax
+=====================
+| (Experimental)
+| (Since 3.2.0)


may catch up 3.1.0?

penghuo · 2025-06-20T15:06:24Z

docs/user/ppl/admin/settings.rst

+Description
+-----------
+
+This setting is present from 3.2.0. Enabling Calcite is a prerequisite. You can use this setting to decide whether to allow parsing a query of Splunk SPL compatible grammar.


What is expected behavior of PPL commands/function if is not compatible with Splunk grammer?

Current implementation is SPL-compatible, not SPL-only.
For example:
When compatible config is true, user can run both rex and parse command. When compatible config is false, only parse command works.

If we want SPL-only, when SPL-only config is true, parse command won't work any more. Which one do you prefer?

penghuo · 2025-06-20T15:24:07Z

docs/user/ppl/cmd/join.rst

+| (prerequisite: plugins.ppl.spl_compatible.enabled=true)
+| join [type=<joinType>] [overwrite=<bool>] <join-field-list> <right-dataset>
+
+* type=<joinType>: optional. The type of join to perform. The default is ``INNER`` if not specified. Other option is ``LEFT``, ``RIGHT``, ``FULL``, ``CROSS``, ``SEMI``, ``ANTI``.


we could limit to inner | outer | left, then add other join types with perf-test gradually.

Em, I think we can add a config to enable other join type, and disable them by default, instead of removing from code.

penghuo · 2025-06-20T15:30:25Z

docs/user/ppl/cmd/join.rst

+| (Experimental)
+| (Since 3.2.0)
+| (prerequisite: plugins.ppl.spl_compatible.enabled=true)
+| join [type=<joinType>] [overwrite=<bool>] <join-field-list> <right-dataset>


change <right-dataset> to <dataset> | <subsearch>

is optional, change to []

add [left=<left-alias>] [right=<right-alias>] where ... in syntax?

add [left=<left-alias>] [right=<right-alias>] where ... in syntax?

this is spl2 syntax, I don't think we need to support it in compatible mode.

[left=<left-alias>] [right=<right-alias>] on ... is listed in default PPL syntax

penghuo · 2025-06-20T15:35:32Z

docs/user/ppl/cmd/join.rst

+
+* type=<joinType>: optional. The type of join to perform. The default is ``INNER`` if not specified. Other option is ``LEFT``, ``RIGHT``, ``FULL``, ``CROSS``, ``SEMI``, ``ANTI``.
+* overwrite=<bool>: optional. Specifies whether duplicate-named fields from <right-dataset> (subsearch results) should replace corresponding fields in the main search results. The default value is ``true``.
+* join-field-list: optional. The fields to use to build join criteria. The ``join-field-list`` must be present in both sides. If no <join-field-list> is present, all fields that are common to both sides are used.


Polish join-filed-list description,

The fields used to build the join criteria. The join field list must exist on both sides. If no join field list is specified, all fields common to both sides will be used as join keys.

penghuo · 2025-06-20T15:41:43Z

docs/user/ppl/cmd/join.rst

@@ -138,3 +178,5 @@ Assume table1 and table2 only contain field ``id``, following PPL queries and th
     - table1.id, t2.id, a
   * - source=table1 | join right=tt on table1.id=t2.id [ source=table2 as t2 | eval b = id ] | eval a = 1
     - table1.id, tt.id, tt.b, a
+
+For the Splunk SPL compatible syntax (since 3.2.0), duplicate-named fields in output results are deduplicated, with field retention determined by the value of 'overwrite' option.


it is expected behavior, not limitation, right?

penghuo · 2025-06-20T15:51:27Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

@@ -315,7 +323,8 @@ tableSourceClause

 // join
 joinCommand
-   : (joinType) JOIN sideAlias joinHintList? joinCriteria? right = tableOrSubqueryClause
+   : joinType JOIN sideAlias joinHintList? joinCriteria right = tableOrSubqueryClause
+   | {SPL_compatible_grammar_enabled}? JOIN (joinOption)* (fieldList)? right = tableOrSubqueryClause


why not support alias? left=l right=r

allow options to appear before or after fieldList, e.g. ... | join id overwrite=true [search index=product]

why not support alias? left=l right=r

in sideAlias

allow options to appear before or after fieldList

Before only.

penghuo · 2025-06-20T16:11:52Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+              "Join type is ambiguous, please remove the join type before JOIN keyword or 'type='"
+                  + " option.");


in spl mode, ... | inner join xxx with throw syntax exception, in which case user will see this error message,

penghuo · 2025-06-20T16:13:11Z

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

+    Argument.ArgumentMap argumentMap = Argument.ArgumentMap.of(arguments);
+    if (argumentMap.get("type") != null) {
+      Join.JoinType joinTypeFromArgument = getJoinType(argumentMap.get("type").toString());
+      if (ctx.joinType() == null) {


JoinType is not a valid token in SPL mode. The AST builder should be aware of SPL mode and handle it separately.

penghuo · 2025-06-20T16:26:25Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

@@ -315,7 +323,8 @@ tableSourceClause

 // join
 joinCommand
-   : (joinType) JOIN sideAlias joinHintList? joinCriteria? right = tableOrSubqueryClause
+   : joinType JOIN sideAlias joinHintList? joinCriteria right = tableOrSubqueryClause


We should expose only one syntax at a time? Concern of enable both PPL and SPL-compatible syntax is

Hard to document and maintain, tooling (like syntax highlighting, query editors) must also be mode-aware.

Increases complexity of semantic validation and testing.

The current design is when the config is set to true, both ppl and spl syntax are supported. For your concerns, let me check the challenge in implementation level next week. I don’t think the document has any blocker. In future we can add a legacy config to disable the current ppl syntax as long as the implementation level has no blocker. For join, the current ppl join is a mutation of join of spl2. We may still need to support since splunk can both support spl1 and spl2. For your second concern, as long as implementation part has no blocker, the validation should work well since validation is done by antlr4 itself.

LantaoJin added 2 commits June 20, 2025 11:29

Support join field list and join options

d94be39

Signed-off-by: Lantao Jin <[email protected]>

Add SPL-compatible syntax setting

3855e89

Signed-off-by: Lantao Jin <[email protected]>

LantaoJin commented Jun 20, 2025

View reviewed changes

LantaoJin marked this pull request as ready for review June 20, 2025 10:29

LantaoJin requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, noCharger and qianheng-aws as code owners June 20, 2025 10:29

LantaoJin added PPL Piped processing language calcite calcite migration releated labels Jun 20, 2025

LantaoJin commented Jun 20, 2025

View reviewed changes

penghuo reviewed Jun 20, 2025

View reviewed changes

		"Join type is ambiguous, please remove the join type before JOIN keyword or 'type='"
		+ " option.");

Support join field list and join options #3803

Are you sure you want to change the base?

Support join field list and join options #3803

Conversation

LantaoJin commented Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

SPL Compatible Syntax

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LantaoJin commented Jun 20, 2025 •

edited

Loading

LantaoJin Jun 23, 2025 •

edited

Loading

LantaoJin Jun 23, 2025 •

edited

Loading