Milvus Vector - Support for Milvus Native Filter Expressions in doSimilaritySearch

[waileong]

Support for Milvus Native Filter Expressions in doSimilaritySearch

Description

Currently, the implementation of doSimilaritySearch in SearchRequest does not provide direct support for passing native Milvus filter expressions. The existing approach converts Filter.Expression into a Milvus-compatible filter using filterExpressionConverter, which limits flexibility, especially for advanced filtering capabilities such as the LIKE operator, which is currently unsupported.

To address this limitation, we propose adding a new parameter, String nativeExpression, to allow direct passing of native Milvus filter expressions. This parameter should take priority over the existing filterExpressionConverter.convertExpression(request.getFilterExpression()) logic when specified.

Proposed Changes

1. Add a new optional parameter String nativeExpression to SearchRequest.
2. Modify doSimilaritySearch to prioritize nativeExpression if provided, bypassing filterExpressionConverter.convertExpression.
3. Ensure that nativeExpression integrates seamlessly with the existing search request pipeline.

Rationale

• Enables support for Milvus-specific filter expressions, including operators like LIKE, IN, and other advanced filtering capabilities.
• Reduces dependency on filterExpressionConverter, providing more flexibility in query execution.
• Improves compatibility with Milvus query syntax for users needing finer control over search filtering.

Example Usage

SearchRequest request = SearchRequest.builder()
    .query("example query")
    .topK(10)
    .similarityThreshold(0.8)
    .nativeExpression("city LIKE 'New%' AND price > 1000") // Direct Milvus expression
    .build();

Expected Behavior

If nativeExpression is provided, it should be used in the search query instead of the converted filter expression. If nativeExpression is null, fallback to the existing behavior.

Additional Context

Milvus documentation on boolean expressions: https://milvus.io/docs/boolean.md

This feature would greatly enhance the flexibility of doSimilaritySearch, allowing for more complex and native-compatible queries in Milvus.

[waileong]

Related to #2294

[waileong]

@ilayaperumalg, I would be grateful if you could kindly look into this issue. Thank you very much.

[ilayaperumalg]

@waileong Thank you for creating descriptive report and the follow up PR.