RFC : Point In Time Search

[rajkthakur]

Is your feature request related to a problem? Please describe.

Today, in OpenSearch, if you want to run different queries on the same data set chances are you will get different result as data is constantly changing. However, in real world scenario when analyzing data or trying to provide a consistent user experience to your end users you may want the result from a query not to change while the context remains the same and control when changes should appear in the result set. You want to be able to query the same data set and paginate through the data set expecting consistent result. This is not possible using current available options in OpenSearch.

Opensearch currently supports the following options to achieve pagination, each having a certain limitation:

1. Scroll API : Scroll API cannot share point in time context with other queries. Moreover, the scroll API only allows to move forwards(next page) in the search, cases when the client sends the request for a page but fails to get a response, a subsequent retry call skips the page(retried for) and returns the next page in the scroll.
2. Search After : The search_after mechanism doesn't preserve the state of data when the search was issued, so one can paginate using the key (search_after) and fetch subsequent pages while getting more recent results since the search was issued as the pagination progresses.
3. From To : This mechanism does not support deep pagination since every page request requires the shard to process all previous results and then filter the requested page which might be taxing deeper the pagination goes

Describe the solution you'd like

Point in Time allows users to run different queries against the same fixed data set in time. Point in time only takes data into account up until the moment it is created. Hence, none of the resources that are required to return the data from the initial request are modified or deleted. Segments are retained, even though the segment might already have been merged away and is not needed for the live data set. In short, Point in Time Search allows user to maintain a state which can be re-used by different queries in order to achieve consistent results.

Key goals:

1. Optimize resource consumption compared to a scroll by providing a consistent, shareable view of data set across queries. More segments are otherwise needed to be retained as needed by individual queries which means more file handles, more disk and more heap to keep metadata from segments in the heap.
2. Resilient to
   1. Network failures : allows searches to move forward with a search_after parameter
   2. Shard failures for read-only data : allows retries on other shard copies that share the same segments (Phase - II)
3. Replaces scroll API, as a more comprehensive solution for deep pagination when used with search_after
4. Point in Time will be supported by Asynchronous Search and Cross Cluster searches

APIs

Create Point In Time API

Unlike a Scroll, by creating a dedicated Point in Time, we decouple the context from a single query and make it re-usable across arbitrary search requests by passing the Point in Time Id. We can achieve this by using the Create Point in Time API.

POST <index>/_point_in_time?keep_alive=1m
{
   "id" : "s9O9QAIFaW5kZXgWOFVaMXFTc3pTV3lLMGE4VU42dmo4dwAWekthUVBmYnRUWk9XVzh4WW56TG5lZwAAAAAAAAAAARZQd3JkNlE4WlJicXRuS0M1VzNDaHV3BWluZGV4FjhVWjFxU3N6U1d5SzBhOFVONnZqOHcBFnpLYVFQZmJ0VFpPV1c4eFluekxuZWcAAAAAAAAAAAIWUHdyZDZROFpSYnF0bktDNVczQ2h1dwEWOFVaMXFTc3pTV3lLMGE4VU42dmo4dwAA",
   "created_time" : 1632727466283,
   "end_time" : 1632727526283
}

Delete Point In Time API

Point-in-times are automatically closed when the keep_alive is elapsed. However, keeping point-in-times has a cost; hence, point-in-times should be closed as soon as they are no longer used in search requests. We may also delete a Point in Time and free the resources before its keep alive using the Delete Point in Time API.

DELETE /_point_in_time/<id>

List All Active Point In Time API

A useful admin API to have is to list all active Points in Time and their keep-alives.

GET /_point_in_time

[
    {
        "point_in_time_id_1",
        "created_time" : 1632727466283,
        "end_time" : 1632727526283
    },
    {
        "point_in_time_id_2",
        "created_time" : 1674662833272,
        "end_time" : 1632727526283
    }
    ...
    ...
]

Using a Point in Time in a search request:

In the search request we pass the point in time id and (optionally) a keep alive to extend the Point In Time. (Passing PIT id in search request is supported in Opensearch)
Search request with PIT ID will not accept indices, preference, routing and indices options as these are already passed at the time of creating a Point In Time.

GET /_search
{
  "pit": {
        "id":  "ID_RETURNED_FROM_CREATE_POINT_IN_TIME_REQUEST", 
        "keep_alive": "1m" //optional to extend a Point In Time
  },
  "sort": [
    {
      "name.keyword": {
        "order": "desc"
      }
    }
  ],
  "search_after" : ["Opensearch", 1] //optional to fetch further results 
}

[stockholmux]

@rajkthakur Can you make this issue a little more... fleshed out? I'm just seeing the dummy text.

[anasalkouz]

Hi @rajkthakur, are you actively working on this issue? if yes, please could you assign it to yourself and add a comment?

[eirsep]

@anasalkouz I am actively working on this issue.

[stockholmux]

Can we mark this as a proposal?

[eirsep]

I have done a small POC to check feasibility of the proposed APIs

We will not be able to provide a List All Points In Time APIs, as a point in time is not tied to any coordinator node.
Rather the Point In Time ID is simply a Base 64 encoded hash of list of reader context to node mappings and a UUID.

[eirsep]

We will provide new information in the Nodes stats api -
a nested object in the search section mentioning the number of currently active Point In Time contexts, total number of Point In Time contexts.

This will help keep track of point in time related statistics.

[eirsep]

We will provide settings to:

1. restrict the max open point in time contexts on a node
2. restrict the maximum keep alive allowed for point in times.

[eirsep]

similar to scroll, we will provide all points in time via the api by passing _all in the path for Delete Point In Time API.

[eirsep]

Currently doing a POC to add an API for PIT disk utilization i.e. segments retained by PIT Ids. I am trying to put out stats similar to cat segments, but for Points In Time.

[nknize]

Segments are retained

This can get expensive. I see the objective is to optimize resource consumption ✔️ . I think this is a specialized use case for archival or time based analysis use cases and not the normal search use case so this should be configured through an Index Scoped setting.

Segment replication will heavily change this design as it will push a lot of the PIT burden on the storage layer.

[eirsep]

Segments are retained

This can get expensive. I see the objective is to optimize resource consumption ✔️ . I think this is a specialized use case for archival or time based analysis use cases and not the normal search use case so this should be configured through an Index Scoped setting.

@nknize
I agree that segment retention is expensive, but this is exactly what scrolls do today. The resource consumption is being optimised only in comparison to Scrolls(i.e. being able to share PIT Id across queries while scrolls create different contexts per scroll).

(Among other things) PIT would be a replacement for scrolls and hence be the pagination solution for Opensearch, which is also an important use case to keep in mind. We would limit the number of PIT contexts that can be opened via a setting and would also provide an API to provide info about PITs' disk consumption

Will look into how-to and benefits of configuring PIT through Index Scoped setting.

Segment replication will heavily change this design as it will push a lot of the PIT burden on the storage layer.

Segments retained by PIT/Scrolls will not be replicated I think. Can you plz elaborate the caveats you see?