Provide an option to paginate through all pages

[displague]

Provide an option to paginate through all pages.

An option like --all-pages would be helpful and prevent workflows like this:

(
linode-cli --text kernels list --page=100
for p in {1..3}; do
  linode-cli --no-headers --text  kernels list --page $p
done
)

[Dorthu]

I don't see much of a benefit to this. It's simple enough to paginate already, and since the CLI is generated I'd either need to tell it more about the Linode API (making it less generic and potentially requiring more maintenance long-term) or I'd need to build in more metadata to the OpenAPI spec (making the spec less generic and potentially harder to parse for other tools). I don't think the benefits outweigh the detriments here.

[lattwood]

Handling pagination makes for an unweildy shell script. Additionally, there's no pagination indicators when using the linode-cli with json.

[Dorthu]

I hadn't considered the lack of pagination metadata in json output 🤔 I think as it is we only include the data part of the response, but pagination metadata is contained within that same envelope.

I'm going to reopen this and think on it. If you have any ideas on how this might work, I'm open to them.

[lattwood]

I’m not sure there’s a straightforward way to allow for external control of pagination without changing the output shape of the data- I think it needs to be handled internally.

[displague]

https://www.transposit.com/devops-blog/itsm/2019.12.10-using-extensions-to-support-pagination-in-openapi/ explores this topic fairly well and discusses https://spec.openapis.org/oas/v3.0.0#link-object and https://spec.openapis.org/oas/v3.0.0#runtime-expressions.

It may help if the API returned the actual next page as a string URL. Tools like the LinodeCLI could then be instructed to follow this page. This is not generic. An OAS3 generic approach would demand some indicators about which response fields represent "next pages" (regardless of the name). It seems that capability does not exist in OAS3, as discussed in the article above.

If this pattern existed, it would need a method to indicate that all relevant and valid query parameters should be persisted in paginated URLs. (next_page = 'current_path?current_query&replacement_page')

In the Linode API, this may not be sufficient, however, since some filter parameters may be supplied as header instructions, unless those headers can be translated into query parameters in the API response. The Linode API would need to do this translation as OAS3 runtime expressions would likely not be capable of doing this client-side, especially with some of the URL components only being described in the request and not the response.

[displague]

Using Link rel="next" headers (rather than response body components) is intriguing:

• https://docs.github.com/en/rest/guides/traversing-with-pagination
• Question: Create Links from "complex" headers OAI/OpenAPI-Specification#2402

Rather than being an OAS3 feature, this is HTTP convention (https://datatracker.ietf.org/doc/html/rfc8288).

Also related to OAS3 header parsing: OAI/OpenAPI-Specification#1980

[Dorthu]

Instead of changing how the API includes pagination metadata or how the CLI interprets the spec, how about a built-in plugin that handles pagination for you?

linode-cli paginate linodes list 

A command like that might return all pages in the linodes list collection, requesting them in order and printing them in sequence. All normal CLI arguments could be accepted (and would be forwarded to the CLI when fetching pages), which should allow normal use of output format, filters, etc. If the action invoked this way isn't a paginated action, it's just handled normally with no error produced.

[zliang-akamai]

Implemented in #455