Incorrect order of CSI response

[leiserfg]

When a program sends a series of CSI to the terminal they are supposed to be handled in FIFO order, not the other way around, else is impossible to know when to stop reading for a response (as one does not know if such response will ever arrive).

I don't know if this affects every CSI, but with this ones is easy to reproduce: use sh to test it so the shell does not affect the messages:
This is what I get in kitty (the correct behavior)

sh-5.2$ printf "\033[?996n\033[c"     # This is what I typed, the second line is "typed" by the terminal
sh-5.2$ 997;1n62;c

This is what ghostty does:

sh-5.2$ printf "\033[?996n\033[c"     # This is what I typed, the second line is "typed" by the terminal
sh-5.2$ 62;22c997;2n

Notice that the response to \033[c is arriving before the response to "\033[?996n.

[leiserfg]

Other series including \033[c show that it gets replied first regardless of the order.

[leiserfg]

I stand corrected.

[leiserfg]

A sequence involving another CSI:
printf "\e[>0q\e[?996n\e[c"
in kitty:

:>|kitty(0.39.1)^[\^[[?997;1n^[[?62;c

in ghostty

:>|ghostty 1.1.2^[\^[[?62;22c^[[?997;2n

[mitchellh]

Is the guarantee of FIFO specified anywhere? Genuine question, because it would have some pretty significant implications. For example, reading from the clipboard is pretty slow (i.e. on macOS requires system APIs that have to happen on the main thread or with a lock) so we put that on a background job to respond "later" (often very fast, but definitely no guarantee of FIFO). Other queries such as system appearance could result in similar things.

FIFO requirements in general would make some performance tank, although admittedly in specific scenarios.

If its not specified, then I'd rather take a look at this at a per request basis.

cc @rockorager on this one

[leiserfg]

It's a common practice to send a series of CSI codes ending with [c so the response to that one can be used to know it's the end of the reply. Very useful because it's hard to know a priority if the terminal understands every one of the CSI sent, but they all understand [c

[leiserfg]

In case that "common practice" is not a strong enough argument, then consider at least adding [?996n to the ones handled in the same thread, the background of the terminal is internal to it so knowing if it's dark or not does not take very long, and there is no capability one can check to know if the terminal supports it or not.

For reference:
https://github.com/contour-terminal/contour/blob/master/examples/detect-dark-light-mode.cpp

[rockorager]

AFAIK, there is no specific requirement that terminals process in FIFO order. Obviously there are some expectations because sequences can change state. I think the guarantee stops there, and responses such as queries have no guaranteed order.

That said, I think it is helpful to follow the convention because of what you said - it's impossible to know when to stop waiting for a response unless you bracketed it with something else. In general I think your application should be designed such that it can react to the responses if they occur at any point, but this is impractical for short lived processes like some CLI tools.

The simplest is to respond in FIFO order. Ghostty could be a little smarter, and wait to respond to DA1 queries if it has no other pending queries. This is additional state to track, but maybe prevents some perf hit for cases where the queries aren't bracketed. This is kind of hacky, but should work.

@leiserfg another approach you could take is to query for the color scheme subscription mode. I believe all terminals that support 996 also support the subscription mode (2031).

[leiserfg]

another approach you could take is to query for the color scheme subscription mode. I believe all terminals that support 996 also support the subscription mode (2031).

I'm aware, but I'm working on rod a tool to determine the current lightness/darkness of the background to set the correct flags/enviroment variables and then execute programs that don't support this protocol, so I need to get this value right away. For instance, one can run rod run fzf and it will exec fzf -color=dark or fzf -color=light depending on the terminal background.
The tool works already, but I had to use \x1b[>0q to detect ghostty and then use a single [?996n as workaround, while the [?996n[c is the advised "combo" in contour's spec.

[the-mikedavis]

I have also noticed this while adding mode 2031 support for the Helix editor: helix-editor/helix#13223

Responding to requests in order is useful from the client/application side because it allows us to quickly query for a non-standard extension: we can query the current theme mode and then send a request for the primary device attributes (widely supported). If the terminal gives the primary device attribute response then we know it doesn't support mode 2031 - this is the recommendation for detecting the kitty keyboard protocol: https://sw.kovidgoyal.net/kitty/keyboard-protocol/#detection-of-support-for-this-protocol. If responses arrive out of order then we must query with a timeout. Specifically for terminal theme mode this increases time-to-first-render for us on terminals that don't support the query since choosing the wrong theme and then switching if CSI ? 997 ; _ n arrives can cause a "flash" of the wrong theme.

Note that this doesn't appear to affect Neovim because they do some extra querying of the background color to decide which theme to use (or so I believe - I'm not very familiar with that code).

I'm not sure if FIFO order makes sense for all requests/responses but specifically for mode 2031 and the CSI ? 996 n query, FIFO order would be very useful from the application side.

[mitchellh]

Note that these are already accepted to be put in order: #5922

I'm absolutely fine with one-off examples like this (and this is already accepted!), I was just curious about the general statement that all CSI responses should be in order, which is suspicious to me.

[the-mikedavis]

Ah I totally missed #5922, thank you!