Use VT for COOKED_READ_DATA (#17445)

By rewriting `COOKED_READ_DATA` to use VT for its output we make it
possible to pass this VT output 1:1 straight to the hosting terminal
if we're running under ConPTY. This is also possible with the current
console APIs it uses, but it's somewhat janky. In particular the
usage of `ReadConsoleOutput` to backup/restore the popup contents
could be considered bad faith "rules for thee, not for me",
given that we're telling people to move away from those APIs.

The new implementation contains a bare bones "pager" to fit even
very long prompt contents into the VT viewport.
I fully expect this initial PR to not be entirely bug free, because
writing a proper pager with line wrapping is a little bit complex.
This PR takes some significant shortcuts by leveraging the fact
that the prompt line is always left-to-right and always a series
of fully filled lines followed by one potentially semi-full line.
This allows us to skip using a front/back-buffer for diffing the
contents between two redisplay calls.

Part of #14000

## Validation Steps Performed
* ASCII input
* Chinese input (中文維基百科) ✅
* Surrogate pair input (🙂) ✅
* In cmd.exe
  * Create 2 files: "a😊b.txt" and "a😟b.txt"
  * Press tab: Autocomplete to "a😊b.txt" ✅
  * Navigate the cursor right past the "a"
  * Press tab twice: Autocomplete to "a😟b.txt" ✅
* Execute `printf("    "); gets(buffer);` in C (or equivalent)
  * Press Tab, A, Ctrl+V, Tab, A ✅
  * The prompt is "        A^V     A" ✅
  * Cursor navigation works ✅
  * Backspacing/Deleting random parts of it works ✅
  * It never deletes the initial 4 spaces ✅
* Backspace deletes preceding glyphs ✅
* Ctrl+Backspace deletes preceding words ✅
* Escape clears input ✅
* Home navigates to start ✅
* Ctrl+Home deletes text between cursor and start ✅
* End navigates to end ✅
* Ctrl+End deletes text between cursor and end ✅
* Left navigates over previous code points ✅
* Ctrl+Left navigates to previous word-starts ✅
* Right and F1 navigate over next code points ✅
  * Pressing right at the end of input copies characters
    from the previous command ✅
* Ctrl+Right navigates to next word-ends ✅
* Insert toggles overwrite mode ✅
* Delete deletes next code point ✅
* Up and F5 cycle through history ✅
  * Doesn't crash with no history ✅
  * Stops at first entry ✅
* Down cycles through history ✅
  * Doesn't crash with no history ✅
  * Stops at last entry ✅
* PageUp retrieves the oldest command ✅
* PageDown retrieves the newest command ✅
* F2 starts "copy to char" prompt ✅
  * Escape dismisses prompt ✅
  * Typing a character copies text from the previous command up
    until that character into the current buffer (acts identical
    to F3, but with automatic character search) ✅
* F3 copies the previous command into the current buffer,
  starting at the current cursor position,
  for as many characters as possible ✅
  * Doesn't erase trailing text if the current buffer
    is longer than the previous command ✅
  * Puts the cursor at the end of the copied text ✅
* F4 starts "copy from char" prompt ✅
  * Escape dismisses prompt ✅
  * Erases text between the current cursor position and the
    first instance of a given char (but not including it) ✅
* F6 inserts Ctrl+Z ✅
* F7 without modifiers starts "command list" prompt ✅
  * Escape dismisses prompt ✅
  * Entries wider than the window width are truncated ✅
  * Height expands up to 20 rows with longer histories ✅
  * F9 starts "command number" prompt ✅
  * Left/Right replace the buffer with the given command ✅
    * And put cursor at the end of the buffer ✅
  * Up/Down navigate selection through history ✅
    * Stops at start/end with <10 entries ✅
    * Stops at start/end with >20 entries ✅
    * Scrolls through the entries if there are too many ✅
  * Shift+Up/Down moves history items around ✅
  * Home navigates to first entry ✅
  * End navigates to last entry ✅
  * PageUp navigates by $height items at a time or to first ✅
  * PageDown navigates by $height items at a time or to last ✅
* Alt+F7 clears command history ✅
* F8 cycles through commands that start with the same text as
  the current buffer up until the current cursor position ✅
  * Doesn't crash with no history ✅
* F9 starts "command number" prompt ✅
  * Escape dismisses prompt ✅
  * Ignores non-ASCII-decimal characters ✅
  * Allows entering between 1 and 5 digits ✅
  * Pressing Enter fetches the given command from the history ✅
* Alt+F10 clears doskey aliases ✅
* In cmd.exe, with an empty prompt in an empty directory:
  Pressing tab produces an audible bing and prints no text ✅
* When Narrator is enabled, in cmd.exe:
  * Typing individual characters announces only
    exactly each character that is being typed ✅
  * Backspacing at the end of a prompt announces
    only exactly each deleted character ✅

Author: lhecker

doc/COOKED_READ_DATA.md

@@ -4,12 +4,8 @@
 
 All of the following ✅ marks must be fulfilled during manual testing:
 * ASCII input
-* Chinese input (中文維基百科) ❔
-  * Resizing the window properly wraps/unwraps wide glyphs ❌
-    Broken due to `TextBuffer::Reflow` bugs
-* Surrogate pair input (🙂) ❔
-  * Resizing the window properly wraps/unwraps surrogate pairs ❌
-    Broken due to `TextBuffer::Reflow` bugs
+* Chinese input (中文維基百科) ✅
+* Surrogate pair input (🙂) ✅
 * In cmd.exe
   * Create 2 file: "a😊b.txt" and "a😟b.txt"
   * Press tab: Autocomplete to "a😊b.txt" ✅
@@ -62,21 +58,20 @@ All of the following ✅ marks must be fulfilled during manual testing:
 * F6 inserts Ctrl+Z ✅
 * F7 without modifiers starts "command list" prompt ✅
   * Escape dismisses prompt ✅
-  * Minimum size of 40x10 characters ✅
-  * Width expands to fit the widest history command ✅
+  * Entries wider than the window width are truncated ✅
   * Height expands up to 20 rows with longer histories ✅
   * F9 starts "command number" prompt ✅
-  * Left/Right paste replace the buffer with the given command ✅
+  * Left/Right replace the buffer with the given command ✅
     * And put cursor at the end of the buffer ✅
   * Up/Down navigate selection through history ✅
     * Stops at start/end with <10 entries ✅
     * Stops at start/end with >20 entries ✅
-    * Wide text rendering during pagination with >20 entries ✅
+    * Scrolls through the entries if there are too many ✅
   * Shift+Up/Down moves history items around ✅
   * Home navigates to first entry ✅
   * End navigates to last entry ✅
-  * PageUp navigates by 20 items at a time or to first ✅
-  * PageDown navigates by 20 items at a time or to last ✅
+  * PageUp navigates by $height items at a time or to first ✅
+  * PageDown navigates by $height items at a time or to last ✅
 * Alt+F7 clears command history ✅
 * F8 cycles through commands that start with the same text as
   the current buffer up until the current cursor position ✅

src/buffer/out/cursor.cpp

@@ -20,7 +20,6 @@ Cursor::Cursor(const ULONG ulSize, TextBuffer& parentBuffer) noexcept :
     _fBlinkingAllowed(true),
     _fDelay(false),
     _fIsConversionArea(false),
-    _fIsPopupShown(false),
     _fDelayedEolWrap(false),
     _fDeferCursorRedraw(false),
     _fHaveDeferredCursorRedraw(false),
@@ -66,11 +65,6 @@ bool Cursor::IsConversionArea() const noexcept
     return _fIsConversionArea;
 }
 
-bool Cursor::IsPopupShown() const noexcept
-{
-    return _fIsPopupShown;
-}
-
 bool Cursor::GetDelay() const noexcept
 {
     return _fDelay;
@@ -126,13 +120,6 @@ void Cursor::SetIsConversionArea(const bool fIsConversionArea) noexcept
     _RedrawCursorAlways();
 }
 
-void Cursor::SetIsPopupShown(const bool fIsPopupShown) noexcept
-{
-    // Functionally the same as "Hide cursor"
-    _fIsPopupShown = fIsPopupShown;
-    _RedrawCursorAlways();
-}
-
 void Cursor::SetDelay(const bool fDelay) noexcept
 {
     _fDelay = fDelay;

src/buffer/out/cursor.h

@@ -44,7 +44,6 @@ class Cursor final
     bool IsBlinkingAllowed() const noexcept;
     bool IsDouble() const noexcept;
     bool IsConversionArea() const noexcept;
-    bool IsPopupShown() const noexcept;
     bool GetDelay() const noexcept;
     ULONG GetSize() const noexcept;
     til::point GetPosition() const noexcept;
@@ -61,7 +60,6 @@ class Cursor final
     void SetBlinkingAllowed(const bool fIsOn) noexcept;
     void SetIsDouble(const bool fIsDouble) noexcept;
     void SetIsConversionArea(const bool fIsConversionArea) noexcept;
-    void SetIsPopupShown(const bool fIsPopupShown) noexcept;
     void SetDelay(const bool fDelay) noexcept;
     void SetSize(const ULONG ulSize) noexcept;
     void SetStyle(const ULONG ulSize, const CursorType type) noexcept;
@@ -99,7 +97,6 @@ class Cursor final
     bool _fBlinkingAllowed; //Whether or not the cursor is allowed to blink at all. only set through VT (^[[?12h/l)
     bool _fDelay; // don't blink scursor on next timer message
     bool _fIsConversionArea; // is attached to a conversion area so it doesn't actually need to display the cursor.
-    bool _fIsPopupShown; // if a popup is being shown, turn off, stop blinking.
 
     bool _fDelayedEolWrap; // don't wrap at EOL till the next char comes in.
     til::point _coordDelayedAt; // coordinate the EOL wrap was delayed at.

src/cascadia/TerminalCore/terminalrenderdata.cpp

@@ -48,7 +48,7 @@ til::point Terminal::GetCursorPosition() const noexcept
 bool Terminal::IsCursorVisible() const noexcept
 {
     const auto& cursor = _activeBuffer().GetCursor();
-    return cursor.IsVisible() && !cursor.IsPopupShown();
+    return cursor.IsVisible();
 }
 
 bool Terminal::IsCursorOn() const noexcept

src/host/VtIo.cpp

@@ -518,3 +518,66 @@ bool VtIo::IsResizeQuirkEnabled() const
     }
     return S_OK;
 }
+
+// Formats the given console attributes to their closest VT equivalent.
+// `out` must refer to at least `formatAttributesMaxLen` characters of valid memory.
+// Returns a pointer past the end.
+static constexpr size_t formatAttributesMaxLen = 16;
+static char* formatAttributes(char* out, const TextAttribute& attributes) noexcept
+{
+    static uint8_t sgr[] = { 30, 31, 32, 33, 34, 35, 36, 37, 90, 91, 92, 93, 94, 95, 96, 97 };
+
+    // Applications expect that SetConsoleTextAttribute() completely replaces whatever attributes are currently set,
+    // including any potential VT-exclusive attributes. Since we don't know what those are, we must always emit a SGR 0.
+    // Copying 4 bytes instead of the correct 3 means we need just 1 DWORD mov. Neat.
+    //
+    // 3 bytes.
+    memcpy(out, "\x1b[0", 4);
+    out += 3;
+
+    // 2 bytes.
+    if (attributes.IsReverseVideo())
+    {
+        memcpy(out, ";7", 2);
+        out += 2;
+    }
+
+    // 3 bytes (";97").
+    if (attributes.GetForeground().IsLegacy())
+    {
+        const uint8_t index = sgr[attributes.GetForeground().GetIndex()];
+        out = fmt::format_to(out, FMT_COMPILE(";{}"), index);
+    }
+
+    // 4 bytes (";107").
+    if (attributes.GetBackground().IsLegacy())
+    {
+        const uint8_t index = sgr[attributes.GetBackground().GetIndex()] + 10;
+        out = fmt::format_to(out, FMT_COMPILE(";{}"), index);
+    }
+
+    // 1 byte.
+    *out++ = 'm';
+    return out;
+}
+
+void VtIo::FormatAttributes(std::string& target, const TextAttribute& attributes)
+{
+    char buf[formatAttributesMaxLen];
+    const size_t len = formatAttributes(&buf[0], attributes) - &buf[0];
+    target.append(buf, len);
+}
+
+void VtIo::FormatAttributes(std::wstring& target, const TextAttribute& attributes)
+{
+    char buf[formatAttributesMaxLen];
+    const size_t len = formatAttributes(&buf[0], attributes) - &buf[0];
+
+    wchar_t bufW[formatAttributesMaxLen];
+    for (size_t i = 0; i < len; i++)
+    {
+        bufW[i] = buf[i];
+    }
+
+    target.append(bufW, len);
+}

src/host/VtIo.hpp

@@ -20,6 +20,9 @@ namespace Microsoft::Console::VirtualTerminal
     class VtIo
     {
     public:
+        static void FormatAttributes(std::string& target, const TextAttribute& attributes);
+        static void FormatAttributes(std::wstring& target, const TextAttribute& attributes);
+
         VtIo();
 
         [[nodiscard]] HRESULT Initialize(const ConsoleArguments* const pArgs);

src/host/_stream.cpp

@@ -255,6 +255,13 @@ void WriteCharsLegacy(SCREEN_INFORMATION& screenInfo, const std::wstring_view& t
     }
 }
 
+// This is the main entrypoint for conhost to write VT to the buffer.
+// This wrapper around StateMachine exists so that we can add the necessary ConPTY transformations.
+void WriteCharsVT(SCREEN_INFORMATION& screenInfo, const std::wstring_view& str)
+{
+    screenInfo.GetStateMachine().ProcessString(str);
+}
+
 // Routine Description:
 // - Takes the given text and inserts it into the given screen buffer.
 // Note:

src/host/_stream.h

@@ -20,6 +20,7 @@ Revision History:
 #include "writeData.hpp"
 
 void WriteCharsLegacy(SCREEN_INFORMATION& screenInfo, const std::wstring_view& str, til::CoordType* psScrollY);
+void WriteCharsVT(SCREEN_INFORMATION& screenInfo, const std::wstring_view& str);
 
 // NOTE: console lock must be held when calling this routine
 // String has been translated to unicode at this point.