Strange behavior with ENABLE_VIRTUAL_TERMINAL_INPUT | ENABLE_LINE_INPUT | ENABLE_PROCESSED_INPUT

[alexrp]

Environment

Windows build number: Microsoft Windows [Version 10.0.19041.153]
PowerShell Core 7.0.0
.NET Core 3.1.200

Steps to reproduce

C# program:

using System;
using System.Diagnostics;
using System.Text;
using static Vanara.PInvoke.Kernel32;

namespace Test
{
    static class Program
    {
        static void Main()
        {
            var stdin = GetStdHandle(StdHandleType.STD_INPUT_HANDLE);

            Debug.Assert(GetConsoleMode(stdin, out CONSOLE_INPUT_MODE inMode));

            inMode |= CONSOLE_INPUT_MODE.ENABLE_VIRTUAL_TERMINAL_INPUT;

            Debug.Assert(SetConsoleMode(stdin, inMode));

            Console.WriteLine(Console.ReadLine());
        }
    }
}

(Uses the Vanara.PInvoke.Kernel32 NuGet package for convenience.)

You will need to test this program in both CMD and PowerShell Core.

Try doing these:

1. Type aaa bbb ccc and press backspace a couple times.
2. Type aaa bbb ccc and press control + backspace several times.

Also, try using special keys (such as arrow keys) after the program closes.

Actual behavior

For the first part:

1. Words are deleted with each backspace press.
2. Characters are deleted with each control + backspace press.

This behavior happens in both CMD and PowerShell Core.

For the second part:

After closing the program, you'll notice that e.g. using the arrow keys echoes VT sequences (such as ^[[D^[[A^[[C for left, up, right) rather than doing what they're supposed to. This behavior persists even if the ReadLine + WriteLine is removed from the program.

This behavior only happens in CMD. PowerShell Core seems fine.

Expected behavior

I have no idea to what extent, if any, this behavior is actually buggy/unintended. But here is what I would expect as a user:

• Backspace and control + backspace behavior should not be swapped like this. This seems very strange and unlike other terminals.
• Console modes set from within a program should not persist after that program exits. PowerShell Core appears to get this right, while CMD does not, for whatever reason.

Perhaps worth noting that running CMD and PowerShell Core under Windows Terminal has no impact on the behavior seen here.

Also, I wasn't sure whether to file this as two separate issues. I can do that if necessary.

[DHowett-MSFT]

Thanks for doing all the legwork here! This is, unfortunately, to a fairly great extent "by design." The input and output modes are global state, and while we're thinking about what it would take to change that we're definitely concerned about backwards compatibility.

When an application exits and leaves the console in a state different from the one it has when it launched (which can happen for a number of reasons: it wasn't authored correctly, it crashes, it returns control of the console so the shell before it's ready (#4921)), we have little choice but to get into a weird state.

You see this with most terminals on the market, unfortunately. If vim is running in mouse mode, and is abruptly terminated, suddenly moving your mouse over your terminal generates an absolute heap of garbage input. It's impossible to solve in that case, but given that we own the console subsystem in its entirety we can put together a better story.

I'm going to close this one as a /dupe of #4954, which we'll use to track the overall scenario.

Hi! We've identified this issue as a duplicate of another one that already exists on this Issue Tracker. This specific instance is being closed in favor of tracking the concern over on the referenced thread. Thanks for your report!

[alexrp]

Thanks for the response @DHowett-MSFT!

I just wanted to clarify a few things:

For the mode issue, my understanding (after reading the linked issues) is that the reason my particular sample here works properly in PowerShell is that PowerShell actually makes an effort to set a sensible console mode before/after running a program, while CMD does not. Is that correct?

If so, then I suppose the best I can do right now is try really hard to set the console mode back to its original state, even in the case of a crash, to make things work under CMD?

Regarding the backspace issue I also brought up here, as far as I can tell, this isn't explained by the linked issues. To be clear, this issue happens even if you run the sample program from a 'clean' console.

[DHowett-MSFT]

while CMD does not

Exactly this.

The backspace issue is an intricacy of how backspace is encoded in ENABLE_VIRTUAL_TERMINAL_INPUT mode, actually. When an application requests VT input, backspace is translated as 0x7f (DEL) instead of 0x8 (BS). Ctrl+Backspace is, then, translated as 0x8 (BS) instead of 0x7f (DEL). It's simply a matter of coincidence that they're flipped, but COOKED_READ then can't tell if the user actually pressed ctrl+backspace or it's only receiving 0x7f because it's in VT input mode.

[DHowett-MSFT]

(because COOKED_READ isn't supposed to be used in VT input mode)

[alexrp]

What does COOKED_READ refer to here?