UI Automation in Windows Console and Windows Terminal: block SV2M2 notification events for now and preserve accessibility after UIA class name change

[codeofdusk]

Link to issue number:

Related to microsoft/terminal#12358 among others (to be released).

Summary of the issue:

In upcoming Windows Terminal and Windows Console, UIA notification events will be sent when new text is inserted (i.e. for text output) to improve Narrator support. This will result in double-reporting of all terminal output: once from LiveText and once from UIA notifications (but without appropriate filtering for typed characters/passwords etc).

As part of the implementation of notifications, I asked Microsoft to change the UIA class name to allow new terminal support (including notifications) in a follow-up NVDA PR, since notifications will require a departure from the terminal strategy used by NVDA in the past. Changing the class name will break NVDA's current ability to identify the terminal and implement accessibility.

Description of how this pull request fixes the issue:

• Suppress any received UIA notification events in UIA console and terminal for now, but log them for development.
• Support the new UIA class name used by terminals that send notifications.

Ideally, NVDA would use these events in place of LiveText. If we could get away without registering for TextChange at all, this could be an extreme stability improvement in the terminal as #11002 would be completely circumvented. Also, having new text pushed to us (instead of having to diff) should improve performance considerably in high-volume scenarios.

Testing strategy:

Tested that support works as expected with the new terminal.

Known issues with pull request:

It is imperative that this PR be included in 2022.1, as upcoming builds of Windows Terminal will have severely degraded accessibility without it.

Change log entries:

None (workaround for bug in unreleased code)

Code Review Checklist:

• Pull Request description:
  • description is up to date
  • change log entries
• Testing:
  • Unit tests
  • System (end to end) tests
  • Manual testing
• API is compatible with existing add-ons.
• Documentation:
  • User Documentation
  • Developer / Technical Documentation
  • Context sensitive help for GUI changes
• UX of all users considered:
  • Speech
  • Braille
  • Low Vision
  • Different web browsers
  • Localization in other languages / culture than English

[josephsl]

Hi, for reference (for others): SV2 = Sun Valley 2 a.k.a. Windows 11 Version 22H2. I assume this was tested in build 22500 series, specifically 22533 and later? Thanks.

[codeofdusk]

Not yet, staying ahead of upcoming changes, though this PR doesn't break existing functionality and should be merged to ensure stability when the changes land.

CC @carlos-zamora, @DHowett.

[codeofdusk]

CC @seanbudd.

[seanbudd]

Is there a way to confirm that the console/terminal changes are guaranteed to be implemented?
Is there a way to test this PR with the proposed console/terminal changes?

[codeofdusk]

Is there a way to confirm that the console/terminal changes are guaranteed to be implemented?

"guarantee" is hard, but wt will become default, Narrator support is blocking, and notifications is the current path to enable this, so it is all but certain at this point.

Is there a way to test this PR with the proposed console/terminal changes?

I don't yet have access to a build with notifications available, but I've tested this event handler in other contexts that emit UIA notifications (Word with UIA) and it works as intended. Merging this PR is strictly better than the current behaviour and results in no change in situations where UIA notifications are not emitted.

[seanbudd]

I think it might be worth holding off on merging this until a Windows Insider build comes that it can be tested with.
Considering this is a small, low risk change, it can be merged into alpha late in the release cycle (eg just before we release beta).

[codeofdusk]

I'll work on obtaining a Windows Terminal build with which I can test this. Thanks for adding to the 2022.1 release.

[josephsl]

Hi, also, ping me when that build is available so I can test it on my VM (at least build 22543)). Thanks. From: Bill Dengler ***@***.***> Sent: Thursday, January 27, 2022 4:14 PM To: nvaccess/nvda ***@***.***> Cc: Joseph Lee ***@***.***>; Comment ***@***.***> Subject: Re: [nvaccess/nvda] UI Automation in Windows Console and Windows Terminal: block SV2M2 notification events for now (PR #13261) I'll work on obtaining a Windows Terminal build with which I can test this. Thanks for adding to the 2022.1 release. — Reply to this email directly, view it on GitHub <#13261 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4AXEA3ZKXWAEQ4BPKIKCTUYHNSXANCNFSM5MLNDXKQ> . You are receiving this because you commented.Message ID: ***@***.***>

[carlos-zamora]

Hi! I'm over from the Console Engineering team at Microsoft. I just wanted to follow up on some things from this thread.

wt will become default

Yes! We're looking to do this this year! Here's our blog post announcement: https://devblogs.microsoft.com/commandline/windows-terminal-as-your-default-command-line-experience/

Is there a way to confirm that the console/terminal changes are guaranteed to be implemented? Is there a way to test this PR with the proposed console/terminal changes?

We're making progress. I'll be working on creating a prototype this month and sharing that with Bill, NVDA, and JAWS (I'll be meeting with them next week to provide support). We're hoping that this'll be the fastest way to get Narrator on board since this change will have a pretty severe impact on Narrator users.

Like Bill, I hope to take a look at what other text providers (like Word) are doing. So if this works with Word already, that's very promising.

CC @DHowett

[codeofdusk]

I can confirm that, with the build in microsoft/terminal#12358, duplicate (buggy) output is present without this PR and absent (correct behaviour) with this PR.

[codeofdusk]

Now that this is known to work with an actual implementation, can this PR please be merged?

[seanbudd]

Thanks @codeofdusk. It's great that we can prepare for this change early, so that users won't have to update NVDA from 2022.1 to fix this.

Do we have an estimate at what time or which Windows releases this will be incorporated into?
I think it would be helpful to add a changelog item, as users on older versions of NVDA will want to search for which NVDA version fixes this bug.

[codeofdusk]

It will arrive in the Sun Valley 2 console and a Windows Store update. Can't speak to anything beyond that sadly. (there is precedent for making fixes to prepare for unreleased software without changelog items, see #7497).

[seanbudd]

@codeofdusk Thanks, perhaps for users "Windows 11" is specific enough. It is helpful to add bug fixes to the changelog where possible.
Would it be accurate to add the following:

- Prevents a bug causing double-reporting when using Windows 11 Console and Terminal. (#13261)

[codeofdusk]

I'd remove "11" as I think they'll backport the terminal code to vibranium at least. So just "Windows Console and Terminal".

[josephsl]

Hi, for reference, Vibranium = Windows 10, so a compromise could be to say “recent Windows 10 and 11” (I understand that the what’s new entry was committed, but hope this could be handy in the future). Thanks.

[seanbudd]

Hi, for reference, Vibranium = Windows 10, so a compromise could be to say “recent Windows 10 and 11” (I understand that the what’s new entry was committed, but hope this could be handy in the future). Thanks.

I think the problem here is that "recent" is relative and won't be helpful in the future. With the uncertainty surrounding which versions this will be included in, as @codeofdusk suggested potentially earlier builds to Vibranium, I opted to just go with "Windows".

[josephsl]

Hi, makes sense, thanks.

[codeofdusk]

Please note that, due to pushback from JAWS, the TermCTRL2 class name is currently not used. However NVDA should maintain support for it in case this changes in the future.