Stablize Rust-Native Completion Engine Tracking Issue

[epage]

Maintainer's notes:

Remaining work for feature parity

• Communicate with bash (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete path ValueHints (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete possible values (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete short flags (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete long flags (feat(complete): Skeleton for Rust-driven completions #3656)
• Complete subcommands (feat(complete): Skeleton for Rust-driven completions #3656)
• Communicate with zsh (zsh support for native completions #3916)
• Communicate with fish (fish support for native completions  #3917)
• Communicate with Powershell (powershell support for native completions  #3918)
• Communicate with Elvish (elvish support for native completions  #3919)
• Flag values (Support flags with values in native completions #3920)
• Multiple values (Support multiple values in native completions #3921)
• Hidden (Support hiding flags/subcommands/possible values/aliases for native compeletions #3951)
• Handling of space (Intentionally control when a space is appended for native completions #5587)
• Fix handling of OsString to preserve non-UTF8 paths
• Document that stdout should not be used before getting to completion processing in case the completions are requested (docs(complete): Expand dynamic docs #5643)
• FIGNORE support?
• Fulfill Document how to source completions #2488 for these new completions. Ideally, we have the registration script sourced on-demand in the docs so that it is "auto-updating" with the users tool
• Switch native completions to encourage env var initialization #3930
• Verify each shell's behavior with quoting to see if we need to add quoting to what we output
• clap_complete::env::Elvish generates code deprecated in 0.18, removed in 0.21 #5729
• Bash outputs everything as one completion, rather than a list of completions #5730
• Lazy loading of dynamic completions #5668

Non-blocking work

• Smarter selection of what to show (Filter and order completions based on input and importance for native completions #5058)
• Delimited values (Support delimited values for native completions #3922)
• is_require_equal_set support (Support is_require_equal_set in native completions #3923)
• last support
• trailing_var_arg support
• Support subcommand flags in native completion engine #5284 (doesn't appear supported by static completions)
• Removing options based on conflicts (self and others)
• Any App or Arg setting not currently handled by the static completions
  • Priority to value delimiter and requires equals
• Complete more ValueHints
• Figure out where this will live as the dependency requires are dramatically different than the static completions and this shouldn't be behind a feature flag for forever
• Custom help messages (Allow using a completion-specific candidate description over the candidate's help message in rust-native completions #5063)
• Completing multiple path elements at once (Complete multiple path elements at once #5279)
• nushell support (Dynamic completion for Nushell #5840)

Design considerations

• Make it easy to use for the natively supported shells but allow other shells to be supported

Prior art

• https://github.com/pacak/bpaf
• https://pypi.org/project/argcomplete
• https://github.com/rsteube/carapace
• https://github.com/microsoft/inshellisense
• https://github.com/sigoden/argc-completions
• https://click.palletsprojects.com/en/8.1.x/shell-completion/

---

#3022 was a tipping point for me in realizing that maybe our current approach to completions doesn't work. We effectively have to implement a mostly-untested parser within each shell. Examples of other problems that seem to stem from this:

• clap_generate fish: ArgEnum completions not working on toplevel command #2729
• Allow completion of partial commands [Powershell] #2750
• clap_generate PowerShell script does not complete case sensitive options like -S. Only completes -s #2145
• zsh: optional flag with takes_value breaks completions #1822
• Allow the completed command to be an alias #1764

If we take the approach of argcomplete where we do the parsing in our core code, rather than in each completion script, this will help us share parsing logic between shell, share some or all parsing logic with clap itself, and make a subset of the logic more testable.

We also need to decide whether to morph the existing parser into supporting this or create a custom parser (since the needs are pretty special). If we do a custom parser, we should probably do #2915 first so we can reuse lexing logic between clap and the completion generation. I've also been considering #2912 which would allow reusing the completion logic with any CLI parser.

[epage]

Bash's expectations

Inputs for -F and -C:

• COMP_LINE: The current command line. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_POINT: The index of the current cursor position relative to the beginning of the current command. If the current cursor position is at the end of the current command, the value of this variable is equal to ${#COMP_LINE}. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• COMP_KEY: The key (or final key of a key sequence) used to invoke the current completion function.
• COMP_TYPE: Set to an integer value corresponding to the type of completion attempted that caused a completion function to be called: TAB, for normal completion, ‘?’, for listing completions after successive tabs, ‘!’, for listing alternatives on partial word completion, ‘@’, to list completions if the word is not unmodified, or ‘%’, for menu completion. This variable is available only in shell functions and external commands invoked by the programmable completion facilities
• 1: name of the command whose arguments are being completed
• 2: word being completed
• 3: word preceding the word being completed on the current command line

Inputs for -F:

• COMP_WORDS, COMP_CWORD when used with -F

Output for -C:

• print a list of completions, one per line, to the standard output. Backslash may be used to escape a newline, if necessary.

Output for -F:

• It must put the possible completions in the COMPREPLY array variable, one per array element.

See

• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion.html#Programmable-Completion
• https://www.gnu.org/software/bash/manual/html_node/A-Programmable-Completion-Example.html#A-Programmable-Completion-Example
• https://www.gnu.org/software/bash/manual/html_node/Programmable-Completion-Builtins.html#Programmable-Completion-Builtins
• https://www.gnu.org/software/bash/manual/html_node/Bash-Variables.html

[epage]

argcomplete emulates bash's interface in fish by

    set -x COMP_LINE (commandline -p)
    set -x COMP_POINT (string length (commandline -cp))
    set -x COMP_TYPE

and then just registers that function

https://github.com/kislyuk/argcomplete/blob/develop/argcomplete/shell_integration.py#L60

[epage]

Value hints are supported on

• zsh
• fish

Tooltips are supported on

• zsh
• powershell
• elvish
• fish

We should make sure we don't lose these features as part of this transition, ie we shouldn't drop to the lowest common denominator.

[epage]

For Powershell

Register-ArgumentCompleter
        -CommandName <String[]>
        -ScriptBlock <ScriptBlock>
        [-Native]
        [<CommonParameters>]

https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/register-argumentcompleter?view=powershell-7.2

The block receives

When you specify the Native parameter, the script block must take the following parameters in the specified order. The names of the parameters aren't important because PowerShell passes in the values by position.

• $wordToComplete (Position 0) - This parameter is set to value the user has provided before they pressed Tab. Your script block should use this value to determine tab completion values.
• $commandAst (Position 1) - This parameter is set to the Abstract Syntax Tree (AST) for the current input line. For more information, see Ast Class.
• $cursorPosition (Position 2) - This parameter is set to the position of the cursor when the user pressed Tab.

The block provides CompletionResult

The CompletionResult object allows you to provide additional details to each returned value:

• completionText (String) - The text to be used as the auto completion result. This is the value sent to the command.
• listItemText (String) - The text to be displayed in a list, such as when the user presses Ctrl+Space. This is used for display only and is not passed to the command when selected.
• resultType (CompletionResultType) - The type of completion result.
• toolTip (String) - The text for the tooltip with details to be displayed about the object. This is visible when the user selects an item after pressing Ctrl+Space.

So it seems like Powershell can fit within rust-driven completions and provide the full feature set.

[epage]

I'm starting small and prototyping for just bash support. Looking at argcomplete, it seems they had to use their own shlex implementation. Hoping we can avoid that. The first step is comex/rust-shlex#12.