Skip to content

Commit 3b9f446

Browse files
committed
cli: replace clap with lexopt and supporting code
ripgrep began it's life with docopt for argument parsing. Then it moved to Clap and stayed there for a number of years. Clap has served ripgrep well, and it probably could continue to serve ripgrep well, but I ended up deciding to move off of it. Why? The first time I had the thought of moving off of Clap was during the 2->3->4 transition. I thought the 3.x and 4.x releases were great, but for me, it ended up moving a little too quickly. Since the release of 4.x was telegraphed around when 3.x came out, I decided to just hold off and wait to migrate to 4.x instead of doing a 3.x migration followed shortly by another 4.x migration. Of course, I just never ended up doing the migration at all. I never got around to it and there just wasn't a compelling reason for me to upgrade. While I never investigated it, I saw an upgrade as a non-trivial amount of work in part because I didn't encapsulate the usage of Clap enough. The above is just what got me started thinking about it. It wasn't enough to get me to move off of it on its own. What ended up pushing me over the edge was a combination of factors: * As mentioned above, I didn't want to run on the migration treadmill. This has proven to not be much of an issue, but at the time of the 2->3->4 releases, I didn't know how long Clap 4.x would be out before a 5.x would come out. * The release of lexopt[1] caught my eye. IMO, that crate demonstrates exactly how something new can arrive on the scene and just thoroughly solve a problem minimalistically. It has the docs, the reasoning, the simple API, the tests and good judgment. It gets all the weird corner cases right that Clap also gets right (and is part of why I was originally attracted to Clap). * I have an overall desire to reduce the size of my dependency tree. In part because a smaller dependency tree tends to correlate with better compile times, but also in part because it reduces my reliance and trust on others. It lets me be the "master" of ripgrep's destiny by reducing the amount of behavior that is the result of someone else's decision (whether good or bad). * I perceived that Clap solves a more general problem than what I actually need solved. Despite the vast number of flags that ripgrep has, its requirements are actually pretty simple. We just need simple switches and flags that support one value. No multi-value flags. No sub-commands. And probably a lot of other functionality that Clap has that makes it so flexible for so many different use cases. (I'm being hand wavy on the last point.) With all that said, perhaps most importantly, the future of ripgrep possibly demands a more flexible CLI argument parser. In today's world, I would really like, for example, flags like `--type` and `--type-not` to be able to accumulate their repeated values into a single sequence while respecting the order they appear on the CLI. For example, prior to this migration, `rg regex-automata -Tlock -ttoml` would not return results in `Cargo.lock` in this repository because the `-Tlock` always took priority even though `-ttoml` appeared after it. But with this migration, `-ttoml` now correctly overrides `-Tlock`. We would like to do similar things for `-g/--glob` and `--iglob` and potentially even now introduce a `-G/--glob-not` flag instead of requiring users to use `!` to negate a glob. (Which I had done originally to work-around this problem.) And some day, I'd like to add some kind of boolean matching to ripgrep perhaps similar to how `git grep` does it. (Although I haven't thought too carefully on a design yet.) In order to do that, I perceive it would be difficult to implement correctly in Clap. I believe that this last point is possible to implement correctly in Clap 2.x, although it is awkward to do so. I have not looked closely enough at the Clap 4.x API to know whether it's still possible there. In any case, these were enough reasons to move off of Clap and own more of the argument parsing process myself. This did require a few things: * I had to write my own logic for how arguments are combined into one single state object. Of course, I wanted this. This was part of the upside. But it's still code I didn't have to write for Clap. * I had to write my own shell completion generator. * I had to write my own `-h/--help` output generator. * I also had to write my own man page generator. Well, I had to do this with Clap 2.x too, although my understanding is that Clap 4.x supports this. With that said, without having tried it, my guess is that I probably wouldn't have liked the output it generated because I ultimately had to write most of the roff by hand myself to get the man page I wanted. (This also had the benefit of dropping the build dependency on asciidoc/asciidoctor.) While this is definitely a fair bit of extra work, it overall only cost me a couple days. IMO, that's a good trade off given that this code is unlikely to change again in any substantial way. And it should also allow for more flexible semantics going forward. Fixes #884, Fixes #1648, Fixes #1701, Fixes #1814, Fixes #1966 [1]: https://docs.rs/lexopt/0.3.0/lexopt/index.html
1 parent dbc62fe commit 3b9f446

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+12724
-6141
lines changed

.github/workflows/ci.yml

+9-24
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ jobs:
5353
include:
5454
- build: pinned
5555
os: ubuntu-latest
56-
rust: 1.72.1
56+
rust: 1.74.0
5757
- build: stable
5858
os: ubuntu-latest
5959
rust: stable
@@ -94,7 +94,7 @@ jobs:
9494
rust: nightly-x86_64-gnu
9595
steps:
9696
- name: Checkout repository
97-
uses: actions/checkout@v3
97+
uses: actions/checkout@v4
9898

9999
- name: Install packages (Ubuntu)
100100
if: matrix.os == 'ubuntu-latest'
@@ -131,6 +131,7 @@ jobs:
131131
run: |
132132
echo "cargo command is: ${{ env.CARGO }}"
133133
echo "target flag is: ${{ env.TARGET_FLAGS }}"
134+
echo "target dir is: ${{ env.TARGET_DIR }}"
134135
135136
- name: Build ripgrep and all crates
136137
run: ${{ env.CARGO }} build --verbose --workspace ${{ env.TARGET_FLAGS }}
@@ -164,26 +165,6 @@ jobs:
164165
if: matrix.target != ''
165166
run: ${{ env.CARGO }} test --verbose --workspace ${{ env.TARGET_FLAGS }}
166167

167-
- name: Test for existence of build artifacts (Windows)
168-
if: matrix.os == 'windows-2022'
169-
shell: bash
170-
run: |
171-
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
172-
ls "$outdir/_rg.ps1" && file "$outdir/_rg.ps1"
173-
174-
- name: Test for existence of build artifacts (Unix)
175-
if: matrix.os != 'windows-2022'
176-
shell: bash
177-
run: |
178-
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
179-
# TODO: Check for the man page generation here. For whatever reason,
180-
# it seems to be intermittently failing in CI. No idea why.
181-
# for f in rg.bash rg.fish rg.1; do
182-
for f in rg.bash rg.fish; do
183-
# We could use file -E here, but it isn't supported on macOS.
184-
ls "$outdir/$f" && file "$outdir/$f"
185-
done
186-
187168
- name: Test zsh shell completions (Unix, sans cross)
188169
# We could test this when using Cross, but we'd have to execute the
189170
# 'rg' binary (done in test-complete) with qemu, which is a pain and
@@ -197,11 +178,15 @@ jobs:
197178
shell: bash
198179
run: ${{ env.CARGO }} test --manifest-path crates/cli/Cargo.toml ${{ env.TARGET_FLAGS }} --lib print_hostname -- --nocapture
199180

181+
- name: Print available short flags
182+
shell: bash
183+
run: ${{ env.CARGO }} test --bin rg ${{ env.TARGET_FLAGS }} flags::defs::tests::available_shorts -- --nocapture
184+
200185
rustfmt:
201186
runs-on: ubuntu-latest
202187
steps:
203188
- name: Checkout repository
204-
uses: actions/checkout@v3
189+
uses: actions/checkout@v4
205190
- name: Install Rust
206191
uses: dtolnay/rust-toolchain@master
207192
with:
@@ -214,7 +199,7 @@ jobs:
214199
runs-on: ubuntu-latest
215200
steps:
216201
- name: Checkout repository
217-
uses: actions/checkout@v3
202+
uses: actions/checkout@v4
218203
- name: Install Rust
219204
uses: dtolnay/rust-toolchain@master
220205
with:

.github/workflows/release.yml

+164-69
Original file line numberDiff line numberDiff line change
@@ -1,47 +1,39 @@
1-
# The way this works is the following:
2-
#
3-
# The create-release job runs purely to initialize the GitHub release itself
4-
# and to output upload_url for the following job.
5-
#
6-
# The build-release job runs only once create-release is finished. It gets the
7-
# release upload URL from create-release job outputs, then builds the release
8-
# executables for each supported platform and attaches them as release assets
9-
# to the previously created release.
10-
#
11-
# The key here is that we create the release only once.
12-
#
13-
# Reference:
14-
# https://eugene-babichenko.github.io/blog/2020/05/09/github-actions-cross-platform-auto-releases/
15-
161
name: release
2+
3+
# Only do the release on x.y.z tags.
174
on:
185
push:
19-
# Enable when testing release infrastructure on a branch.
20-
# branches:
21-
# - ag/work
226
tags:
237
- "[0-9]+.[0-9]+.[0-9]+"
8+
9+
# We need this to be able to create releases.
10+
permissions:
11+
contents: write
12+
2413
jobs:
14+
# The create-release job runs purely to initialize the GitHub release itself,
15+
# and names the release after the `x.y.z` tag that was pushed. It's separate
16+
# from building the release so that we only create the release once.
2517
create-release:
2618
name: create-release
2719
runs-on: ubuntu-latest
2820
# env:
2921
# Set to force version number, e.g., when no tag exists.
30-
# RG_VERSION: TEST-0.0.0
31-
outputs:
32-
rg_version: ${{ env.RG_VERSION }}
22+
# VERSION: TEST-0.0.0
3323
steps:
34-
- uses: actions/checkout@v3
24+
- uses: actions/checkout@v4
3525
- name: Get the release version from the tag
36-
shell: bash
37-
if: env.RG_VERSION == ''
26+
if: env.VERSION == ''
27+
run: echo "VERSION=${{ github.ref_name }}" >> $GITHUB_ENV
28+
- name: Show the version
3829
run: |
39-
echo "RG_VERSION=$GITHUB_REF_NAME" >> $GITHUB_ENV
40-
echo "version is: ${{ env.RG_VERSION }}"
30+
echo "version is: $VERSION"
4131
- name: Create GitHub release
4232
env:
43-
GH_TOKEN: ${{ github.token }}
44-
run: gh release create ${{ env.RG_VERSION }}
33+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
34+
run: gh release create $VERSION --draft --verify-tag --title $VERSION
35+
outputs:
36+
version: ${{ env.VERSION }}
4537

4638
build-release:
4739
name: build-release
@@ -52,25 +44,50 @@ jobs:
5244
# systems.
5345
CARGO: cargo
5446
# When CARGO is set to CROSS, this is set to `--target matrix.target`.
55-
TARGET_FLAGS: ""
47+
TARGET_FLAGS:
5648
# When CARGO is set to CROSS, TARGET_DIR includes matrix.target.
5749
TARGET_DIR: ./target
50+
# Bump this as appropriate. We pin to a version to make sure CI
51+
# continues to work as cross releases in the past have broken things
52+
# in subtle ways.
53+
CROSS_VERSION: v0.2.5
5854
# Emit backtraces on panics.
5955
RUST_BACKTRACE: 1
6056
# Build static releases with PCRE2.
6157
PCRE2_SYS_STATIC: 1
6258
strategy:
59+
fail-fast: false
6360
matrix:
64-
build: [linux, linux-arm, macos, win-msvc, win-gnu, win32-msvc]
6561
include:
6662
- build: linux
6763
os: ubuntu-latest
6864
rust: nightly
6965
target: x86_64-unknown-linux-musl
70-
- build: linux-arm
66+
strip: x86_64-linux-musl-strip
67+
- build: stable-x86
7168
os: ubuntu-latest
72-
rust: nightly
73-
target: arm-unknown-linux-gnueabihf
69+
rust: stable
70+
target: i686-unknown-linux-gnu
71+
strip: x86_64-linux-gnu-strip
72+
qemu: i386
73+
- build: stable-aarch64
74+
os: ubuntu-latest
75+
rust: stable
76+
target: aarch64-unknown-linux-gnu
77+
strip: aarch64-linux-gnu-strip
78+
qemu: qemu-aarch64
79+
- build: stable-powerpc64
80+
os: ubuntu-latest
81+
rust: stable
82+
target: powerpc64-unknown-linux-gnu
83+
strip: powerpc64-linux-gnu-strip
84+
qemu: qemu-ppc64
85+
- build: stable-s390x
86+
os: ubuntu-latest
87+
rust: stable
88+
target: s390x-unknown-linux-gnu
89+
strip: s390x-linux-gnu-strip
90+
qemu: qemu-s390x
7491
- build: macos
7592
os: macos-latest
7693
rust: nightly
@@ -90,15 +107,17 @@ jobs:
90107

91108
steps:
92109
- name: Checkout repository
93-
uses: actions/checkout@v3
110+
uses: actions/checkout@v4
94111

95112
- name: Install packages (Ubuntu)
96113
if: matrix.os == 'ubuntu-latest'
114+
shell: bash
97115
run: |
98116
ci/ubuntu-install-packages
99117
100118
- name: Install packages (macOS)
101119
if: matrix.os == 'macos-latest'
120+
shell: bash
102121
run: |
103122
ci/macos-install-packages
104123
@@ -109,64 +128,140 @@ jobs:
109128
target: ${{ matrix.target }}
110129

111130
- name: Use Cross
131+
if: matrix.os == 'ubuntu-latest' && matrix.target != ''
112132
shell: bash
113133
run: |
114-
cargo install cross
134+
# In the past, new releases of 'cross' have broken CI. So for now, we
135+
# pin it. We also use their pre-compiled binary releases because cross
136+
# has over 100 dependencies and takes a bit to compile.
137+
dir="$RUNNER_TEMP/cross-download"
138+
mkdir "$dir"
139+
echo "$dir" >> $GITHUB_PATH
140+
cd "$dir"
141+
curl -LO "https://github.com/cross-rs/cross/releases/download/$CROSS_VERSION/cross-x86_64-unknown-linux-musl.tar.gz"
142+
tar xf cross-x86_64-unknown-linux-musl.tar.gz
115143
echo "CARGO=cross" >> $GITHUB_ENV
144+
145+
- name: Set target variables
146+
shell: bash
147+
run: |
116148
echo "TARGET_FLAGS=--target ${{ matrix.target }}" >> $GITHUB_ENV
117149
echo "TARGET_DIR=./target/${{ matrix.target }}" >> $GITHUB_ENV
118150
119151
- name: Show command used for Cargo
152+
shell: bash
120153
run: |
121154
echo "cargo command is: ${{ env.CARGO }}"
122155
echo "target flag is: ${{ env.TARGET_FLAGS }}"
123156
echo "target dir is: ${{ env.TARGET_DIR }}"
124157
125158
- name: Build release binary
126-
run: ${{ env.CARGO }} build --verbose --release --features pcre2 ${{ env.TARGET_FLAGS }}
159+
shell: bash
160+
run: |
161+
${{ env.CARGO }} build --verbose --release --features pcre2 ${{ env.TARGET_FLAGS }}
162+
if [ "${{ matrix.os }}" = "windows-latest" ]; then
163+
bin="target/${{ matrix.target }}/release/rg.exe"
164+
else
165+
bin="target/${{ matrix.target }}/release/rg"
166+
fi
167+
echo "BIN=$bin" >> $GITHUB_ENV
127168
128-
- name: Strip release binary (linux, macos and macos-arm)
129-
if: matrix.build == 'linux' || matrix.os == 'macos'
130-
run: strip "target/${{ matrix.target }}/release/rg"
169+
- name: Strip release binary (macos)
170+
if: matrix.os == 'macos'
171+
shell: bash
172+
run: strip "$BIN"
131173

132-
- name: Strip release binary (arm)
133-
if: matrix.build == 'linux-arm'
174+
- name: Strip release binary (cross)
175+
if: env.CARGO == 'cross'
176+
shell: bash
134177
run: |
135178
docker run --rm -v \
136179
"$PWD/target:/target:Z" \
137-
rustembedded/cross:arm-unknown-linux-gnueabihf \
138-
arm-linux-gnueabihf-strip \
139-
/target/arm-unknown-linux-gnueabihf/release/rg
180+
"rustembedded/cross:${{ matrix.target }}" \
181+
"${{ matrix.strip }}" \
182+
"/target/${{ matrix.target }}/release/rg"
140183
141-
- name: Build archive
184+
- name: Determine archive name
142185
shell: bash
143186
run: |
144-
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
145-
staging="ripgrep-${{ needs.create-release.outputs.rg_version }}-${{ matrix.target }}"
146-
mkdir -p "$staging"/{complete,doc}
187+
version="${{ needs.create-release.outputs.version }}"
188+
echo "ARCHIVE=ripgrep-$version-${{ matrix.target }}" >> $GITHUB_ENV
147189
148-
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$staging/"
149-
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$staging/doc/"
150-
cp "$outdir"/{rg.bash,rg.fish,_rg.ps1} "$staging/complete/"
151-
cp complete/_rg "$staging/complete/"
190+
- name: Creating directory for archive
191+
shell: bash
192+
run: |
193+
mkdir -p "$ARCHIVE"/{complete,doc}
194+
cp "$BIN" "$ARCHIVE"/
195+
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$ARCHIVE"/
196+
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$ARCHIVE"/doc/
152197
153-
if [ "${{ matrix.os }}" = "windows-latest" ]; then
154-
cp "target/${{ matrix.target }}/release/rg.exe" "$staging/"
155-
7z a "$staging.zip" "$staging"
156-
certutil -hashfile "$staging.zip" SHA256 > "$staging.zip.sha256"
157-
echo "ASSET=$staging.zip" >> $GITHUB_ENV
158-
echo "ASSET_SUM=$staging.zip.sha256" >> $GITHUB_ENV
159-
else
160-
# The man page is only generated on Unix systems. ¯\_(ツ)_/¯
161-
cp "$outdir"/rg.1 "$staging/doc/"
162-
cp "target/${{ matrix.target }}/release/rg" "$staging/"
163-
tar czf "$staging.tar.gz" "$staging"
164-
shasum -a 256 "$staging.tar.gz" > "$staging.tar.gz.sha256"
165-
echo "ASSET=$staging.tar.gz" >> $GITHUB_ENV
166-
echo "ASSET_SUM=$staging.tar.gz.sha256" >> $GITHUB_ENV
167-
fi
198+
- name: Generate man page and completions (no emulation)
199+
if: matrix.qemu == ''
200+
shell: bash
201+
run: |
202+
"$BIN" --version
203+
"$BIN" --generate complete-bash > "$ARCHIVE/complete/rg.bash"
204+
"$BIN" --generate complete-fish > "$ARCHIVE/complete/rg.fish"
205+
"$BIN" --generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
206+
"$BIN" --generate complete-zsh > "$ARCHIVE/complete/_rg"
207+
"$BIN" --generate man > "$ARCHIVE/doc/rg.1"
208+
209+
- name: Generate man page and completions (emulation)
210+
if: matrix.qemu != ''
211+
shell: bash
212+
run: |
213+
docker run --rm -v \
214+
"$PWD/target:/target:Z" \
215+
"rustembedded/cross:${{ matrix.target }}" \
216+
"${{ matrix.qemu }}" "/$BIN" --version
217+
docker run --rm -v \
218+
"$PWD/target:/target:Z" \
219+
"rustembedded/cross:${{ matrix.target }}" \
220+
"${{ matrix.qemu }}" "/$BIN" \
221+
--generate complete-bash > "$ARCHIVE/complete/rg.bash"
222+
docker run --rm -v \
223+
"$PWD/target:/target:Z" \
224+
"rustembedded/cross:${{ matrix.target }}" \
225+
"${{ matrix.qemu }}" "/$BIN" \
226+
--generate complete-fish > "$ARCHIVE/complete/rg.fish"
227+
docker run --rm -v \
228+
"$PWD/target:/target:Z" \
229+
"rustembedded/cross:${{ matrix.target }}" \
230+
"${{ matrix.qemu }}" "/$BIN" \
231+
--generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
232+
docker run --rm -v \
233+
"$PWD/target:/target:Z" \
234+
"rustembedded/cross:${{ matrix.target }}" \
235+
"${{ matrix.qemu }}" "/$BIN" \
236+
--generate complete-zsh > "$ARCHIVE/complete/_rg"
237+
docker run --rm -v \
238+
"$PWD/target:/target:Z" \
239+
"rustembedded/cross:${{ matrix.target }}" \
240+
"${{ matrix.qemu }}" "/$BIN" \
241+
--generate man > "$ARCHIVE/doc/rg.1"
242+
243+
- name: Build archive (Windows)
244+
shell: bash
245+
if: matrix.os == 'windows-latest'
246+
run: |
247+
7z a "$ARCHIVE.zip" "$ARCHIVE"
248+
certutil -hashfile "$ARCHIVE.zip" SHA256 > "$ARCHIVE.zip.sha256"
249+
echo "ASSET=$ARCHIVE.zip" >> $GITHUB_ENV
250+
echo "ASSET_SUM=$ARCHIVE.zip.sha256" >> $GITHUB_ENV
251+
252+
- name: Build archive (Unix)
253+
shell: bash
254+
if: matrix.os != 'windows-latest'
255+
run: |
256+
tar czf "$ARCHIVE.tar.gz" "$ARCHIVE"
257+
shasum -a 256 "$ARCHIVE.tar.gz" > "$ARCHIVE.tar.gz.sha256"
258+
echo "ASSET=$ARCHIVE.tar.gz" >> $GITHUB_ENV
259+
echo "ASSET_SUM=$ARCHIVE.tar.gz.sha256" >> $GITHUB_ENV
168260
169261
- name: Upload release archive
170262
env:
171-
GH_TOKEN: ${{ github.token }}
172-
run: gh release upload ${{ needs.create-release.outputs.rg_version }} ${{ env.ASSET }} ${{ env.ASSET_SUM }}
263+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
264+
shell: bash
265+
run: |
266+
version="${{ needs.create-release.outputs.version }}"
267+
gh release upload "$version" ${{ env.ASSET }} ${{ env.ASSET_SUM }}

0 commit comments

Comments
 (0)