Description
Traditionally, cargo-test-support has had bespoke assertions, requiring hand implemented diff algorithms. As there is less of a community around this, the knowledge is more specialized, its less likely to be documented, and we are on our own for feature development.
In #13980 and #14031, we introduced the use of snapbox as replacement for our own assertions and need to work to migrate to them.
Aside: doc updates related to this transition
- feat(test): Add cargo_test to test-support prelude #14243
- fix(test): Move path2url to CargoPathExt::to_url #14266
- fix(test)!: Clarify extension trait role with rename #14269
- feat(test): Re-export ProcessBuilder #14268
- fix(test): Move 'cargo_home' from 'install' to 'paths' #14270
- docs(test): Expand documentation of cargo-test-support #14272
Porting Instructions
1. Select a file to port
The files should have #![allow(deprecated)] at the top, e.g.
$ git pull --rebase
$ rg '#!.allow.deprecated' testsPlease check the comments on this issue for anyone to have claimed the file and then post that you claim the file
2. Remove #![allow(deprecated)] to identify what work is needed
If nothing, congrats, that was easy!
3. Resolve basic deprecations
Replace Execs::with_stdout("...") with
use cargo_test_support::prelude::*;
use cargo_test_support::str;
.with_stdout_data(str![])Replace Execs::with_stderr("...") with
use cargo_test_support::prelude::*;
use cargo_test_support::str;
.with_stderr_data(str![])(prelude is only needed for some steps)
Side note: If the test is specifically wanting to assert an empty string, use
""rather thanstr![]so that people are less likely to accidentally do a snapshot update to bless output that gets added.
Commit these changes
Run SNAPSHOTS=overwrite cargo test && cargo check --test testsuite. In rare cases, SNAPSHOTS=overwrite may make bad edits. Feel free to create a reproduction case and create an issue. They are usually obvious how to fix, so don't worry.
Diff the snapshots. Is there anything machine or run specific in them that previously was elided out with [..]?
- Can you auto-redact it like 5ea1c8f? If so, revert the snapshot updates, add the auto-redaction (as a separate commit before any test changes), and re-run the snapshots
- Otherwise, replace it with
[..]
Once its working, amend your commit with the snapshots
4. Resolve non-literal deprecations
Like Step 3, but not just with_stdout("...") but with variables or format!.
Evaluate whether a literal could be used
- Is the variable important for showing that output remains unchanged between instances, then keep it
- Are we specifically testing for what we are using
format!for, then keep it- If not, see if it can be expressed with an auto-redaction
So this can end up with either
with_std*_data(expression)with_std*_data(str![])like in Step 3
5. Repeat with "straight forward" deprecations
with_stdout_unordered(expected)->with_stdout_data(expected.unordered())with_stderr_unordered(expected)->with_stderr_data(expected.unordered())with_json(expected)->with_stdout_data(expected.json_lines())orwith_stdout_data(expected.json())- May also require
.unordered()
- May also require
6. Contains / Does not Contains deprecations
A judgement needs to be made for how to handle these.
A contains can be modeled by surrounding a entry with ... (text) or "...", (json lines). The question is whether we should still do this or use a different method.
- If replacing a
containswith an equality check, use multi-line globs (...) for rustc errors that weren't previously matched to minimize tying Cargo's tests to the exact output of rustc - Multiple
containscan be merged into a singleeqwith..interspersed (or one...and.unordered()). For example, see 3054936- When using
unordered, likely string literals, rather thanstr![]should be used, as these can't correctly be updated
- When using
A does_not_contain/line_without cannot be modeled at this time. The challenge with these is that they are brittle and you can't tell when they are no longer testing for what you think because the output changed. We should evaluate what to do with these on a case-by-case basis.
When in doubt, feel free to put #[allow(deprecated)] on a statement and move on. We can come back to these later.