Skip to content

feat(testing): add assertInlineSnapshot() #6530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Conversation

WWRS
Copy link
Contributor

@WWRS WWRS commented Mar 29, 2025

closes #3301

Updating the snapshots is similar to Jest: Jest uses Babel's AST parser to find where to insert the snapshot and then Prettier to format. This instead uses a Deno lint plugin to find where to insert and deno fmt to format.

The expectedSnapshots in _assert_inline_snapshot_test.ts were automatically generated by assertInlineSnapshot!

@WWRS WWRS requested a review from kt3k as a code owner March 29, 2025 17:05
Copy link

codecov bot commented Mar 29, 2025

Codecov Report

Attention: Patch coverage is 71.19205% with 174 lines in your changes missing coverage. Please review.

Project coverage is 94.44%. Comparing base (bfd0096) to head (a58c340).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
testing/_assert_inline_snapshot.ts 32.61% 157 Missing ⚠️
testing/_assert_snapshot.ts 94.40% 16 Missing ⚠️
testing/_snapshot_utils.ts 98.57% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6530      +/-   ##
==========================================
- Coverage   94.75%   94.44%   -0.31%     
==========================================
  Files         584      587       +3     
  Lines       46560    46825     +265     
  Branches     6539     6545       +6     
==========================================
+ Hits        44116    44224     +108     
- Misses       2401     2558     +157     
  Partials       43       43              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@WWRS WWRS mentioned this pull request Mar 31, 2025
20 tasks
@kt3k
Copy link
Member

kt3k commented Apr 3, 2025

Thanks for the PR, but this seems only supporting the initial creation of snapshot. In my view the capability of updating the snapshots are essential for snapshot testing tool. Can we also support the updating somehow? (I guess we can do that only by using AST analysis as jest does)

@WWRS
Copy link
Contributor Author

WWRS commented Apr 3, 2025

I agree that this limitation of assertInlineSnapshot means it's not as nice as Jest's.

You can update a snapshot by manually replacing the existing one with `CREATE` and running the test. This does force users to follow the commonly recommended practice of ensuring each updated snapshot should actually have been updated. Also, users always have the option of assertSnapshot() for tests that will need to be updated frequently enough that manual updating would be a hassle.

If we wanted to support updating failing snapshots automatically, it's likely that the best solution would involve AST analysis. I don't think Deno exposes its AST parser, and Jest uses Babel which is presumably something we cannot do here. So a system using AST analysis is not likely to be ready anytime soon.

I'm not sure, but I suspect that even without the ability to update automatically, users might find this feature useful. On the other hand, I do recognize the need to maintain high quality throughout the std library, and shipping an incomplete feature is not great either.

@stefnotch
Copy link

So if I understand this correctly, there are 4 different options here

  • Accept the simple implementation
  • Use a Deno.lint plugin to update the snapshots, since they have an AST inspection and mutation API.
    • That plugin should ideally ship with Deno by default.
    • Bonus points for getting IDE integration for free. The light bulb would be able to suggest updating the snapshot.
  • Make Deno.assertSnapshot a built-in feature of Deno, just like Deno.test and Deno.bench

@WWRS
Copy link
Contributor Author

WWRS commented Apr 3, 2025

Also an option would be to expose an AST parsing API as a built-in feature of Deno or as a library. From looking at #2355 it seems like this functionality will not be added to Deno but is in deno_swc. Would it be reasonable to include that as a dependency here?

I think adding assertSnapshot to Deno is not very likely. All the assertions are in std/assert so it doesn't make a ton of sense to have just one or two assertion functions in default Deno.

For deno_lint, I don't really know what our options look like there. Really all we need to do is find assertInlineSnapshot (and later expect.toMatchInlineSnapshot) and manipulate the third param. But like adding to default Deno, it seems unlikely we would want support for just one assertion function in the default linter.

@stefnotch
Copy link

deno_swc has not been updated in 3 years, so I wouldn't rely on it, even if denoland/deno#2355 points at it.

Deno's current parser is a Rust library, and thus a bit trickier to use from JS land. Hence my suggestion of adding the necessary plumbing to Deno itself.
https://github.com/denoland/deno_ast

@stefnotch
Copy link

I was about to write a comment about how the lint plugin option wouldn't work that well, until I realised something.

Deno.lint.runPlugin is a function that Deno provides in test mode.
Which exactly fits the bill of what we want! That's amazing!

So I quickly whipped up a prototype which makes use of that to give us inline snapshots!

Entire project is this, run it as per usual with deno test and deno test -A -- --update
temp-deno-linter-for-snapshots.zip

Video.mp4

And just the code. I'm okay with you copy-pasting and adapting this straight into Deno, and am releasing this under CC0 and MIT.

import { serialize, SnapshotOptions } from "@std/testing/snapshot";
import { equal } from "@std/assert/equal";
import { AssertionError } from "@std/assert/assertion-error";

const isUpdateMode = Deno.args.some((arg) =>
  arg === "--update" || arg === "-u"
);

interface SnapshotUpdateRequest {
  fileName: string;
  lineNumber: number;
  columnNumber: number;
  actual: string;
}

// Batch all writes until the very end, and then update all files at once
globalThis.addEventListener("unload", () => {
  updateSnapshots();
});
function updateSnapshots() {
  if (updateRequests.length === 0) {
    return;
  }
  console.log(`Updating ${updateRequests.length} snapshots...`);
  const filesToUpdate = Map.groupBy(updateRequests, (v) => v.fileName);

  for (const [fileName, requests] of filesToUpdate) {
    const fileContents = Deno.readTextFileSync(fileName);
    const pluginRunResults = Deno.lint.runPlugin(
      makeSnapshotUpdater(requests),
      "dummy.ts",
      fileContents,
    );
    const fixes = pluginRunResults.flatMap((v) => v.fix ?? []);
    if (fixes.length !== requests.length) {
      console.error(
        "Something went wrong, not all update requests found their snapshot",
      );
    }
    // Apply the fixes
    fixes.sort((a, b) => a.range[0] - b.range[0]);
    let output = "";
    let lastIndex = 0;
    for (const fix of fixes) {
      output += fileContents.slice(lastIndex, fix.range[0]);
      output += wrapForJs(fix.text ?? "");
      lastIndex = fix.range[1];
    }
    output += fileContents.slice(lastIndex);
    Deno.writeTextFileSync(fileName, output);
  }
}

const updateRequests: SnapshotUpdateRequest[] = [];

export function assertInlineSnapshot<T>(
  actual: T,
  expected: string,
  options?: SnapshotOptions<T>,
) {
  const _serialize = options?.serializer ?? serialize;
  const _actual = _serialize(actual);

  if (equal(_actual, expected)) {
    return;
  }
  if (isUpdateMode) {
    // Uses the V8 stack trace API to get the line number where this function was called
    const oldStackTrace = (Error as any).prepareStackTrace;
    try {
      const stackCatcher = { stack: null as SnapshotUpdateRequest | null };
      (Error as any).prepareStackTrace = (
        _err: unknown,
        stack: unknown[],
      ): SnapshotUpdateRequest | null => {
        const callerStackFrame = stack[0] as any;
        if (callerStackFrame.isEval()) return null;
        return {
          fileName: callerStackFrame.getFileName(),
          lineNumber: callerStackFrame.getLineNumber(),
          columnNumber: callerStackFrame.getColumnNumber(),
          actual: _actual,
        };
      };
      // Capture the stack that comes after this function.
      Error.captureStackTrace(stackCatcher, assertInlineSnapshot);
      // Forcibly access the stack, and note it down
      const request = stackCatcher.stack;
      if (request !== null) {
        const status = Deno.permissions.requestSync({
          name: "write",
          path: request.fileName,
        });
        if (status.state !== "granted") {
          console.error(
            `Please allow writing to ${request.fileName} for snapshot updating.`,
          );
        } else {
          updateRequests.push(request);
        }
      }
    } finally {
      (Error as any).prepareStackTrace = oldStackTrace;
    }
  } else {
    throw new AssertionError("Assertion failed blabla");
  }
}

/// <reference lib="deno.unstable" />
/**
 * Makes a Deno.lint plugin that can find inline snapshots.
 * Also deals with multiple `assertInlineSnapshot` in a single line.
 */
function makeSnapshotUpdater(
  updateRequests: SnapshotUpdateRequest[],
): Deno.lint.Plugin {
  const linesToUpdate = Map.groupBy(updateRequests, (v) => v.lineNumber);

  return {
    name: "snapshot-updater-plugin",
    rules: {
      "update-snapshot": {
        create(context) {
          return {
            'CallExpression[callee.name="assertInlineSnapshot"]'(
              node: Deno.lint.CallExpression,
            ) {
              // Find the update request that corresponds to this snapshot.
              // Successful snapshots don't have an update request.
              const callPosition = toLineAndColumnNumber(
                node.range[0],
                context.sourceCode.text,
              );
              const endColum = callPosition.column +
                (node.range[1] - node.range[0]);
              const lineUpdateRequests = linesToUpdate.get(callPosition.line);
              if (lineUpdateRequests === undefined) {
                return;
              }
              const updateRequest = lineUpdateRequests.find((v) =>
                callPosition.column <= v.columnNumber &&
                v.columnNumber <= endColum
              );
              if (updateRequest === undefined) {
                return;
              }

              context.report({
                node,
                message: "",
                fix(fixer) {
                  return fixer.replaceText(
                    node.arguments[1],
                    updateRequest.actual,
                  );
                },
              });
            },
          };
        },
      },
    },
  };
}

/**
 * Takes an index and returns the 1-based line number and 1-based column number */
function toLineAndColumnNumber(index: number, text: string) {
  const textBefore = text.slice(0, index);
  // TODO: Verify that Chrome's V8 uses the same logic for returning line numbers. What about the other line terminators?
  const lineBreakCount = (textBefore.match(/\n/g) || []).length;

  // Also deals with the first line by making use of the -1 return value
  const lastLineBreak = textBefore.lastIndexOf("\n");
  return {
    line: lineBreakCount + 1,
    column: (index - lastLineBreak),
  };
}

function wrapForJs(str: string) {
  return "`" + escapeStringForJs(str) + "`";
}

function escapeStringForJs(str: string) {
  return str
    .replace(/\\/g, "\\\\")
    .replace(/`/g, "\\`")
    .replace(/\$/g, "\\$");
}

@stefnotch
Copy link

stefnotch commented Apr 4, 2025

@WWRS Would you be willing to integrate the Deno.lint approach into this PR? I'd really appreciate it, since that'd save me the effort of creating and polishing a pull request for this particular feature.
I'm asking since your PR already has documentation, unit tests, code cleanups and other necessary bits.

If yes, here are some quick pointers

  • I think we could get away with dropping the first argument for assertInlineSnapshot. One already gets the correct file from the stack trace.
  • If you have any ideas on how to simplify parts of the code, I'd be delighted to hear them. I couldn't figure out a simpler way of doing the "find the correct assertInlineSnapshot". The simple approach of counting wouldn't work here, since the linter doesn't know in which order the inline snapshot functions are called.
  • Re-assigning assertInlineSnapshot to a new variable is unsupported. So assertMonochromeInlineSnapshot wouldn't work. I wonder if there's a good solution for that.
  • Wrapping assertInlineSnapshot in an extra function would lead to very unexpected results. I'm not sure if one can do much about that.
  • I wonder what happens if one types deno test --watch -A -- --update

@WWRS
Copy link
Contributor Author

WWRS commented Apr 5, 2025

I'm not 100% sold on using the error throwing location as the way to find where to insert the snapshot, it seems a bit obscure and dependent on V8. As noted, we would need to double check how V8 counts line numbers, but also how it counts unicode. (For reference, Jest does use this error throwing approach.)

For finding the right snapshot to update, would the numbering system that's in the current implementation of assertSnapshot work? It seems like tests are always run from top to bottom within the same file. I'm not sure if this works when running a subset of the tests, but if it breaks here, it might have issues in assertSnapshot as well.

For whether or not we need a context argument, if we were to remove the argument here, we would probably want to make a similar change in assertSnapshot to keep the call signatures parallel. I'm not sure how messy that would get, do you think it would be possible? Also, #6541 is worth thinking about: If we were to add a context-injected expect in bdd tests, then we could make toMatchSnapshot and toMatchInlineSnapshot use that expect and get its context for free, though possibly at the cost of reduced support outside bdd tests.

I'd like to hear some other opinions on the tradeoffs of the two proposed implementations. As noted above, the magic-word implementation lacks a core capability because it requires screenshots to be manually flagged for updates. Are the missing capabilities of the lint implementation (assertMonochromeInlineSnapshot or wrapping in a helper function) also core?

(It does occur to me that if we want to use this in expect.toMatchInlineSnapshot we'll probably have to update the lint rule.)

@WWRS
Copy link
Contributor Author

WWRS commented Apr 5, 2025

I checked, the existing assertSnapshot-style counting system does work. This means we can avoid using errors to find where to insert snapshots, though in this case we do still need the TestContext.

So I do have a working version using the lint implementation, but I'll wait to push it until we decide which we prefer.

@stefnotch
Copy link

Thank you so much for taking a look at this. Those are some very good points, so I did some reading.

Error Throwing

Yes, the error throwing approach relies on V8 APIs. I wish we had a standard API for this https://github.com/tc39/proposal-error-stacks

For now, I think it's fine, because updating the snapshots relies on filesystem access. So, it's limited to server-side JS runtimes, which generally implement the V8 APIs. Even Bun does.

And yes, we absolutely have to check how Unicode gets counted for the column number. We'll have to verify whether the following behaves as expected.

/* 🐈‍⬛🐈‍⬛🐈‍⬛🐈 */ assertInlineSnapshot(2 + 3, `'5'`)

On that note, the linter's ranges also rely on an important detail: They're reported in UTF-16 code-somethings, which matches Javascript string indexing. So at least that part is sensible.

Numbering system

assertSnapshot relies on a numbering system. It computes a stable ID for each assertSnapshot call. That ID only changes when one renames the tests, or starts calling assertSnapshot in a different order.
To update a snapshot, it runs the test, and saves the new value with the ID.

(Amusingly, it means that if(Math.random() < 0.5) { await assertSnapshot(t, 1); } else { await assertSnapshot(t, 2); }) would repeatedly break. But that's a non-deterministic test, so it deserves to break.)

This does not work for assertInlineSnapshot.
To update a snapshot, it has to find the correct place in the source code.

One quick example where the numbering system would not work is

test("foo", () => {
  if(false) {
    assertInlineSnapshot("never happens");
  }
  assertInlineSnapshot("wait, why am I not getting updated");
});

I'm sure one can also think of a counterexample with promises, like calling assertInlineSnapshot in a .then

Context argument

Good point, that'd be an odd inconsistency.

So #3964 has an open issue about implementing a expect(foo).toMatchSnapshot() API.

But clearly the normal snapshotting API needs a test context. So I looked at what Jest and Vitest do.

Vitest passes the context to expect when run synchronously, and tells you to use the correct expect when you have async tests.
https://vitest.dev/guide/snapshot.html#use-snapshots

test('math is easy', async ({ expect }) => {
  expect(2 + 2).toMatchSnapshot()
})

Jest instead assumes that it runs within Node.js, and makes use of the AsyncLocalStorage API.
jestjs/jest#14139
Deno also implements that API https://docs.deno.com/api/node/async_hooks/~/AsyncLocalStorage
The downside of it is that it doesn't work in browsers.

I think you're better than I am at judging which of the trade-offs is the best option here.

assertMonochromeInlineSnapshot

There are four options for dealing with the limitation.

  1. Only expose it as expect(foo).toMatchInlineSnapshot('bar'). Now the temptation to alias it is much lower.
  2. Don't look for a specific work in the syntax tree, only look for a function call at the correct spot. As in, update it as long as there is a foo(bar, ...) in the expected position.
  3. Add a setting where one can pass in the name to look for, like
const assertMonochromeInlineSnapshot = createAssertInlineSnapshot<string>({
    serializer: stripAnsiCode,
    name: "assertMonochromeInlineSnapshot" // Users are very likely to forget this, and it would not survive minification or re-assignment.
  });
  1. Just document the limitation. Inline snapshots are wizardry anyways.

@stefnotch
Copy link

stefnotch commented Apr 5, 2025

I just found out that there's a proposal for async context tracking https://github.com/tc39/proposal-async-context

If we had that, then we'd no longer need the first argument for the regular assertSnapshot function. And we could finally perfectly re-implement the expect().toMatchSnapshot() function from Jest.

@stefnotch
Copy link

@WWRS I figured I'd check back in: How is it going? Which trade-offs should we choose? Is there anything I could do to help move this PR forward?

@WWRS WWRS force-pushed the inline branch 3 times, most recently from 6e802a8 to 12d7fd8 Compare April 29, 2025 02:04
@WWRS
Copy link
Contributor Author

WWRS commented Apr 29, 2025

@kt3k This now supports AST analysis, allowing us to update by passing the --update flag instead of a magic string.

@stefnotch I am now using the error throwing location instead of the numbering system. For assertMonochromeInlineSnapshot I am looking at every function call and grabbing the one that matches. I also added a test to ensure this counts lines and columns the same way V8 does.

Remaining weird stuff:

  • The updater fails if someone tries to wrap it in a function where the expected snapshot is not the third variable. This should be rare, but if anyone is actually doing this, we can add an option for which index the expected snapshot is at. It's also something that might end up happening when this gets added to expect.
  • I think this is a Deno issue, but if an error is thrown in the updater, the error printed is very unhelpful:
error: null
This error was not caught from a test and caused the test runner to fail on the referenced module.
It most likely originated from a dangling promise, event/timeout handler or top-level code.

* the file will not be formatted, and we will throw.
*/
format?: boolean;
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm debating whether Pick<...> is the most elegant option here. The alternatives would be

  • Duplicating the snapshot interface. It's not that much duplication.
  • Having 3 snapshot interfaces and using extends.

I suppose Pick is pretty fine, the alternatives don't sound better.

Copy link

@stefnotch stefnotch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much for all your hard work! I can't wait for this to make its way into Deno

"CallExpression"(node: Deno.lint.CallExpression) {
const snapshot = locationToSnapshot[node.range[0]];
const argument = node.arguments[2];
if (snapshot === undefined || argument === undefined) return;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woah, I had no idea that this would end up so short and surprisingly elegant. I approve👍

) {
const [lineNumber, columnNumber] = lineColumn.split(":")
.map(Number);
const location = (lineBreaks[lineNumber! - 2] ?? 0) + columnNumber!;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why - 2?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah I'll leave a comment. The line numbers are 1-indexed so we need to do a -1 to get to 0-indexed. Then we need to get the position of the break before this line, so that's the n-1th break, which is where the second -1 comes from.

}

const testFilePath = fromFileUrl(this.#testFileUrl);
ensureFileSync(testFilePath);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I probably left this in from snapshot.ts. The test file should always exist, and even if it doesn't the read will error. I'll remove this.

assertEquals(
await Deno.readTextFile(countTestFile),
`import { assertInlineSnapshot } from "${SNAPSHOT_MODULE_URL}";
\n \r \n\r \r\n

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet, thank you for testing this!

I checked the Ecmascript specification, and there are two more valid line breaking characters.
U+2028 (LINE SEPARATOR) and U+2029 (PARAGRAPH SEPARATOR)
https://tc39.es/ecma262/#sec-line-terminators

I double checked it in Chrome with this code.

eval(`a=1\u2028throw new Error("nya")`)
eval(`a=1\u2029throw new Error("nya")`)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some investigation,

denoland/deno_core#1123

const options = getOptions(msgOrOpts);
const assertInlineSnapshotContext = AssertInlineSnapshotContext.fromContext(
context,
);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a reasonable use of the test context. Am happy with this API.

If we ever want to build the expect(foo).toMatchInlineSnapshot() API, we'll have to extract the current file name from the stack trace. But that's a future feature request that I do not absolutely need. As long as I have one API for inline snapshots, I am really happy.

Copy link

@stefnotch stefnotch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@stefnotch
Copy link

I believe this is ready for the Deno team to review, right?

On that note, a question for @kt3k : Do you think that this is interesting enough to be noted somewhere in the Deno blog or in a Deno Tweet? While it is a major feature for me, I have no idea how widely used inline snapshots are. Hence this question.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[feature request] Add support for inline snapshots tests
3 participants