Closed
Description
It's been a while since we update the Web Platform Tests in test/parallel
in bulk and there have been some recent updates upstream that make the tests easier to run in our test suite, so here is a tracking issue for a revamp of existing tests (hopefully building the foundation of more test integration in the future).
I have a WIP in node-core-utils that currently pulls the tests/harness, track the commit hashes to a file, and makes sure that the license stuff are in place. There is also a WIP branch for this repo with the revamp for URL tests. I was thinking about writing a design doc and get some feedback before opening PRs in both repo, but probably a GitHub issue may work better so here it goes.
- Tooling to pull tests from upstream automatically (right now they are copy-pasted by hand), maintain the commit hashes in a local file and in the git history, and makes sure the license stuff are in place. (In my WIP I put them all in
test/fixtures/wpt
so the linter will ignore them) - Have a test runner that uses the harness in WPT instead of our mocks in
test/common/wpt.js
- Sort out a status file for test expectations (including requirement about intl/crypto, and expected failures, .etc). I have implemented this in my fork with a separate format, but we may also consider changing our python test runner to consume a status file with finer-granularity of control and communicate with Jenkins nicely in the future.
- Migrate the
.any.js
tests upstream into a separatetest/wpt
directory with a test runner that creates a sandbox with the WPT harness to run the tests and use the status file to check expectations. - Work with the upstream to migrate more tests out of HTML and put them into JS files that are easier for us to pull down into our repository, as well as split out the parts that cannot be run by us (involving certain APIs that we have not implemented) from the parts that we can run.
------------ TODO ----------
- Fix existing test failures: mostly IDL failures. See status files
- Port encoding tests (need to figure out ICU requirements for the CI): test: port encoding WPT into core #25321
- Port timer tests and mark expected failures
- Port performance tests and mark expected failures
- Port messaging tests and mark expected failures
- Upstream/split existing
test/parallel/test-whatwg-url-custom-*.js
and remove them - Upstream/split existing
test/parallel/test-whatwg-encoding-custom-*.js
and remove them - Support running
.window.js
(but we are not a window, so we need to add another scope for us in the upstream and potentially WebIDL) - Enable the driver to be ported out to another project so that it can be reused by our "official" modules: POC in [POC]: Porting WPT utilities in Node.js core whatwg-stream#2