custom (ESM) loader + extension-less "bin" files

[JakobJingleheimer]

Currently, when a custom ESM loader is used in combination with an extension-less commonjs file, it erroneously fails. For example

project-a/
  ├ node_modules/
     └ mocha/
        ├ bin/
           └ mocha
        └ package.json
  ├ src/
  ├ a-loader.cjs
  ├ a-loader.mjs
  └ package.json

NODE_OPTIONS="--loader=./a-loader.mjs" npx mocha ./src

There have been several disparate discussions about this issue. I’ve tried to capture the main points here from the ones I’ve found so far. If I’ve missed any, please comment and I’ll update.

Related: #33226, #41275, #41465, #41552, mochajs/mocha#4645

Problems

1A: The presence of the --loader flag causes Node.js to use the "wrong" loader (ESMLoader instead of CJSLoader), regardless of the entry-point’s actual type.

1B: ESMLoader::defaultGetFormat() ignores any file without a .cjs, .mjs, .js, .json, or .node file extension. This means, for a commonjs file without a file extension (which is allowed), Node.js will eventually throw when it should not.

Considerations

WASM

ESMLoader intentionally does not handle extension-less files due to the possibility of extension-less wasm files in future.

Loader for entry-point

There is not an authoritative way to dictate which Loader (CJS vs ESM) should be used: A user specifying --loader my intend for that to be used when loading the entry-point (and thus manipulate how it’s handled). However, the logic in shouldUseESMLoader() does not account for CJS’s equivalent flag, --require (and friends). So if both are supplied together:

NODE_OPTIONS="--loader=./a-loader.mjs --require=./a-loader.cjs" npx mocha ./src

What does that mean? One might reasonably expect the ESM loader to merely be used when the parent is ESM and the CJS require to merely be used when the parent is CJS (there is currently no other way for the ESM loader) and neither to affect which Loader is used for the entry-point.

Previously proposed solutions

I've created threads below with each I found.

Workarounds

Add a snippet to the custom loader

I use this in my own projects (my actual uses a map of extensions to load “hooklets” à la { '': loadBin, jsx: loadJSX, … })

sample

export async function load(resolvedUrl, context, defaultLoad) {
  const url = new URL(resolvedUrl);
  const ext = path.extname(url.pathname).slice(1);

  if (ext === '') loadBin(url, context, defaultLoad);

  // … other custom loader code

  return defaultLoad(resolvedUrl, context);
}

async function loadBin(url, context, defaultLoad) {
  const dirs =  path
    .dirname(url.pathname)
    .split(path.sep);
  const parentDir = dirs.at(-1);
  const grandparentDir = dirs.at(-3);

  let format;

  if (parentDir === 'bin' && grandparentDir === 'node_modules') {
    const libPkgUrl = new URL('../package.json', url)
    const { type } = await fs.readFile(libPkgUrl).then(JSON.parse);

    format = type === 'module' ? 'module' : 'commonjs';
  }

  return defaultLoad(url.href, {
    ...context,
    format,
  });
}

@nodejs/loaders

[cspotcode]

1A: The presence of the --loader flag causes Node.js to use the "wrong" loader (ESMLoader instead of CJSLoader), regardless of the entry-point’s actual type.

Another way to phrase this might be that the ESMLoader is the correct loader in all cases, but it arrives at a different / "wrong" conclusion compared to the CJSLoader. Both loaders should come to the same conclusion regarding the format of the entrypoint.

Teaching the ESMLoader today to draw the same conclusion as the CJSLoader for entrypoints does not prevent node from, in the future, updating both loaders to arrive at a different conclusion. But leaving node's two loader behaviors inconsistent as it is today makes this a problem.

[guybedford]

node -e "import('bin/foo')"

Would fail since it is not an entrypoint. This kind of workflow of wrapping an entrypoint is used in the wild and doesn't have a workaround while adding an extension to the entrypoint is a simple workaround to problems with extension-less files.

In the above, using a child_process spawn instead would still ensure the workflow is supported. That is - there is still a sensible way to achieve this, while solving the major use case.

So I don't think that is a compelling argument against this approach.

[cspotcode]

This is not a goal / unlikely to ever happen due to sheer amount of breakage so we can move discussion of it elsewhere.

Agreed, I was just trying to play along.

To note, using the ESM loader can still load CJS files if it knows to do so so using the ESM loader for the entry point is very different from treating the entry point as ESM (terms here are quite confusing unfortunately).

Correct, in everything I've said about using the ESM loader for entrypoints, I am referring to invoking ESM loader getFormat. I am not saying that the entrypoint is necessarily executed as ESM. The output of getFormat should be respected, whatever that may be, wherever it may come from (external loader or node's internal loader / default* functions)

[bmeck]

child_process.spawn causes things like PID tracking to need to change / is complex regarding process.execArgv / loses global mutations / etc. I agree it is a mitigation but not a simple or easy one. This doesn't just affect process entry points as well, this mitigation also affects Workers which have similar complexities vs adding a file extension.

[guybedford]

Yes, but if we are able to let users write extensionless Node.js applications that can support loader hooks and ESM that is still a win.

[bmeck]

@guybedford we are in agreement on being able to use loader hooks with extension-less files being a good thing, but the solution isn't without downsides or mitigations to those downsides. We are just trying to figure out a solution on how to handle that in this discussion. I think having exceptions based upon way in which something is imported is complex. Additionally, even without the concerns above doing this type of exception cannot be statically determined if it relies on something being an entry point. This approach just has a lot of complexity that spills out vs other ways to allow loaders to handle extension-less files.

[JakobJingleheimer]

Proposal: Limited extension-less “bin” support in defaultGetFormat()

Rationale: It is simple to determine when the situation is occurring, and similarly simple to address here.

This proposal was all but rejected.

[bmeck]

I'm wondering if this should be revisited but not as an exception based upon entry point / extension-less. Perhaps having ESM loader default return the format from/as CJS if in a type:commonjs context would fix most issues/all? It has a major downside of any new extension that ESM picks up it wouldn't go through the CJS loader anymore which is a forwards compatibility hazard, but only a hazard in type:commonjs contexts.

[JakobJingleheimer]

I don't quite understand what you're suggestion. Are you saying ESMLoader::defaultGetFormat() should pass anything it doesn't understand to CJSLoader (where it may eventually error)?

[bmeck]

Whenever an unknown format is found in a type:commonjs context, yes. Since resolve() can return a hinted type this actually works pretty well.

/a #type:module
/a/foo.mts

/a/bin #type:commonjs
/a/bin/bar

# no --loader
node /a/foo #in type:module
# esm resolve('file:///a/foo', {parentUrl: undefined}) -> {url: 'file:///a/foo', format: undefined}
# esm load(...) // ERROR: unknown format + no resolved format

node --loader mts /a/foo.mts #in type:module
# esm CUSTOM resolve('file:///a/foo.mts', {parentUrl: undefined}) 
  # esm resolve -> {url: 'file:///a/foo.mjs', format: undefined}
# -> {url: 'file:///a/foo.mjs', format: 'typescript'}
# esm CUSTOM load(...)
  # esm load(...) // unknown format but has resolved format, return source but type:'typescript'
# -> {...}

node /a/bin/bar #in type:commonjs
# esm resolve('file:///a/bin/bar', {parentUrl: undefined}) -> {url: 'file:///a/bin/bar', format: undefined}
# esm load(...) // unknown format + no resolved format
# -> {..., format:'commonjs'}

node --loader mts /a/bin/bar #in type:commonjs
# esm CUSTOM resolve('file:///a/bin/bar', {parentUrl: undefined})
    # esm resolve -> {url: 'file:///a/bin/bar', format: undefined}
# esm CUSTOM load(...)
  # esm load(...) // unknown format + no resolved format -> {..., format:'commonjs'}

[JakobJingleheimer]

Proposal: Remove flag checks from shouldUseESMLoader()

Rationale: The current inclusion of the flag checks is undocumented behaviour and violates documented behaviour for determining how a file is loaded. Consideration of this other issue is larger than the bug at hand.

This proposal seemed to have a fair number of supporters, but also some opponents. Opposition cited the (very small) burden it could put on certain users, despite very simple and very viable workarounds.

[GeoffreyBooth]

Are you suggesting that eg "type": "module" would apply to .ts files?

No. --loader hooks into the ESM loader, and only works for files that are evaluated as a format other than commonjs (for now). The goal is to extend support to CommonJS too but that's a ways off.

[guybedford]

--loader must always be able to hook the resolver and treat any sensible Node.js entry point as ESM if necessary.

[GeoffreyBooth]

--loader must always be able to hook the resolver and treat any sensible Node.js entry point as ESM if necessary.

Yes, in my proposal it would do that. That's why I'm suggesting ESM always tries to resolve first. The CommonJS resolution would only happen if the ESM defaultResolve, and any custom loaders that have been registered, all fail to resolve a runnable file.

[cspotcode]

Currently loaders don’t really work with CommonJS files, so for now extensionless or unknown-extension files that are enabled via loader would need to be ESM.

Worth pointing out for anyone reading along: in practice, they do support CJS. Today a --loader can also install a require.extensions hook ¹, so it can both classify any file as CJS and transform that file before execution. Effectively we have full support today while we wait for node to implement a cleaner alternative.

My point is not that this is pretty, but it's possible. This allows the ecosystem to experiment and for loader authors to keep users unblocked in the interim.

Footnotes

1. I know this is because it's in the main thread and that may change ↩

[GeoffreyBooth]

can also install a require.extensions hook

Yes, this is true. When we were discussing loaders I was thinking of what could be achieved through the loaders API, not in a loader's code but using a different API. But yes, the “CommonJS loaders API” of require.extensions can fill some gaps.

[JakobJingleheimer]

Proposal: ESM custom loader opt-in via export flag

Allow ESM custom loaders to opt into affecting entry-points. Ex

// a-loader.mjs

export const useForEntryPoint = true;

export async function resolve() {}
export async function load() {}

This may introduce a cart-before-the-horse scenario (the custom loader might not have been itself loaded, so reading that export flag may currently not be possible).

[JakobJingleheimer]

Proposal: Create a “higher” class of custom loader (eg ambient-loader)

This class of custom loader would always be applied to entry-points (and may do other things too).

[JakobJingleheimer]

This sounds to me a potentially viable solution down the road but too large in scope for merely the current issue (and would thus take quite a while to hash out the technical design).

[bmeck]

I think this might be a different solution to a different problem, the size of this would be very large and have many potential side effects vs other potential solutions.

[JakobJingleheimer]

Proposal: Treat extension-less file with "magic phrase" as JS

If and only if the file starts with #!/usr/bin/env node\n, parse as JavaScript text file (and follow regular rules).

[JakobJingleheimer]

I think this could work and aligns with historic precedent, but seems a little convoluted and cart-before-the-horse.

[ljharb]

This still doesn’t answer whether it’s ESM or CJS tho - overloading type module would mean that i couldn’t have .js ESM in the same folder as extensionless CJS, or vice versa.

[guybedford]

I would probably have a slight preference towards any magic string checks as applying specifically to the main node x entry point of the execution environment, as opposed to applying to any import.

Overloading the type is the same problem with wanting to have multiple .js meanings in the same app - one could create bin/package.json with "type": "commonjs" set if wanting to have variation between files in the package per current process.

[JakobJingleheimer]

This still doesn’t answer whether it’s ESM or CJS tho

@ljharb it would, per

(and follow regular rules).

[JakobJingleheimer]

I would probably have a slight preference towards any magic string checks as applying specifically to the main node x entry point of the execution environment, as opposed to applying to any import.

Yes, I think that was the intention of the original proposal 🙂

[guybedford]

Definining "extensionless"

In relation to https://github.com/nodejs/node/discussions/41711#discussioncomment-2054908, another topic worth bringing up here is what is the definition of an extensionless file? There are a few that might apply:

• Is it a file without a . at all?
• Or is it a file that does not use a supported Node.js extension (ie not .js / .json / .wasm / .node / .cjs / .mjs)
• Or is it a file that does not use an extension from some MIME list or compound list of file extensions deemed to be the known extension list? If so, how would one construct such a list, and would it be a possible approach?

Under a specific definition of extensionless in the ESM loader it could then be possible for this definition to exhibit both backwards and forwards compatibility.

The backwards compatibility risk would remain that extensions in the reserved list would work in the CJS loader but not the ESM loader.

The forwards compatibility risk would remain that new extensions in future not in the reserved list would be breaking changes to support. Although perhaps this isn't such a huge risk since surely this is what MIME registries are designed to avoid in the first place.

[bmeck]

Is it a file without a . at all?

That is what I believe this issue is about

Or is it a file that does not use a supported Node.js extension (ie not .js / .json / .wasm / .node / .cjs / .mjs)

This seems to have an extension, but it is unknown to Node.js, so unknown extension might be a good term

Or is it a file that does not use an extension from some MIME list or compound list of file extensions deemed to be the known extension list? If so, how would one construct such a list, and would it be a possible approach?

I'm not sure what this means

[ljharb]

if it has a dot, it’s not extensionless. Whether node knows about the extension has no bearing on whether it has one.

[guybedford]

Okay, that's definitely much easier to deal with then.

[cspotcode]

Coming back to this, it looks like the best options are either:

• Attempt the ESM loader for all entrypoints and fallback to the CommonJS loader when the ESM loader fails to return a URL or a format
• Teach the ESM loader's default* hooks to emulate CJS-style fallback when node:entrypoint condition / parentURL sentinel / context.isMain / internally-bound isMain

[cspotcode]

For that one, I'm thinking defaultResolve() can do the deferral if it's resolving an entrypoint. So if a loader hook calls defaultResolve(), and it is resolving an entrypoint, and the CJS fallback is necessary, then the return value of defaultResolve() will be the result of the CJS fallback.

[bmeck]

I think the fallback semantics need to be worked out, they show up in at least 2 of the threads above but with different situations causing them. However, since 2 threads are proposing a similar solution perhaps we can gather around that.

[cspotcode]

Agreed; I was reluctant to create a new thread, but since the threads above are long enough that they're hard to follow, a new one makes sense to me.

To avoid confusion, it will help to define vocabulary for referring to node's internal ESM loader vs referring to --loader hooks. The internal ESM loader is capable of classifying a file as CommonJS and causing execution of that file as CJS, so it is often possible to use the ESM loader in CJS contexts. I worry that saying "ESM loader" might imply to some readers that CJS files cannot pass through that code path.

It feels like there are 3 things that might be thought of as "ESM Loader":

• the internal ESM loader, responsible for invoking --loader hooks if necessary, and triggering CJS execution if format is CJS
• user --loader hooks, which often rely on receiving a meaningful answer from defaultResolve or defaultLoad
• the implementations of defaultResolve, defaultLoad, and defaultGetFormat (defaultGetFormat might be confusing)

[GeoffreyBooth]

I worry that saying “ESM loader” might imply to some readers

I don’t think casual users of Node will need to know the details. The docs will say something like “Node will try to resolve the entry point as an ES module; if it can be, it will be run as an ES module. If it can’t, Node will then try to resolve the entry point as a CommonJS module and run it as that.”

[cspotcode]

Indeed, I'm referring to how we proceed with our discussion here.