Helping Babel move to ES Modules

The Babel project recently moved to running builds on Node.js 14, which means that Babel can now use ES Modules (ESM) instead of CommonJS (CJS) to import/export modules for internal scripts.

Also, with the upcoming Babel 8.0.0 release, the team is aiming to ship Babel as native ES Modules. With this goal in mind, the team is shifting all CommonJS imports/exports to ESM ones. This is where I got the opportunity to contribute to Babel recently.

Why ES Modules though?

For a very long time, JS (or ECMAScript) did not have a standardized module import/export syntax. Various independent packages introduced formats to help work with modules in JS. Most browsers used the AMD API (Asynchronous Module Definition) implemented in the Require.js package, which had its own syntax and quirks.

CommonJS on the other hand was the standard used by Node.js, and it was no less quirky. Inconsistent formatting and poor interoperability between packages irked JS developers enough to demand a standard format.

Lately, the ECMAScript Standardization body (TC39) has adopted ESM (ECMAScript modules) as the standard format for Javascript. Most web browsers already support this format and Node.js 14 now provides stable support for it.

The task at hand

The next task was to convert all internal top-level scripts from using CommonJS to ESM. The finer details of the implementation, along with interoperability issues with non-ESM files, would trouble CommonJS for some time though.

The simplest of changes was to replace require() statements in each file with import statements. For example, files starting like:

"use strict";

const plumber = require("gulp-plumber");
const through = require("through2");
const chalk = require("chalk");

would be modified like here:

import plumber from "gulp-plumber";
import through from "through2";
import chalk from "chalk";

to allow modules to be imported as ES modules.

In the above example also note that because ES modules are in strict mode by default, so "use strict"; declarations were removed from the beginning of these top-level scripts.

Almost all current NPM packages are CommonJS packages, exposing their functionalities using the module.exports syntax.

In case a file/package exports more than one value, we need to use named imports instead:

import { chalk } from "chalk";

Where the default export object from a CommonJS module was named differently, it had to be aliased during import to avoid breaking pre-existing variables' names in the files being converted to ESM. For example,

const rollupBabel = require("@rollup/plugin-babel").default;

had to be replaced with:

import { babel as rollupBabel } from "@rollup/plugin-babel";

so we could keep using the variable rollupBabel in the file.

For instances where require() statements needed to be replaced by the dynamic import() statements

const getPackageJson = (pkg) => require(join(packageDir, pkg, "package.json"));

// replaced by
const getPackageJson = (pkg) => import(join(packageDir, pkg, "package.json"));

the subsequent calls everywhere now needed to be awaited:

   .forEach(id => {
      const { name, description } = getPackageJson(id);
   })

   //await added
   .forEach(id => {
      const { name, description } = await getPackageJson(id);
   })

Other things like importing JSON modules are currently only supported in CommonJS mode. Those imports were left as-is.

Blockers

With all the changes made and committed, we bumped into the next big roadblock: package dependencies. Babel uses Yarn 2 internally, and particularly the PnP feature of Yarn 2. Unfortunately, the ESM loader API was experimental at the time and not being used by PnP. The Babel and Yarn teams coordinated to implement it soon after.

Similarly, Jest has its own custom loader for ESM, which meant it could not support testing ESM modules with Babel. That issue was side-stepped for the time being.

Network effects

The good thing about the whole grind of shifting from CommonJS to ESM is that a lot of other major packages are also considering and implementing ESM support. The shift to ESM-only by Babel is already building confidence in others to do the same. Special thanks to the Babel maintainers for setting a great example and encouraging others to move to ESM.

Conclusion

All told, it was a great experience adding a new feature into a well-maintained and widely-used package. The biggest lesson from this has to be how changes made in Babel affect and influence other major packages, and how maintainers of various major open source packages work in tandem to avoid breaking each other's code. It is a very open and collaborative ecosystem with people discussing and working through github issues, comments, and even twitter threads.

Check out the pull request for more details.