Skip to content

Migration to ES Modules #6520

Open
Open
Migration to ES Modules#6520
Labels
better engineeringNot a bug or feature requestmetaMeta-issue about the project itself. Either project maintenance or a list of other issues.
Milestone
Planned

Description

@Josh-Cena
Josh-Cena
opened on Jan 31, 2022

Have you read the Contributing Guidelines on issues?

Motivation

I did a very naïve attempt to migrate to ESM in #5816, but I far underestimated the difficulty in pushing it through. After some pondering, I think this should be rolled out progressively.

Status of ESM

ESM is a new type of Node modules system, replacing the old common JS system (require + module.exports). For the engine, the parsing goal is different ("module" or "script"), so a file needs to be determined as ESM or CJS before executing it, either through the .mjs extension or through the "type": "module" entry in the nearest package.json.

If a file is ESM, it can import from other ES modules. However, it may not be able to import all CJS modules, depending on how the CJS module is structured, because exported symbols, per the ES spec, need to be lexically determined, while CJS can be far more dynamic.

If a file is CJS, it can't import ESM modules, unless the await import() dynamic import is used. However, this is against the norm of how most modules are imported. This effectively means as soon as a considerable number of the dependencies are ESM, we have to migrate ourselves.

Many packages on NPM are now ESM-only, most notably the packages by Sindre Sorhus, and MDX v2, which is the pillarpost of our architecture.

Benefits

  1. Unlock future dependency upgrades. Many popular libraries are seeking to upgrade to ESM; if we can be ESM, we can interoperate with them. For example, MDX v2, chalk...
  2. A similar transpilation target for client and server code; no need for multiple tsconfigs
  3. Permit top-level awaits

Blockers

  1. TypeScript has very lame support of ESM transpilation so far. It seems their deferred ESM support still won't land in 4.6, so in the meantime we may have to run all our Node code with --experimental-specifier-resolution=node and keep the old resolution
  2. Jest doesn't seem to like importing ESM dependencies
  3. We use import-fresh to bypass cache and always import fresh modules for hot reloading, etc. But ES Modules don't expose caching manipulation yet
  4. Community dividing: although the ESM migration will surely be a major version, it means in the foreseeable future after that, some plugins will not be compatible with the new version (especially those that import utils and logger)

Actions needed

  1. Removing __filename and __dirname. These globals don't exist in the ESM scope, replaced by the import.meta.url
  2. Setting target: 'nodenext' in the tsconfig.
  3. Changing our import paths. ESM requires the index.js name and the .js extension to be explicit.
  4. Setting "type": "module" in our package.json.
  5. Tweaking the configuration for related tools?

Plan

  1. Config files: JS config files like docusaurus.config.js and sidebars.js can be allowed in ESM once we figure out how to bypass cache and fresh-import ESM
  2. Utils: this includes utils, utils-validation, logger. They should always be distributed as dual-package because plugin authors are likely to import them as CJS.
  3. Core: the biggest blocker is still the extensive use of import-fresh. Migration to ESM means we can import both CJS and ESM plugin modules with await import. However, because users only interact with the core through its own CLI, we don't have to care about others importing this.
  4. Plugins: migration of plugins can only happen after migration of core (or at least solving import-fresh in the core), because they have to be imported as ESM.

Related issues/PRs

If an issue or PR is related to the ESM migration process, please link to this meta-issue and we will add it to the list below for tracking purposes.

Activity

  1. Josh-Cena
    added
    proposalThis issue is a proposal, usually non-trivial change
    on Jan 31, 2022
  2. Josh-Cena
    pinned this issue on Jan 31, 2022
  3. Josh-Cena
    added
    metaMeta-issue about the project itself. Either project maintenance or a list of other issues.
    better engineeringNot a bug or feature request
    and removed
    proposalThis issue is a proposal, usually non-trivial change
    on Jan 31, 2022

  4. rafaelhorvat commented on Feb 7, 2022

    @rafaelhorvat
    rafaelhorvat
    on Feb 7, 2022

    Yes !!!! +1 !
    Finally I stumbled on this issue, I had updated @mdx-js/react and had errors and it took me a while to understand what happened...

    Error was something like

    _mdx_js_react__WEBPACK_IMPORTED_MODULE_2__.mdx is not a function

    and

    export 'mdx' (imported as 'mdx') was not found in '@mdx-js/react' (possible exports: MDXContext, MDXProvider, useMDXComponents, withMDXComponents)

  5. Josh-Cena commented on Feb 8, 2022

    @Josh-Cena
    Josh-Cena
    on Feb 8, 2022
    CollaboratorAuthor

    @iPurpl3x Your problem is probably not because of ESM, because @mdx-js/react is used on client-side, where Webpack can take care of ESM syntax. Instead, it's because of the API changes in v2. MDX will be compiled to a JSX file containing a line import { mdx } from '@mdx-js/react'; which doesn't exist in the latest MDX version.

  6. Josh-Cena commented on Feb 19, 2022

    @Josh-Cena
    Josh-Cena
    on Feb 19, 2022
    CollaboratorAuthor

    I ran yarn outdated and below are all dependencies that are ESM and we can't upgrade:

    Outdated dependencies 
    Package                          Current Wanted Latest
@mdx-js/mdx                      1.6.22  1.6.22 2.0.0
@mdx-js/react                    1.6.22  1.6.22 2.0.0
boxen                            5.1.2   5.1.2  6.2.1
chalk                            4.1.2   4.1.2  5.0.0
escape-string-regexp             4.0.0   4.0.0  5.0.0
globby                           11.1.0  11.1.0 13.1.1
hast-util-to-string              1.0.4   1.0.4  2.0.0
is-root                          2.1.0   2.1.0  3.0.0
leven                            3.1.0   3.1.0  4.0.0
mdast-util-to-string             2.0.0   2.0.0  3.1.0
rehype-parse                     7.0.1   7.0.1  8.0.4
remark                           12.0.1  12.0.1 14.0.2
remark-emoji                     2.2.0   2.2.0  3.0.2
remark-math                      3.0.1   3.0.1  5.1.1
remark-mdx                       1.6.22  1.6.22 2.0.0
remark-parse                     8.0.3   8.0.3  10.0.1
remark-stringify                 8.1.1   8.1.1  10.0.2
stringify-object                 3.3.0   3.3.0  4.0.1
to-vfile                         6.1.0   6.1.0  7.2.3
unified                          9.2.2   9.2.2  10.1.1
unist-builder                    2.0.3   2.0.3  3.0.0
unist-util-remove-position       3.0.0   3.0.0  4.0.1
unist-util-visit                 2.0.3   2.0.3  4.1.0

    Mostly Sindre Sorhus packages and MDX ones

  7. sachaw commented on Mar 9, 2022

    @sachaw
    sachaw
    on Mar 9, 2022

    This is just going to get worse with time, @Josh-Cena Thanks for the great work, hopefully we can get Docusaurus using ESM soon.

  8. sambacha commented on Mar 11, 2022

    @sambacha
    sambacha
    on Mar 11, 2022

    sindre is working on converting his packages to pure ESM, see https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

    also, the latest @docusaurus/migrate package does not work, the index.mjs is re-written as index.js causing it to fail. Renaming the file extension fixes it

  9. slorber commented on Mar 11, 2022

    @slorber
    slorber
    on Mar 11, 2022
    Collaborator

    also, the latest @docusaurus/migrate package does not work, the index.mjs is re-written as index.js causing it to fail. Renaming the file extension fixes it

    What do you mean by "does not work"? How do you run it and how do you see it not working exactly? It seems to work for me

  10. Josh-Cena commented on Mar 11, 2022

    @Josh-Cena
    Josh-Cena
    on Mar 11, 2022
    CollaboratorAuthor

    @slorber It was a mistake made in the last publish😅 The file is called bin/index.mjs but in package.json it's referred to as bin/index.js. You probably need to run it outside our workspace. See #6897

  11. Josh-Cena commented on Apr 9, 2022

    @Josh-Cena
    Josh-Cena
    on Apr 9, 2022
    CollaboratorAuthor

    TS 4.7 seems promising: https://devblogs.microsoft.com/typescript/announcing-typescript-4-7-beta/#ecmascript-module-support-in-node-js We should be able to investigate after that

  12. slorber commented on Apr 13, 2022

    @slorber
    slorber
    on Apr 13, 2022
    Collaborator

    Yes, also great to have "exports" field support!

  13. arobbins commented on Apr 15, 2022

    @arobbins
    arobbins
    on Apr 15, 2022

    This might be a silly question, but If I'm creating a React library that I want to import into Docusaurus, what is my workaround at the moment? Do I just need to build the library as CJS and not ESM?

  14. 21 remaining items

  15. Josh-Cena commented on Nov 30, 2023

    @Josh-Cena
    Josh-Cena
    on Nov 30, 2023
    CollaboratorAuthor

    @Zamiell type: commonjs is not equivalent for Webpack. When type is absent, Webpack allows both ESM and CJS syntax in the same file; when the module type is unambiguous, it only allows one.

  16. amanaknows commented on Aug 12, 2024

    @amanaknows
    amanaknows
    on Aug 12, 2024 · Hidden as spam

  17. vtempest commented on Dec 27, 2024

    @vtempest
    vtempest
    on Dec 27, 2024

    It is frustrating this is causing endless issues after 2 years. How hard it is just to use requre vs import.

    Every time my docs open they have a great red error require ESM!!!!

    How to avoid setup search when all the community search plugins require.resovle??? what to replace that with???

  18. clicktodev commented on Jan 24, 2025

    @clicktodev
    clicktodev
    on Jan 24, 2025
    Contributor

    Any plans to finish the migration in the near future?

  19. slorber commented on Jan 31, 2025

    @slorber
    slorber
    on Jan 31, 2025
    Collaborator

    No immediate plan, and I'd prefer it to be done incrementally, one package after another.

    If you want to help migrate one package, feel free to submit a PR.

  20. facebook
    deleted a comment from on Feb 20, 2025
  21. facebook
    deleted a comment from on Feb 20, 2025

  22. shreedharbhat98 commented on May 17, 2025

    @shreedharbhat98
    shreedharbhat98
    on May 17, 2025
    Contributor

    Hi @Josh-Cena 👋,

    I’d like to tackle the migration to @docusaurus/logger for this issue.
    Before I dive in, is there anything you’d like me to keep in mind?

    Thanks for your guidance!

  23. slorber commented on May 19, 2025

    @slorber
    slorber
    on May 19, 2025
    Collaborator

    Thanks for your help proposal @shreedharbhat98

    The plan is to upgrade Docusaurus v4 to Node 22 soon, uncloking the synchronous require(esmModule). Once we have upgraded, we can start converting some of our packages to ESM incrementally.

    I'd prefer we don't do anything before the upgrade because even though some packages can probably be converted to ESM today, in practice, the 2 ones we already converted caused some troubles already.

  24. CMCDragonkai commented on May 25, 2025

    @CMCDragonkai
    CMCDragonkai
    on May 25, 2025 · Hidden as off-topic

  25. slorber commented on May 27, 2025

    @slorber
    slorber
    on May 27, 2025 · Hidden as off-topic

  26. CMCDragonkai commented on May 28, 2025

    @CMCDragonkai
    CMCDragonkai
    on May 28, 2025 · Hidden as off-topic

  27. slorber commented on May 28, 2025

    @slorber
    slorber
    on May 28, 2025 · Hidden as off-topic

  28. vtempest commented on May 28, 2025

    @vtempest
    vtempest
    on May 28, 2025

    https://github.com/vtempest/grab-api/tree/master/docs-config
    https://github.com/vtempest/grab-api/blob/master/docs-config/package.json

    Docusaurus with OpenAPI and Typedoc and Lunr Search
    This is the ultimate setup for Docusaurus to unlock Docs-Driven Development where your docs are autogenerated by your code so others can understand all functions clearly.

    There are insane compatibility headaches especially with the paloaltonetworks openapi docusaurus versions please check up on these plugins so they are compatible

  29. slorber
    unpinned this issue on Mar 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    better engineeringNot a bug or feature requestmetaMeta-issue about the project itself. Either project maintenance or a list of other issues.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions