Moving into 2024 with latest JavaScript Ecosystem
A journey of going from Yarn & Lerna (legacy) to Pnpm & Lerna (Powered by Nx)
I actively manage an Open Source project called Operational Transformation on GitHub. The crisp of it is to have lightweight packages to handle eventual consistent updates on a text content with multiple editors (Ace and Monaco are supported out of the box).
During inception, the standard of Monorepo was to use Lerna and then provide it with peripheral tools it required. This was back in the July of 2021. There had been a lot in between. Lerna was not maintained for a long time and then briefly died, only to be resurrected by Nrwl, the company behind Nx.
While this drama was taking shape in the front stage, a silent change has been passed on in the dressing room. The traditional way of handling dependency and packages in the ecosystem had moved from traditional local environment store to a global store-based Plug and Play system. The pioneer on this during early days was in fact then de-facto package manager Yarn. However, it did not really catch-up so well with the community while it continues to use the older version quite comfortably. Keep a note that the older version is no longer being maintained since 2019 beyond security fixes.
Enter The Dragon: Pnpm
Pnpm, as the name suggests, is a Plug and Play alternative of Npm. It is high time we discuss what is Plug and Play. In the dawn of Node package managers, everything was up for grab. With no authentication at place, npm might take a second or two to query the registry and download packages into local file system. With a layer of Authentication and Proxy, this gets increased significantly. And then with transitive dependency, this gets multifolded. Another problem that arises, is the high amount of storage usage, since every instance of the dependency has to be installed. The two step solution that eventually came up was to use a runtime cache to hash and store metadata so that only different packages are installed (Classic Yarn does this very well), and using flat directory structure as opposed to hirearchial to minimize the number of install of same package and keep the path short (both Classic Yarn and Npm does this). However, this were mostly remedies to more fundamental problem as Kat aptly discussed in her presentation.
Pnpm takes the problem and solves it in a bit different way. They keep the directory structure of node_modules nested to make the depenencies hirearchial. However, all the packages are symlinked with their store location (~/.pnpm) where they are all kept in a flat manner in the format of <package_name>@<package_version>. This ensures that any package in the machine would never be installed twice. Also there will not be any conflicts among transitive dependencies, that too without creating long paths. In fact, it also supports running application without even installing the dependencies first but install them on demand during the runtime.
Lerna Unchained with Nx/Nx Cloud
Lerna version 6 was released on October 2022 which came default packed with Nx and other Nrwl tools. The core change was around task resolution and scheduling, which earlier depended upon two community maintained Open Source projects p-map and p-queue. With the introduction of Nx, this became blazingly fast instantly.
With that, a lot of legacy commands such as bootstrap, add and link got deprecated in favour of package management tools of preference. Workspaces became the standard of Lerna and quite some CLI options got redefined under Nx terminologies. Also since Nx powers it, all cloud native power of it starting from Distributed Task Execution to Distributed Cache are now available with Lerna as well.
All Roads Lead to Rome
Enough talk, let’s get to the code. We will obviously start with package.json and update lerna to 8.x.x. We then configure .npmrc to the following setup and install all dependencies using pnpm.
# Lock Dependency Versions
save-exact=true
# Hoist Dependency to the Root node_modules
hoist=true
shamefully-hoist=true
# Use LTE Node version
engine-strict=true
node-version=20.10.0
use-node-version=20.10.0
# Workspace Settings
link-workspace-packages=true
prefer-workspace-packages=true
shared-workspace-lockfile=true
# Script Execution Configuration
enable-pre-post-scripts=trueWe then update our lerna.json to take full advantage of the latest features.
"$schema": "https://json.schemastore.org/lerna.json",
"command": {
"run": {
- "npmClient": "yarn",
+ "npmClient": "pnpm",
"scope": "@otjs/*"
},
},
- "npmClient": "yarn",
+ "npmClient": "pnpm",
"npmClientArgs": ["--frozen-lockfile"],
- "packages": ["packages/*"],
- "useWorkspaces": true,
"version": "0.2.1"
}We make further changes to package.json to make it in sync with the best practices with Lerna.
"scripts": {
"test": "TEST_ENV=true jest --colors",
- "test:ci": "yarn test --ci --silent"
+ "test:ci": "pnpm run test --ci --silent"
}
- "workspaces": [
- "packages/*",
- "__tests__/*",
- "examples/*"
- ],
+ "engines": {
+ "node": ">= 20",
+ "pnpm": ">= 8"
+ },Also it is good practice to version the First-party dependencies within the project like workspace:* to ensure it symlinks local workspace projects.
Finally we update our build pipelines to fully utilise automation with Pnpm.
build:
name: Build Artifacts
runs-on: ubuntu-latest
strategy:
matrix:
node-version: ["20.10"]
pnpm-version: [8]
steps:
- uses: actions/checkout@v3
- - uses: actions/cache@v2
+ - uses: pnpm/action-setup@v2
+ with:
+ path: |
+ node_modules
+ $HOME/.npm
+ $HOME/.yarn-cache
+ key: ${{ runner.OS }}-node-${{ hashFiles('**/yarn.lock') }}
+ restore-keys: |
+ ${{ runner.OS }}-node-
+ ${{ runner.OS }}-
+ version: ${{ matrix.pnpm-version }}
- - name: Install Lerna
- run: yarn global add lerna@4
+ - uses: actions/setup-node@v3
+ with:
+ cache: "pnpm"
+ node-version: ${{ matrix.node-version }}
+ registry-url: "https://registry.npmjs.org"
- name: Install Dependencies
- run: lerna bootstrap --ignore-scripts
+ run: pnpm install
- name: Build Artifacts
- run: yarn build
+ run: pnpm buildAnd that’s it. Now we have the power of Pnpm and Lerna in our project.
