Airbnb's Mocha suite took 12+ minutes in CI, 45 minutes locally, and flaked on 12% of builds. Jest cut it to 4m30s and ~1% flake — once they found the 480ms hiding in their spec helper.

Airbnb’s JavaScript test suite ran on Mocha across several thousand test files. On the build server, after work to parallelize across machines, it took 12+ minutes. Locally it took around 45 minutes — and sometimes it never completed at all, because running the whole suite in a single thread put too much pressure on memory.

Code coverage was its own project. Airbnb rolled out coverage with Istanbul and Mocha in January 2016 and found that instrumenting source files was expensive enough to add an unreasonable amount of time to the tests. The workaround was custom logic to batch tests into chunks, run them in separate processes, collect coverage on each, and merge it all into one report at the end — running in a separate CI job from the test suite itself.

On the same beefy CI build machines, the full Jest suite ran in 4 minutes 30 seconds. Locally it dropped to 14.5 minutes. Jest gets there partly through parallelization — but Airbnb was already parallelizing, by splitting the list of test files evenly across workers, which left workers stuck with abnormally slow or abnormally fast queues and wasted CPU cycles. Jest instead uses a round-robin approach and runs the slowest tests first, squeezing more out of the same processors. It also keeps a babel transform cache shared across processes, so CPU goes to running code rather than re-transforming it.

Because every test file runs in isolation, it becomes impossible for one file’s side effects to fail another — particularly errors thrown inside a setTimeout after a test has already finished. After migrating and fixing the tests that failed when run alone, the flake rate fell to about 1%. That saves developers hours per work day that used to go into waiting for a build to fail and rerunning it until it passed, and when flake does happen now, it is guaranteed to be coming from inside the file that failed.

The tests themselves barely moved. Airbnb had invested in a strict coding style for tests, partly to mitigate flakiness and partly so tests looked the same across teams, and keeping that property through the migration mattered to them. The Jest and Mocha APIs are similar with slightly different function names, so a single snippet was enough to paper over the differences between how Airbnb used one and the other — plus a few test calls swapped one-for-one to it, which they settled on as the standard.

Jest was faster out of the box, but at first not as much faster as expected. Profiling pointed at spec_helper.js, the global setup file Airbnb used to wire up convenience helpers so developers didn’t have to configure them in every test — including chai-enzyme, for testing React with Enzyme.

Under Mocha, which isn’t parallelized, that file ran exactly once. Under Jest, each test file gets a clean virtual machine, so the helper reran for every single file. Importing chai-enzyme starts a chain that pulls in all of Enzyme, which pulls in React and ReactDOM: 480ms, even for tests that never touch React. Multiplied across several thousand files, Airbnb was spending over a minute of every run just setting up one library. The fix was to use the callback on jest.mock() to intercept Enzyme imports and load chai-enzyme only for the tests that actually need it.

That was the trade they accepted: isolation is what killed the flake, and isolation is also what turned a once-per-run cost into a once-per-file cost. Developer feedback afterward was, in Airbnb’s telling, nothing but positive — one engineer reported that what used to take three hours to write now took about thirty minutes.

When your test runner stops sharing one process, every shortcut you took in global setup stops being free — and the convenience that cost you 480 milliseconds once now costs you 480 milliseconds per file.