Mozilla spent about a year replacing google-breakpad's minidump processor with a pure-Rust rewrite, tested it against the old C++ system on real production crash reports, and shipped it — half the runtime, more reliable, and, by the article's own admission, still not done fighting the format's edge cases.

Minidumps started as a Microsoft idea: instead of shipping a full multi-gigabyte coredump off a crashed machine, collect just the important regions — the stacks, the crash site, some process and system metadata — into a file small enough to actually send over the internet. It’s a Windows-only format, with no discussion of other platforms in Microsoft’s own documentation. Around 2006-2007, Google extended it to Linux, macOS, BSD, and Solaris, and those extensions became google-breakpad. Mozilla adopted it heavily around the same time, because Firefox’s crash-reporting infrastructure back then — called Talkback — was, in the article’s words, “miserable.”

Years later, Google mostly moved on to a successor called Crashpad, and Breakpad became increasingly under-staffed. By the time Aria Beingessner started working on Mozilla’s crash reporting, Mozilla had given up upstreaming fixes and was running its own patched fork — and “every change to Breakpad filled us with dread.” Proposed improvements to the crash-reporting infrastructure routinely stalled out at the step of actually implementing them in Breakpad.

The underlying reason Breakpad was miserable to work on wasn’t Breakpad specifically — it was the problem itself. “Parsing and analyzing minidumps is basically an exercise in writing a fractal parser of platform-specific formats nested in formats nested in formats,” across many operating systems and hardware architectures, reading input that’s frequently garbage: a corrupted stack, a broken toolchain’s malformed debug info, a minidump writer that “completely freaked out and wrote a bunch of garbage to one stream.” The parser has to produce a backtrace anyway, every time, no matter how bad the input is.

Mozilla built and deployed two pieces in Rust: dump_syms, which turns native build artifacts into symbol files, and rust-minidump — specifically its command-line interface, minidump-stackwalk — which parses and analyzes the minidumps themselves. (A Rust-based minidump writer, the client-side half of the problem, was still in progress with help from Embark Studios, not yet recommended for use.) The stackwalker, which reconstructs a thread’s backtrace from registers and stack memory, was “the most complicated and subtle part of the new implementation” — its STACK WIN/CFI parser and evaluator alone run 1,700 lines, 700 of them roughly 80 test cases covering every corner case the team could think of. Two known-failing tests were checked in on purpose, as an honest record of unfixed gaps, before both were resolved.

The team didn’t just unit-test in isolation. They built minidump-synth, a synthetic minidump generator descended from earlier work by Ted Mielczarek (the original rust-minidump author) and Jim Blandy on Breakpad’s own test suite; ported most of Breakpad’s stackwalker tests over directly; added CLI snapshot tests with the insta crate; and built a dedicated tool called socc-pair that pulls real crash reports from Mozilla’s crash-reporting system and diffs rust-minidump’s output against Breakpad’s, frame by frame, on production data.

Staging deployment, handling about 10% of the traffic that hit production, immediately surfaced new corner cases: “we very quickly found several new corner cases and we were getting tons of crashes, which is mildly embarrassing for the thing that handles other people’s crashes.” They were, the team says, “all fairly easy for us to fix.” Turning it on in production was, in the author’s own words, a moment of “pure terror” — and then: “It worked fine. After all that stress and anxiety, we turned it on and it was fine. Heck, I’ll say it: it ran well.” By the time of publication, Mozilla had it running as Firefox’s production crash-processing backend for six months, in half the runtime of the old system and, they say, more reliably — fixing several issues that had existed in the old Breakpad implementation along the way.

The honesty doesn’t stop at the win. Despite the synthetic tests, the ported Breakpad test suite, and months of differential testing against production data, “the fuzzer still kicked our butts afterwards” — an external researcher’s fuzzing pass found bugs, including an integer overflow in the STACK WIN preprocessing step, that all of the prior testing had missed. That was left as the subject of a second part the team flagged as still to come.

No amount of synthetic tests, ported test suites, and production diffing substitutes for a hostile fuzzer finding the one input nobody thought to write by hand — “everything is broken” isn’t false modesty, it’s the correct starting assumption for a parser that has to make sense of a program’s own corrupted memory.