THE MIGRATION LINE
INCIDENT // GITHUB-2022 SHIPPED

Every git push to GitHub ran Ruby hooks that spent about 880 milliseconds loading dependencies before doing any work — so pushing nothing at all took over two seconds.

GITHUB · 2022 · GO / RUBY / PERFORMANCE / GIT
System stress over time Breach at T+2
880 ms → 10 ms Hook time (median)
over 2 seconds Empty push, before
~95% Improvement
SITUATION

A guard on every push

Every time anyone pushes to GitHub, server-side hooks run first. They date back to 2013, when they were created to warn users that a repository had been renamed, and they’ve since taken on real gatekeeping: rejecting files over 100MB, verifying Git LFS, and generally checking that what you’re pushing is allowed.

They were written in Ruby, as part of GitHub’s Rails monolith, and they ran as a subprocess for each push.

COMPLICATION

Two seconds to push nothing

That last detail is where it goes wrong. A fresh Ruby subprocess has to load its dependencies before it can execute a single check — and inside a Rails monolith, that’s a lot of dependencies. Loading them took roughly 880 milliseconds on average, which was very nearly the entire runtime of the hook. Almost none of that time was spent deciding anything about your push.

QUESTION

Can you just make the startup cheaper?

The obvious move is to attack the dependency load directly: figure out what the hooks actually need and stop loading the rest. GitHub tried exactly that, selectively removing Rails dependencies, and it worked — they cut 350 to 400 milliseconds.

It wasn’t enough. Nearly half a second of savings, and the process was still spending the bulk of its life preparing to run rather than running. That’s the tell: when a genuinely successful optimization leaves the shape of the problem intact, the problem isn’t the cost of the startup. It’s that there’s a startup at all.

ANSWER

Take the hooks out of the monolith entirely

So GitHub extracted the hooks into a dedicated service written in Go — outside the Rails monolith. Pulling them out of the app was what removed most of the boot time; Go then took the last few milliseconds, and it was a natural fit because the Git Systems Team already worked in Go and had spent years moving service code out of the monolith into it.

It ran on github.com for months before being shipped to self-hosted customers in GitHub Enterprise Server 3.4.

TAKEAWAY

Where the time actually went

This is the Ruby-to-Go story that gets told as a language win, and it mostly isn’t one. The 880 milliseconds weren’t Ruby being slow at checking file sizes; they were the cost of standing up a monolith’s worth of dependencies to answer a question that needed almost none of them. Go closed the final gap. Leaving the monolith closed the rest.

If a process spends most of its life getting ready to work, the fix isn’t a faster startup — it’s not starting.

Source — read the original

https://github.blog/engineering/architecture-optimization/improving-git-push-times-through-faster-server-side-hooks/

A plain-language, AI-drafted and human-edited retelling of the article published on github.blog, reorganized and explained in our own structure and words, with original analysis in the editor's note above. The facts, numbers, and decisions belong to the original author and are not altered. For the full depth, read the source.

← All systems