Etsy's search index grew from 2 GB to over 28 GB, and Solr's HTTP replication couldn't feed the replicas fast enough. So they made the search servers seed to each other, like a torrent.

Etsy started using Solr for search in January 2009, in a standard primary-replica setup. All changes to the search index are written to the primary; the replicas are read-only copies that serve production traffic. To stay in sync, each replica polls the primary and, when the index changes, downloads the new files over HTTP.

To keep searches fast, Etsy optimizes its indexes nightly — and optimization creates a completely new copy of the index. So every night, the full weight of a freshly rebuilt index has to travel from the primary out to every replica.

Over two years, Etsy’s search indexes grew from 2 GB to over 28 GB. As they added boxes, a disturbing trend appeared: Solr’s HTTP replication took longer and longer after each nightly optimization. Benchmarking pinned the ceiling at just 2 MB to 8 MB per second. Tweaking compression and chunk size did nothing.

The pain was concrete. The 4 GB optimized listings index took over an hour to reach 11 replicas. Standing up a brand-new replica meant pulling every index at 8 MB/s — over 4 hours. A netcat test flooded the network card cleanly, so the bottleneck wasn’t the wire; it was HTTP replication itself.

The obvious idea was to send the index once and let it fan out. Etsy first tried mrsync, a multicast-UDP flavor of rsync — in the development environment it moved the entire index in under 3 minutes.

Then they tried it in production, and it took the whole site down.

BitTorrent is a peer-to-peer protocol built to move large files by having every downloader also upload to its peers. Etsy’s Ops team first proved it out with a tool called herd sitting on top of BitTornado: their largest index moved in 15 minutes on the first try, and after 8 hours of tuning — plus pigz for compression — they cut the biggest index’s replication from 60 minutes to just 6 minutes.

To make this a permanent replacement for HTTP replication rather than a nightly side-process, they integrated BitTorrent directly into Solr. On commit, the primary hashes the index files, writes a torrent, and seeds it; replicas fetch the torrent over HTTP, hash-verify what they already have, and pull only the missing pieces over BitTorrent. Crucially, every replica seeds the index too — so new or stale replicas can be filled by the whole swarm, not just the primary. Of the Java BitTorrent libraries they surveyed, only ttorrent fit; they forked it to add multi-file torrents and fix support for files larger than 2 GB.

It wasn’t free. BitTorrent’s hash verification means the primary must SHA1 every index file to build the torrent, and each replica must hash-check its files against it — roughly 60 seconds for 9 GB, and Java’s SHA1 isn’t thread-safe, forcing it to run serially. That created a 2-minute lag before any transfer began. Etsy wrote a thread-safe SHA1 and a DigestPool to hash in parallel, cutting 9 GB down to about 16 seconds and the total lag to around 32 seconds — trading CPU for speed.

The upside beyond raw throughput: because every file is hash-verified, replica indexes are provably consistent with the primary, which also fixed a nagging version-mismatch bug that HTTP replication had with static indexes.

When one source has to feed many identical consumers, the bottleneck isn’t the pipe — it’s the star topology, and the fix is to let the consumers feed each other.