Post

Ranked #3 on Hacker News with 658 points and 104 comments.

At the end of last year, our uptime was pretty shaky . You can see this trend on our status page , and that instability continued into the new year. Many of these outages were caused by a single bug, deep in SQLite . It took months of intense forensics to track it down.

Now we’re in summer, we’re confident that we’ve found the bug, that we understand it—and more importantly, that we’ve fixed it.

We know our customers expect Tailscale to be a reliable service, and for several months we didn’t live up to that promise. That’s disruptive, and we’re sorry. We’re publishing this blog post to explain what went wrong, how we responded, and how we ultimately helped to uncover a long-standing bug in the heart of the SQLite database.

While our clients interact with our control plane as a single public endpoint ( controlplane.tailscale.com ), internally, our control plane is split into a series of coordination servers (or “shards”). Each tailnet lives on one internal shard at a time, but can migrate seamlessly from one to another. These shards are an internal implementation detail: you don’t know what shard your tailnet is on, and you never need to.

Each shard has an SQLite database that holds all the information about the tailnets on that shard. A single Go process exclusively accesses that database, and serves the control plane for those tailnets. This single-writer design is exactly how SQLite is meant to be used.

We’ve used SQLite as our primary database since 2022 , and we chose it because it's well-known, reliable, and widely used. SQLite is “boring technology” —in a good way. Many companies use SQLite in much larger deployments without issue, and we expected the same stress-free usage.