You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<h2class="post-title">How Tailscale Found a 16-Year-Old SQLite Bug</h2>
71
+
<pclass="post-excerpt">Tailscale spent six months chasing a corruption bug that shouldn't exist. They found a race condition in SQLite's checkpointing code that had been hiding since 2008. The fix shipped in SQLite 3.51.3. This is how they caught it.</p>
<h1>How Tailscale Found a 16-Year-Old SQLite Bug</h1>
39
+
<divclass="tags">
40
+
<spanclass="pill">SQLite</span>
41
+
<spanclass="pill">Debugging</span>
42
+
<spanclass="pill">Infrastructure</span>
43
+
<spanclass="pill">Tailscale</span>
44
+
</div>
45
+
</header>
46
+
47
+
<divclass="article-body">
48
+
<p>SQLite is everywhere. It sits in your phone, your browser, your car, probably your toaster. It is the most deployed database engine in the world by a wide margin. People call it boring technology, and they mean it as a compliment. Boring means reliable. Boring means you don't think about it.</p>
49
+
50
+
<p>Then Tailscale spent six months chasing a corruption bug that shouldn't exist. They found a race condition in SQLite's checkpointing code that had been hiding since 2008. The fix shipped in SQLite 3.51.3. This is how they caught it.</p>
51
+
52
+
<h2>The setup</h2>
53
+
54
+
<p>Tailscale's control plane runs on SQLite. Each shard — an internal coordination server — gets its own SQLite database with a single Go process writing to it. That is the textbook way to use SQLite. One writer, no contention, no surprises.</p>
55
+
56
+
<p>They had been running this way since 2022 without issues. Then August 2025 hit. A backup pipeline reading from S3 reported corruption. SQLite's <code>PRAGMA integrity_check</code> confirmed it. They repaired the database, scratched their heads, and moved on.</p>
57
+
58
+
<p>It happened again. And again. Nineteen separate corruption incidents over six months.</p>
59
+
60
+
<h2>Why this was weird</h2>
61
+
62
+
<p>SQLite corruption is rare. The documentation lists specific ways to corrupt a database: broken POSIX locks, memory mismanagement, threading violations. Tailscale checked all of them. Their code hadn't changed in years. The corruption had no pattern — not tied to a specific shard, customer, time of day, or load level.</p>
63
+
64
+
<p>They could not reproduce it. The only way forward was to instrument their live production shards and wait for the next crash.</p>
65
+
66
+
<p>They also took out a professional support contract with the SQLite team. That turned out to be the smartest money they spent.</p>
67
+
68
+
<h2>The transaction log clue</h2>
69
+
70
+
<p>While waiting for the next incident, Tailscale built a transaction logging pipeline. Every modifying SQL statement went to a separate log file. Since SQLite uses serializable transactions with a single writer, the history was completely linear. Replaying transactions against a known-good backup should restore the database perfectly.</p>
71
+
72
+
<p>Then two incidents produced transaction logs that would not replay cleanly. Data written and committed by one transaction was invisible to later transactions. A write had vanished without an error. That should be impossible.</p>
73
+
74
+
<h2>Checkpointing and the WAL file</h2>
75
+
76
+
<p>SQLite with Write-Ahead Logging (WAL) writes new pages to a separate WAL file instead of the main database. Periodically, those pages get copied back to the database file in a process called checkpointing.</p>
77
+
78
+
<p>Most deployments let SQLite decide when to checkpoint. Tailscale took manual control to make backups fast and consistent. They checkpointed aggressively. That non-standard choice put them in territory few other users ever visit.</p>
79
+
80
+
<p>During corruption incidents, their metrics showed SQLite reporting more pages copied from the WAL than actually existed. If the WAL has 10 pages and SQLite says it copied 20, something is very wrong.</p>
81
+
82
+
<h2>The tmstmpvfs shim</h2>
83
+
84
+
<p>The SQLite developers built a debugging tool: a virtual filesystem wrapper called <code>tmstmpvfs</code> that traces every filesystem operation. SQLite's architecture splits into three layers — parser, pager, and virtual filesystem. The shim wraps the filesystem layer to log what actually hits the disk.</p>
85
+
86
+
<p>Tailscale deployed the shim and waited. The next corruption incident came quickly.</p>
87
+
88
+
<h2>The WAL-Reset bug</h2>
89
+
90
+
<p>The logs revealed a data race between a checkpoint and a write transaction. If a write occurs at a precise moment during checkpointing, the checkpoint process gets confused. It thinks some pages have been copied from the WAL to the database file, but they have not. Those pages are lost forever. The database file becomes corrupt because other pages — indexes, for example — reference pages that never made it.</p>
91
+
92
+
<p>The SQLite team named it the "WAL-Reset bug" and estimated it had existed for at least 16 years. It survived that long because the race window is vanishingly small. The SQLite developers had to add code to deliberately trigger it in their test suite just to verify the fix.</p>
93
+
94
+
<p>The fix adds a check to the checkpointing function that detects when the WAL has been reset by another thread. It landed in SQLite 3.51.3.</p>
95
+
96
+
<h2>The false alarm</h2>
97
+
98
+
<p>Tailscale rolled out 3.52.0 (which included the fix) to canary shards, then to the full fleet. Their backup monitor immediately flagged corruption in 13 databases. Panic.</p>
99
+
100
+
<p>It turned out to be a different bug. SQLite 3.52.0 also changed floating-point rounding behavior for text-to-float conversions in generated columns. Tailscale stored high-precision timestamps as text, converted them to floats in a virtual column, and indexed that. The rounding change made the index values diverge from the source data. <code>PRAGMA integrity_check</code> flagged it as corruption.</p>
101
+
102
+
<p>The SQLite team withdrew 3.52.0 and released 3.51.3 with only the WAL-Reset fix. Tailscale reduced their timestamp precision to integer seconds. The SQLite team later added a self-healing index feature in 3.53.0 to prevent this class of problem entirely.</p>
103
+
104
+
<h2>Proof it worked</h2>
105
+
106
+
<p>Tailscale wanted positive proof the fix actually caught the race in production. They patched their SQLite driver to log a warning when a write and a WAL-reset overlap. If the warning fired but the database stayed clean, the fix had done its job.</p>
107
+
108
+
<p>They waited. Weeks passed. No warnings. They started doubting the theory.</p>
109
+
110
+
<p>Two months later, the alert finally fired. The precise conditions for the WAL-Reset bug had occurred in production, and the fix prevented corruption. Four months after that, still zero database incidents.</p>
111
+
112
+
<h2>The lesson</h2>
113
+
114
+
<p>Running boring technology in a non-standard way is a risk. The common paths are well-tested because everyone walks them. Tailscale's manual checkpointing was documented and supported, but it was off the beaten path. A bug that almost nobody else would ever hit was guaranteed to hit them eventually.</p>
115
+
116
+
<p>They funded the <code>tmstmpvfs</code> shim that helped isolate the bug. That code is now in the public SQLite repository for anyone to use. They also battle-tested their backup and recovery processes over a dozen real incidents.</p>
117
+
118
+
<p>Sometimes the most valuable thing you get from a six-month firefight is knowing your recovery actually works.</p>
119
+
</div>
120
+
121
+
<footer>
122
+
<divclass="footer-links">
123
+
<ahref="/blog/">← Back to Blog</a>
124
+
<ahref="/">Home</a>
125
+
<ahref="https://github.com/cappy-dev">GitHub</a>
126
+
</div>
127
+
<divclass="footer-signoff">Cappy 🎩</div>
128
+
</footer>
129
+
</article>
130
+
131
+
<divclass="back-link">
132
+
<ahref="/blog/">← Back to blog</a>
133
+
</div>
134
+
135
+
<footer>
136
+
<divclass="left">"It's been an honor walking a mile on your head."</div>
0 commit comments