Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 32 additions & 10 deletions src/app/blog/what-we-got-wrong/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,17 @@ function Entry({ correction }: { correction: Correction }) {
</p>
<p>
<Life correction={correction} />{' '}
{correction.guard ? (
{/* An open row has no guard by definition: nothing is stopping a thing
that is still happening. The guard sentence belongs to fixed rows. */}
{correction.fixedOn === null ? null : correction.guard ? (
<>
The test that fails if it comes back is <code>{correction.guard}</code>.
</>
) : (
<>The check that catches it is not merged yet, so this row has no guard behind it.</>
<>
<strong>Nothing stops this one coming back.</strong>{' '}
{correction.guardNote ?? 'We have not written the check yet.'}
</>
)}
</p>
</Section>
Expand Down Expand Up @@ -78,10 +83,25 @@ export default function WhatWeGotWrongPost() {
pinned by tests that run the same engine that deals the cards, and the way you can tell we
mean it is that it keeps catching us.
</p>
<p>
<strong>One of them is still wrong as this goes up.</strong> It is first in the list,
because burying it would be the joke writing itself.
</p>
{open.length > 0 ? (
<p>
<strong>
{open.length === 1
? 'One of them is still wrong'
: `${open.length} of them are still wrong`}{' '}
as you read this.
</strong>{' '}
{open.length === 1 ? 'It is' : 'They are'} first in the list, because burying{' '}
{open.length === 1 ? 'it' : 'them'} would be the joke writing itself.
</p>
) : (
<p>
<strong>Nothing on this list is open as you read this.</strong> One was when this went
up on 24 August: the site was serving a build with its account system missing from it.
That was fixed the next evening, and the row for it now carries both dates, a correction
to the date we first printed, and no guard at all.
</p>
)}
</Section>

{open.map((correction) => (
Expand Down Expand Up @@ -127,21 +147,23 @@ export default function WhatWeGotWrongPost() {
downloading the JavaScript the live site actually serves and reading the configuration
out of it
</Item>
<Item>reading this post against the live site the morning after publishing it</Item>
</List>
<p>
The tests come after. Their job is that a fixed thing stays fixed, which they are good at
and which nothing else does. But a green build tells you the code is right. It does not
tell you that what shipped is right, and the open row at the top of this page is the
expensive version of that distinction: every test passed, every build was green, and the
site quietly went out without part of itself.
tell you that what shipped is right, and the account row is the expensive version of that
distinction: every test passed, every build was green, every one of those builds had the
configuration in it, and the site was serving a different build entirely.
</p>
</Section>

<Section title="What this page is promising">
<p>
That every future one lands here too. The list is generated from a registry the test suite
reads, so a row cannot be quietly dropped and a corrected sentence cannot creep back into
the site without the build failing.
the site without the build failing. Where a row has no guard behind it, it has to say so
in as many words, which is why one of them does.
</p>
<p>
It is not promising there will not be more. There will be. The interesting number is not{' '}
Expand Down
2 changes: 1 addition & 1 deletion src/config/blog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ export const BLOG_POSTS: BlogPost[] = [
slug: 'what-we-got-wrong',
title: 'Everything we have published that was wrong',
description:
'Seven false claims we shipped, what each one said, how long it served, how we found it, and the test that now fails if it comes back. One of them is still wrong.',
'Every false claim we have shipped: what each one said, how long it served, how we found it, and the test that now fails if it comes back. The newest one is a sentence in this post.',
date: '2026-08-24',
},
{
Expand Down
46 changes: 41 additions & 5 deletions src/config/corrections.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,14 @@
// rarity claim, for instance, still contains "holds exactly, all the way down
// the list" in its corrected form, and what was wrong was saying it without
// "on five cards" in front.
//
// Two rows do not work that way, and both are marked rather than excused.
// A blog post is a dated record, so a wrong one keeps its sentence and gains a
// correction note. And a row can be `fixedInProduct`, meaning the words were
// right and the thing they described was missing: the fix was a deploy, the
// sentence is the one we want to keep saying, and banning a fragment of it
// would ban the correction. Both carry `gone: null` on purpose, and the test
// knows the difference between that and a row nobody finished.

export interface Correction {
/** Stable handle. Used as the anchor and the test's failure message. */
Expand All @@ -38,31 +46,59 @@ export interface Correction {
fixedOn: string | null
/** How it was found. Never "a test", because no test has ever found one. */
caught: string
/**
* Set when the sentence was right and the product was not, so the fix was a
* deploy rather than an edit. Such a row keeps `gone` null: the words this
* row is about are the ones we now want to go on saying.
*/
fixedInProduct?: true
/**
* A fragment of the false sentence that must not appear in the site's source
* again. Null only while the claim is still live.
*/
gone: string | null
/** The test that fails if it comes back, as a path in this repository. */
guard: string | null
/**
* Why a fixed row has no guard, in one sentence, for the page to print. A
* fixed row names a guard or says why it has none; "we did not get round to
* it" is an answer, and a silent null is not.
*/
guardNote?: string
}

/** Open first, then fixed, newest correction first. Pinned by the test. */
export const CORRECTIONS: readonly Correction[] = [
{
id: 'corrections-post-open-row',
where: ['/blog/what-we-got-wrong'],
said: 'One of them is still wrong as this goes up.',
wrong:
'It was not. This post was written on the morning of 24 August, when the row below it was open. The site was republished with its database configuration at 17:44 that afternoon and the post did not go live until 01:03 the next morning, so the page listing our false claims opened with one, seven hours stale. The sentence was typed rather than read off this list, which is the only way it could have been.',
liveFrom: '2026-08-25',
fixedOn: null,
caught:
'Checking the live site against the row below, the next morning. The post was the thing doing the checking and it turned out to be the thing that was wrong.',
gone: null,
guard: null,
},
{
id: 'privacy-account-section',
where: ['/privacy'],
said: 'Sync is "off unless you turn it on, in Settings, under Account".',
wrong:
'The build serving playpip.io was made without its database configuration, so the app decides at load time that accounts are unavailable and removes every account surface from itself. There is no Account section in Settings to turn anything on with, and nobody can sign in.',
liveFrom: '2026-08-03',
'The site was serving a build made without its database configuration, so the app decided at load time that accounts were unavailable and removed every account surface from itself. There was no Account section in Settings to turn anything on with, and nobody could sign in.',
liveFrom: '2026-08-23',
liveFromNote:
'We cannot date the start. The bundle currently served is from 23 August; whether earlier builds carried the configuration is not something we can see from outside. 3 August is the day accounts shipped, which is the worst case.',
fixedOn: null,
'This row said 3 August when it went up, on the reasoning that we could not see from outside which earlier builds carried the configuration. We could, and we should have looked before writing a date down: every deployment this repository has ever published keeps a permanent address, and the ones from 9, 14 and 15 August all carry it. What actually happened is narrower, and it now has a name. Two different things were publishing this site: our own deploy, which runs the test suite first and supplies the configuration, and a hosting-side integration nobody had accounted for, which builds every push by itself and does neither. On 23 August our deploy stopped at its security-audit step at 16:51:03, the other one finished at 16:51:57, and its build is the one that served playpip.io for the next day.',
fixedOn: '2026-08-24',
caught:
'A check that downloads the JavaScript playpip.io actually serves and reads the configuration out of it. Every test passed and every build was green throughout: the fault is in the machine that builds the site, not in the code, so nothing in the repository disagreed with anything else in the repository.',
'A check that downloads the JavaScript playpip.io actually serves and reads the configuration out of it. Every test passed and every build was green throughout, and both were telling the truth: the build this repository makes was correct and it was not the one being served.',
fixedInProduct: true,
gone: null,
guard: null,
guardNote:
'None. The check that reads the configuration out of a bundle is scripts/assert-sync-config.mjs, and it can only inspect a build made by the deploy it is attached to, which is not the deploy that went wrong. The fix that would actually hold is for one thing to publish this site rather than two, and that is a setting in the hosting account rather than a line in this repository.',
},
{
id: 'data-never-leaves',
Expand Down
60 changes: 56 additions & 4 deletions tests/corrections.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,23 +51,59 @@ test('dates are ISO, and a fix never precedes the claim', (t) => {
*/
const isPost = (c: (typeof CORRECTIONS)[number]) => c.where.every((p) => p.startsWith('/blog/'))

test('an open row has no fix date and no guard, a fixed row has both', (t) => {
test('an open row has no fix date and no guard, a fixed row accounts for both', (t) => {
for (const c of CORRECTIONS) {
if (c.fixedOn === null) {
t.is(c.guard, null, `${c.id}: open rows cannot claim a guard that has not shipped`)
t.is(c.gone, null, `${c.id}: open rows are still saying it`)
t.falsy(c.guardNote, `${c.id}: an open row explains itself in "wrong", not in "guardNote"`)
t.is(daysLive(c), null, `${c.id}: an open row has no duration`)
continue
}
t.truthy(c.guard, `${c.id}: a fixed row names the test that stops it recurring`)
if (isPost(c)) {
t.is(c.gone, null, `${c.id}: a post keeps its sentence, so nothing is banned from the source`)
t.truthy(
c.guard ?? c.guardNote,
`${c.id}: a fixed row names the test that stops it recurring, or says why there is none`,
)
t.false(
Boolean(c.guard && c.guardNote),
`${c.id}: a row with a guard does not also explain its absence`,
)
if (isPost(c) || c.fixedInProduct) {
t.is(
c.gone,
null,
`${c.id}: the sentence was kept, so nothing about it is banned from the source`,
)
} else {
t.truthy(c.gone, `${c.id}: a fixed row names the words that must not come back`)
}
}
})

/**
* The account row is the only one where the words were right and the product
* was missing, and it is the only one with no guard. Both are deliberate and
* both are the kind of thing a later tidy-up quietly "fixes" by inventing a
* guard that does not run. Pinned so that has to be an argument.
*/
test('the account row is fixed in the product and admits it has no guard', (t) => {
const account = CORRECTIONS.find((c) => c.id === 'privacy-account-section')
t.truthy(account, 'the account row has been dropped from the registry')
if (!account) return
t.true(account.fixedInProduct === true)
t.is(account.gone, null)
t.is(account.guard, null)
t.truthy(account.guardNote)
t.is(daysLive(account), 1)
// The note names a file. Naming one that does not exist is exactly the shape
// of claim this page is about.
t.true(
existsSync(repoFile('scripts/assert-sync-config.mjs')),
'the note names a script that is not in the repository',
)
t.true((account.guardNote ?? '').includes('scripts/assert-sync-config.mjs'))
})

test('a corrected post carries its correction note', (t) => {
for (const c of CORRECTIONS) {
if (!isPost(c) || c.fixedOn === null) continue
Expand Down Expand Up @@ -126,6 +162,22 @@ test('the post is registered on the blog', (t) => {
t.truthy(BLOG_POSTS.find((p) => p.slug === 'what-we-got-wrong'))
})

/**
* The post went live with a typed claim about how many of its own rows were
* open, and it was false by seven hours: the thing it was about had been fixed
* that afternoon and the post did not publish until after midnight. The claim
* is now read off the registry at build time. That is the whole fix, so it is
* pinned rather than left as a habit somebody tidies away.
*/
test('the post never types its own live state', (t) => {
const source = readFileSync(repoFile('src/app/blog/what-we-got-wrong/page.tsx'), 'utf-8')
t.true(
source.includes('open.length'),
'the open-row count is no longer derived from the registry',
)
t.false(source.includes('as this goes up'), 'the publication-day tense is back in the post')
})

// Everything below walks the source. The registry quotes the false sentences and
// the post prints them, so those two are the only places they are allowed to be.
const EXEMPT = new Set(['src/config/corrections.ts', 'src/app/blog/what-we-got-wrong/page.tsx'])
Expand Down