Set the rule for an experiment that touches real personal data

CONTRIBUTING.md

@@ -11,9 +11,10 @@ here refuses a violation of any of them: what may never be committed, in
 [0006](docs/decisions/0006-everything-here-is-public.md) and again below; what
 happens when an experiment finds a flaw in shipped software, in
 [0010](docs/decisions/0010-a-flaw-in-shipped-software.md); and what an
-experiment may do with real data, in [docs/privacy.md](docs/privacy.md), which
-is the one this board has not finished writing and says so, with #35 holding
-what is left. Why the door is open at that price is
+experiment may do with real data, in [docs/privacy.md](docs/privacy.md), whose
+conditions are
+[0025](docs/decisions/0025-real-data-in-an-experiment.md). Why the door is open
+at that price is
 [0024](docs/decisions/0024-who-may-run-an-experiment-here.md).
 
 ## Before you push
@@ -140,9 +141,11 @@ than about what is in the current checkout. The reasoning is in
   running server, whoever runs it.
 - No file whose licence forbids it being here.
 
-Where an experiment needs real data to answer its question, the data stays on
-the host it is already on. It does not enter the tree as a fixture, as a sample,
-as an attachment, redacted, or in a screenshot. Redaction is named because it is
+Where an experiment needs real data to answer its question, the data is the
+runner's own and the measurement is named in the record before the work starts,
+which is [0025](docs/decisions/0025-real-data-in-an-experiment.md), and the data
+stays on the host it is already on. It does not enter the tree as a fixture, as
+a sample, as an attachment, redacted, or in a screenshot. Redaction is named because it is
 the failure that feels safe: a partially masked identifier is still an
 identifier, and a screenshot of a list is a copy of the list. What may be
 written down is the measurement and the command that produced it.

docs/decisions/0008-the-experiment-record.md

@@ -1,9 +1,10 @@
 # 0008. What an experiment record looks like
 
 Superseded by 0014, which fixes what a slug may be, by 0015, which adds the
-field naming what an experiment needs beyond the runner, and by 0016, which adds
-the field naming the commit a measurement was produced at. Everything below
-stands as it was written.
+field naming what an experiment needs beyond the runner, by 0016, which adds
+the field naming the commit a measurement was produced at, and by 0025, which
+adds the field declaring real personal data. Everything below stands as it was
+written.
 
 ## What was decided
 

docs/decisions/0025-real-data-in-an-experiment.md

@@ -0,0 +1,142 @@
+# 0025. Real personal data in an experiment
+
+## What was decided
+
+An experiment here may touch real personal data, under two conditions, and it
+declares that it does before the work starts.
+
+The data belongs to the person running the experiment. Their own library, their
+own accounts, their own logs, on their own host. Not a colleague's, not an
+employer's, not a user's, and not a copy of any of those that somebody was
+allowed to hold for a different purpose.
+
+The measurements are agreed in the record before the work starts. What is
+measured is written into the record in the commit that writes the question, so
+the reading that gets published was named while the answer was still unknown.
+A measurement thought of afterwards is a new question and takes a new record.
+
+The rule already written in [../privacy.md](../privacy.md) stands on top of
+both, unchanged and not restated here: the data never enters the tree, in any
+form, and what may be written down is the measurement.
+
+This record supersedes 0008. Everything that record fixes about an experiment
+record stands exactly as it is written there, and this adds one field to the
+header.
+
+`Real-Data` names what category of data, on whose host, and what will be
+written down about it, or the single word `none`.
+
+    Real-Data: my own media library, on my own machine, and what gets written down is the scan time and the item count
+    Real-Data: none
+
+The field exists because the two conditions above are conditions on something
+nobody can see afterwards. An answer that says a library of a hundred thousand
+items took four minutes to scan reads identically whether the library was the
+author's own or an employer's, and whether the timing was the agreed
+measurement or the one that came out best. The declaration is what makes the
+difference visible at the moment it is still cheap to ask about, which is the
+review of the commit that writes the question.
+
+The field is optional, which record 0013 fixes for every field added after it,
+so a record written before this one stays legal and an absent field is never
+refused. What that costs is stated at the bottom of this section rather than
+left for somebody to discover.
+
+`none` is what a record says when the question is answered against generated or
+synthetic data, and it is the answer for almost every experiment. Where
+synthetic data can answer the question it does, and a record that needed real
+data says why in its method. That sentence makes the easy path the default
+without banning the hard one, and it is a sentence for a reader rather than a
+rule for a machine.
+
+One refusal follows, and it reads a declaration rather than an absence. A
+record that declares `Real-Data` and writes nothing after the colon is refused
+under `record-real-data-declaration-is-empty`. It names no data, no host and
+nothing that will be written down, and it does not say `none`, so it carries
+the appearance of a declaration and none of the content of one. That is the
+mistake the shape invites: the field is typed while the question is being
+written, the value is left for the moment the work starts, and the moment the
+work starts is when nobody reopens the header.
+
+WHAT NOTHING REFUSES, and this is most of the rule. An experiment that touches
+real personal data and declares nothing at all is refused by nothing, because
+record 0013 makes an absent field legal and the runner cannot tell a record
+that omitted the field from one written before the field existed. Whose data it
+was, whether the measurement was agreed in advance, and whether the record
+carries the data rather than a measurement of it are all outside every reading
+of a checkout, and the last of those is stated in `docs/privacy.md` as having
+no mechanism and keeps not having one. What stands behind this rule is the
+template, the person writing the record, and whoever reads the change.
+
+## What it applies to
+
+Every experiment on this board that touches real personal data, from the commit
+this record lands on.
+
+The experiment record format, which gains one optional field. Records already
+on the default branch are not edited, not migrated and not marked, which record
+0013 fixes and which this record does not vary.
+
+It applies to `docs/privacy.md`, which said this question was open and now
+carries the answer, and to `docs/experiment-template.md`, which carries the
+field for records written from here on.
+
+It does not apply to what may never be committed, which is record 0006 and is
+unchanged by this. A declaration permits an experiment to read real data on the
+host it is already on; nothing here permits a byte of it into the tree.
+
+It does not apply to data that is not personal. A generated library, a public
+corpus and a synthetic account set are outside this record, and an experiment
+using them says `none` or says nothing.
+
+## What else was considered
+
+Refusing real personal data on this board outright.
+
+Allowing it with no declaration at all, leaving the host rule in
+`docs/privacy.md` to carry the whole of it.
+
+Allowing data belonging to somebody else where that person consented.
+
+Requiring the field of every record rather than making it optional.
+
+A `## Real data` section carrying the three parts as named lines, in the shape
+record 0005 gives the promotion section, rather than one header field.
+
+## What each rejected option would have cost
+
+Refusing it outright costs the board the questions it is most useful for. How a
+library of a hundred thousand items behaves, how a real account set looks after
+a migration and what a real log actually contains are answerable against real
+data and against nothing else, and a board that cannot ask them is less useful
+than it was opened to be. The ban would also not hold: the questions arrive
+anyway, and the first one would be answered somewhere with no record of what
+was touched, which is worse than the thing the ban was for.
+
+No declaration costs the reviewer the only moment the question is cheap. The
+host rule says what happens to real data if an experiment uses it, and says
+nothing about whether this experiment did. A reviewer reading a finished answer
+cannot tell, and asking afterwards asks somebody to remember what they did
+rather than to confirm what they wrote.
+
+Consent from a third party costs a promise this board cannot keep. Consent is a
+thing with a scope, a date and a way of being withdrawn, none of which this
+tree can hold or check, and a record asserting that somebody agreed is an
+assertion nobody here can test. Restricting the rule to the runner's own data
+gives up real questions and gives up a class of failure this board would have
+no way to detect or repair.
+
+Requiring the field turns every record written before today red on the day this
+lands, and record 0013 exists to refuse exactly that trade. Its own words are
+that the repair available under a red board is either editing permanent records
+or weakening the check that just landed, and both destroy something.
+
+The section costs a second syntax for a job the header already does, and it
+would put the declaration below the fold, where the header is what a reader
+scans and what every other field of this format lives in. The promotion section
+earns its shape because it carries four values that each need their own line
+and a commit range that has to be read; this carries one sentence. What the
+section would buy is a refusal that could ask for the third part by name, and
+that is not the residual worth paying for: whether words name a real category,
+a real host and a real measurement is a judgement no reading of a checkout
+makes either way.

docs/experiment-template.md

@@ -60,11 +60,31 @@ says nothing and appears in the listing as ordinary unanswered work, which is
 the misreport the field is written against and which only the person writing the
 record can prevent.
 
+Add `Real-Data` where the experiment reads real personal data, naming what
+category of data, on whose host, and what will be written down about it, or
+write `none`. It is not in the header above for the reason `Measurement-Commit`
+is not: a template that ships a field filled in teaches every new record to
+declare a value it does not have, and `none` shipped filled in would be a claim
+made by whoever copied the file rather than by whoever ran the experiment. It
+goes in the commit that writes the question, because the point of it is that the
+measurement was named while the answer was still unknown. Whose data may be read
+and what has to be agreed in advance are
+`docs/decisions/0025-real-data-in-an-experiment.md`, and `docs/privacy.md` is
+where the same rule is written for whoever is running the thing.
+
+What that refusal does not reach is the case the field exists for. A record
+declaring the field with nothing after the colon is refused, and an experiment
+that reads real data and declares nothing at all is not, since an absent field is
+never a refusal. Nothing here reads whose machine the data was on, whether the
+measurement was the agreed one, or whether the record carries the data rather
+than a measurement of it.
+
 The format is `docs/decisions/0008-the-experiment-record.md`, as added to by
 `docs/decisions/0015-an-experiment-declares-the-harness-it-needs.md`, by
 `docs/decisions/0016-an-answer-names-the-commit-it-measured.md`, by
-`docs/decisions/0019-code-under-another-licence.md` and by
-`docs/decisions/0022-how-long-a-held-back-record-waits.md`. This file is a
+`docs/decisions/0019-code-under-another-licence.md`, by
+`docs/decisions/0022-how-long-a-held-back-record-waits.md` and by
+`docs/decisions/0025-real-data-in-an-experiment.md`. This file is a
 convenience and those records are the authority.
 
 ## Question
@@ -78,8 +98,9 @@ record says "media transcoding" can never be shown to have missed its answer.
 What was done, in enough detail that somebody else could do it again or say why
 they cannot. The commands, the machine where it matters, and what was measured.
 
-What may never be committed alongside this record, and what happens where a
-question can only be answered against real data, is
-`docs/decisions/0006-everything-here-is-public.md`.
+What may never be committed alongside this record is
+`docs/decisions/0006-everything-here-is-public.md`. Whether a question may be
+answered against real data at all, and on what conditions, is
+`docs/decisions/0025-real-data-in-an-experiment.md`.
 
 ## Answer

docs/privacy.md

@@ -26,6 +26,41 @@ The list of what may never be committed is in
 [docs/decisions/0006-everything-here-is-public.md](decisions/0006-everything-here-is-public.md).
 This document is the operator-facing half of the same position.
 
+## Whether an experiment may use real data at all
+
+It may, under two conditions, and it says so before it starts.
+
+The data belongs to the person running the experiment. Their own library, their
+own accounts, their own logs, on their own host. Not a colleague's, not an
+employer's, not a user's, and not a copy of any of those that somebody was
+allowed to hold for a different purpose. Consent from somebody else is not a
+route this board offers, because consent has a scope, a date and a way of being
+withdrawn, and none of those is a thing a repository can hold or check.
+
+The measurements are agreed in the record before the work starts. What is
+measured goes into `EXPERIMENT.md` in the commit that writes the question, so
+the reading that gets published was named while the answer was still unknown. A
+measurement thought of afterwards is a new question and takes a new record.
+
+The declaration is the `Real-Data` field of the record header. It names what
+category of data, on whose host, and what will be written down about it, or the
+single word `none`:
+
+    Real-Data: my own media library, on my own machine, and what gets written down is the scan time and the item count
+    Real-Data: none
+
+Where synthetic data can answer the question, it does. A record that needed real
+data says why in its method. That is a sentence for a reader rather than a rule
+for a machine, and it is written this way to make the easy path the default
+without banning the hard one.
+
+The position above stands on top of all of it, unchanged. Real data never enters
+the tree, in any form, and what may be written down is the measurement.
+
+The reasoning is
+[docs/decisions/0025-real-data-in-an-experiment.md](decisions/0025-real-data-in-an-experiment.md),
+which is also where the field is fixed.
+
 ## What deliberately means
 
 Deliberately is the load-bearing word above, so it is defined here rather than
@@ -61,10 +96,9 @@ It says nothing about the operator's own machine. What else is running on the
 host, what the host's own logs retain, and what a backup of it carries are all
 outside anything written here.
 
-It does not settle whether an experiment may use real data at all. This document
-says what happens to real data if an experiment uses it. Whether that is
-permitted in the first place is a wider question, open on issue #46, and issue
-#35 is where the answer gets written down.
+It does not reach the case where somebody ignores the section above. What is
+written there is a rule about conduct on somebody else's machine, and the only
+thing this repository ever sees is the record that came back.
 
 It is not a legal characterisation. This is a statement of what the tooling does
 and does not do, not advice about what any particular law requires of an
@@ -106,3 +140,14 @@ own machine from a number they made up, and no check reads what an experiment
 did before it wrote its record. Review is where a record carrying something it
 should not is caught, and this is stated plainly here rather than left for a
 reader to assume otherwise.
+
+One line of the section above is read by the runner, and it is much less than
+that section. A record that declares `Real-Data` and writes nothing after the
+colon is refused, because it carries the appearance of a declaration and none of
+its content. That is the whole of it. A record that declares nothing at all is
+not refused and cannot be, since an absent field is legal for every field added
+after
+[docs/decisions/0013-how-the-record-format-changes.md](decisions/0013-how-the-record-format-changes.md);
+whose data it was and whether the measurement was agreed in advance are claims
+nothing here can test; and a green run says a declaration was written rather
+than that it is true.

internal/check/check.go

@@ -523,6 +523,7 @@ func walkExperiments(fsys fs.FS, root string, res *Result) error {
 		res.Refusals = append(res.Refusals, refuseHeaderDates(record, data)...)
 		res.Refusals = append(res.Refusals, refuseMeasurementCommit(record, data)...)
 		res.Refusals = append(res.Refusals, refuseHeldBack(record, data)...)
+		res.Refusals = append(res.Refusals, refuseRealData(record, data)...)
 		res.Refusals = append(res.Refusals, refuseDates(record, data, res.Now)...)
 		res.Refusals = append(res.Refusals, refusePromotion(record, data)...)
 		// The two rules here that read the directory as well as the record,