geoqa-study-harness

Running a user study on a geo-analytical question answering system, and refusing to export the data until it is safe to publish.

Written by M. Daud Tasleem in August 2026. It is the evaluation half of a pair: typed-workflow-layer synthesises and ranks candidate workflows, and this measures whether the answers they produce are any good to the people who would use them.

Python 3.10, no dependencies, including the web server. A study that needs a pip install and a reverse proxy before anyone can sit down at it does not happen.

python3 demo.py
python3 -m unittest discover -s tests          # 199 tests, nothing to install
python3 -c "from harness.server import serve; ..."   # see "Running a session" below

The problem

Three things go wrong around a user study, and all of them are quiet.

The system gets scored against 100 percent. A geo-analytical question has more than one defensible answer, so participants disagree with each other. Reporting that a system agreed with participants 60 percent of the time says nothing until you know how often they agreed with each other. In the demo study that figure is 50 percent. The system is not sitting 40 points short of perfect, it is sitting 10 points above the rate at which the humans agree, and that is a different finding requiring a different explanation.

The data is published with the names deleted and called anonymised. Deleting the name column is not anonymisation. In a study of twelve people, PhD candidate, 3 years, Delft University of Technology identifies one person to anyone who knows the group, and no name was ever stored. This package measures that formally, and stops the export when the answer is bad.

The workflow is scored by string equality. A workflow is a nested expression, so it is a tree, and comparing trees as strings calls every near miss a total miss. On a project where the workflow is the object of study, the difference between a workflow with one redundant step and a workflow that computes something else is the finding, and a rate of 0 out of 3 hides it.

What it does

• Runs the session. A small HTTP server on the standard library serves the tasks and collects the answers, writing to disk after every response rather than at shutdown.
• Enforces consent on the data model rather than in the interface, so a script that imports the module is refused exactly as the browser is.
• Issues each participant a token at enrolment and requires it afterwards, so a participant code on its own cannot consent or answer as somebody else from another tab.
• Keeps answers append-only. A participant who revises produces a second record and the first stays, because a study file edited in place cannot be audited afterwards.
• Rotates task order per participant, derived from the participant id so a re-run reproduces it.
• Separates three numbers that are usually run together: accuracy against a reference, agreement among participants, and agreement between participants and the system.
• Scores the workflow separately from the answer, because reaching the right number by a route nobody would defend is a finding rather than a success.
• Measures k-anonymity over the quasi-identifiers, generalises them, measures again, and refuses to write anything at all if the result is still unsafe.
• Writes the deposit README from the data, so the description of what was removed cannot drift from what was actually removed.

The refusal

export() runs the anonymisation check and raises ExportRefused when it fails. It takes no force, no skip_check, no override. There is a test asserting those parameters do not exist and a CI job that fails if one is added, because an override gets used under deadline pressure by the person least placed to judge the risk, who is the person about to publish.

The names of the quasi-identifiers are checked against the participants before anything is measured, and an unknown one raises. A name that did not resolve used to fall back to an empty string, which is the same value for everybody, and a column that never differs splits no groups: it raised the measured k instead of lowering it. Three mutually unique people scored against rol, yrs and org came back as k = 3 and safe to export.

The demo's twelve-participant study is refused, and the refusal is correct. Six generalised roles by four experience bands by four sectors is 96 combinations; twelve people cannot fill that space three deep. Recruiting more is not the fix at any realistic study size.

The fix is suppression, and the package measures what it costs rather than leaving it to judgement:

exported attributes	k
role, years_experience, organisation	1	unsafe
role, years_experience	1	unsafe
role, organisation	1	unsafe
years_experience, organisation	1	unsafe
role	1	unsafe
years_experience	3	safe
organisation	1	unsafe

At n = 12, exactly one quasi-identifier can be published. That is the trade: the experience band is kept, role and organisation are dropped, and any analysis that needed them cannot be done on the published data. Knowing that before recruiting is worth more than discovering it after.

Role looks like the coarse one and it is still only k = 1, because the single postdoctoral researcher is a category of one once postdocs are kept distinct from PhD candidates. Collapsing the two would raise role to k = 5 and publish a distinction that is not real, which is the kind of quiet inflation this check exists to refuse.

Comparing workflows as trees

Three workflows, all scored identically against the reference by string equality, all of them wrong, none of them wrong in the same way:

produced	verdict	similarity
Proportion(Coverage(Constrain(Clip(noise, district)), district))	reference plus extra steps	0.71
Proportion(Coverage(noise, district))	missing steps from the reference	0.80
Constrain(Coverage(Proportion(noise), district))	same operations, different composition	0.60
ZonalStatistics(noise, Coverage(Constrain(noise), district))	different workflow	0.67
Buffer(Overlay(stations, district))	different workflow	0.20

The verdict is the useful output and the similarity only orders near misses within one. Two figures are reported side by side: the exact-match rate, which is what string comparison alone would have said, and the rate at which the system applied exactly the reference operations whatever order it put them in. The gap between them is the point.

The distance is Zhang and Shasha's algorithm for ordered labelled trees. It is asserted against nine distances worked out by hand, against identity, symmetry and the triangle inequality, and against the published zss package over all 378 pairs of 28 randomly generated trees, where it matches to nine decimal places. That cross-check skips when zss is absent so the suite still runs with nothing installed, and a CI job fails if it skips in the job that exists to run it.

same operations, different composition is deliberately its own verdict rather than a pass or a fail. Whether reordering changes the answer depends on the operations and the data, and this module does not know that. It says the operations are the same and the composition is not, which is what it can honestly say.

How many participants, and what you may publish about them

Two questions constrain each other and are normally asked months apart:

• before recruiting, how many people are needed to detect the difference worth detecting
• after collecting, how much of it can be published

Asked together they give one number. Assume before recruiting that the participants will agree with each other about 40 percent of the time. To detect a 20 point difference from that baseline, with four closed tasks each, alpha 0.05 and 80 percent power, the answer is twelve participants. Twelve is also the size at which the k-anonymity check above allows exactly one attribute to be published. A study designed for the statistics is not automatically a study you are allowed to publish. (The pilot then measures the ceiling a little higher, at 50 percent, which is the ordinary gap between what you assume before recruiting and what you find after, and does not change the twelve.)

What a smaller study buys, measured rather than guessed:

participants	smallest difference detectable
5	30%
12	20%
30	13%
100	7%

Where even the largest difference the baseline leaves room for would still need more observations than the study has, the function returns nothing rather than the edge of its own search. A ceiling of 95 percent leaves five points of room and five points needs 86 observations, so at n = 5 there is no answer, and a number in that position gets read as one.

Post-hoc power is not implemented, and there is a test asserting the function does not exist. Computing the power a finished study had to detect the effect it happened to observe is a monotone function of the p-value and adds nothing. The minimum detectable effect at the size actually achieved answers the useful version of that question.

The inverse normal is Acklam's approximation with one Halley refinement, since the standard library has none and the package installs with nothing. It is checked against nine published critical values and round-trips through math.erfc to within 1e-12.

Running a session

from harness.server import serve
from harness.session import Study
from harness.tasks import TaskSet

tasks = TaskSet.load("tasks.json")
study = Study("Noise exposure study", task_set=tasks)
serve(study, port=8765, path="study.jsonl")

Participants open http://127.0.0.1:8765, enter an id, read the consent text, and answer. The file is written after every response.

Enrolling issues that participant a token, which the page keeps and presents on everything after it. Consent, answers and the task list are refused without it, so knowing that the codes run p01 to p12 is not enough to consent or answer as somebody else from another tab. Requests that arrive from another origin, or with anything other than Content-Type: application/json, are refused before they reach a route.

Afterwards:

from harness import evaluate, export

print(evaluate.report(evaluate.summarise(study, tasks)))
export.export(study, tasks, "deposit/", salt="a-secret-not-in-the-bundle")

The salt is the study's secret and must not be published with the data. Without it the pseudonyms cannot be reversed by hashing a guessed list of ids, which is otherwise easy when the ids look like p01 to p12.

Design decisions

The consent rule lives on the Study object. The web interface asks for consent, but the interface is not the only way into the data. A rule enforced only at the edge is not enforced.

Declining consent reports, it does not block. Somebody enrolling and then declining is the consent process working. An earlier version treated it as a blocking finding, which would refuse the whole export over a normal outcome and train people to delete the record to get past the check. They are now excluded from the bundle and named in the report.

Open-text and rating tasks are kept out of the headline agreement figures. Two analysts who write "threshold then coverage then proportion" and "constrain to above 70 then coverage per district" have described the same workflow, and exact string match calls that a disagreement; two who rate a map 4 and 5 have agreed closely, and exact match calls that a disagreement too. Scoring every task by exact match put the demo ceiling at 32 percent; the closed tasks alone put it at 50, and that difference is purely wording and degree. Both kinds are still scored per task and reported separately, and acceptance leaves them out for the same reason, so the ceiling and the acceptance rate are read over the same tasks.

A system output is compared once, not once per participant. The system produces one workflow per task and it is stored on every participant's response to that task. Counting per response reported five comparisons where one had been made, and turned a single mistake into five.

Nothing is written when the export is refused. Not a partial bundle, not an empty directory of placeholders. There is a test for that, because a directory containing three of five expected files is the kind of thing somebody zips and sends.

The deposit README is generated from the data. k, the attribute list and the counts come from the check that just ran. A description typed by hand afterwards is the usual way the stated anonymisation stops matching the actual anonymisation.

Redaction is offered, not applied. redact() returns a cleaned version of a free-text answer for a human to approve. Applying it automatically changes what a participant said, and a person should decide that rather than a regular expression.

Limits

Read this before the code.

• No study has been run. The demo's twelve participants and their answers are written by hand. They reproduce the kind of disagreement these questions produce and they exercise every path, but every number in the demo is a demonstration of the machinery, not a finding about a system or about people.
• k-anonymity is a floor, not a guarantee. It does not defend against somebody who already knows a specific person took part, and it says nothing about what a free-text answer reveals: a participant describing their own project may identify themselves whatever the table says. The check reports this in the bundle rather than hiding it behind the word "anonymised".
• No l-diversity or t-closeness. k-anonymity alone is vulnerable to a group that shares a sensitive value, and neither is implemented.
• The server is not a production web server. Single threaded, no TLS, no rate limiting. It binds to localhost by default and should sit behind something that does all of that before it is exposed beyond a trusted network.
• The participant token is a bearer capability with nothing behind it. Whoever holds it is that participant: there is no expiry, no rotation and no way to revoke one short of editing the study file. Without TLS it also travels in clear, so anyone who can read the traffic can replay it. It stops a page in another tab writing to the study, which is the threat a lab session actually has; it is not authentication.
• The study file holds those tokens. It already holds the raw attributes and the answers, so it was never a file to leave lying about, but it is now a file with live credentials in it for as long as the study is running. The deposit bundle does not contain them.
• The origin check only binds browsers. A request with no Origin header is treated as not coming from a page, because that is what curl and every script look like, and any attacker can look like one too. The token is what actually protects the routes; the origin check and the application/json requirement close the case where the attacker is a page rather than a program.
• Free-text scanning is regular expressions, so it catches an email address and a coordinate pair and will not catch "the project I run for the ministry". Phone-shaped strings are flagged for review rather than blocked, because the pattern also matches long identifiers and a check that cries wolf gets ignored.
• Agreement is pairwise percent, not a chance-corrected coefficient. For a chance correction on labelled data see geo-annotation-kit, which implements Krippendorff's alpha and cross-checks it against the reference implementation. This package deliberately does not duplicate that.
• The power calculation treats one participant's answers as independent. They are not: somebody who misreads a task misreads several. Dividing the observation count by the tasks per participant is therefore optimistic, and the report says so and calls the figure a floor. A mixed model on real pilot data is the honest version.
• Structural workflow comparison is not semantic equivalence. Whether a redundant Clip changes the answer depends on the data. The module reports that the step is there and does not claim to know what it did.

What I would build next, in order

1. Run a real session. Everything here is machinery waiting for participants.
2. Chance-corrected agreement on the closed tasks, by importing the coefficients from geo-annotation-kit rather than reimplementing them.
3. Parsing an open-text answer into a workflow tree, so a described workflow can go through the same structural comparison the system's output does. That is what would let open tasks back into the headline figures rather than being excluded from them.
4. l-diversity alongside k, so a group of five who all gave the same sensitive answer is caught rather than passed.
5. Replacing the independence assumption in the power calculation with a mixed model fitted to pilot data, which is the difference between a floor and an estimate.

Files

file	what it holds
harness/tasks.py	tasks, answer validation, counterbalanced order
harness/session.py	participants, consent, append-only responses, JSON Lines
harness/evaluate.py	accuracy, human ceiling, acceptance, workflow scoring
harness/anonymise.py	quasi-identifiers, k-anonymity, generalisation, suppression
harness/export.py	the deposit bundle, and the refusal
harness/server.py	the study server, standard library only
harness/static/index.html	the interface a participant sits in front of
demo.py	a study end to end, including two exports that are refused
harness/workflow.py	workflows compared as trees, Zhang and Shasha edit distance
harness/power.py	how many participants, and what is publishable at that size
tests/	199 unittest cases, including the server over a real socket

Licence

MIT. See LICENCE.