Redaction Coverage Lab

100% processed is not 100% protected.

This interactive lab shows how a telemetry redactor can report success while synthetic sensitive values still escape through log bodies, span events, URL query strings, baggage, resource attributes, and encoded payloads.

The control plane is an executable Open Policy Agent policy written in Rego. A dependency-free Node.js service generates deterministic telemetry surfaces, sends them to the real opa eval CLI, compares four redaction strategies, renders the result, and exposes correlated OTLP-shaped metrics, logs, and traces.

Important

The lab uses fake canaries only: example.test email addresses, RFC 5737 documentation IPs, and tok_demo_* tokens. It sends no external traffic and exports no telemetry.

The failure

Redaction coverage has at least three independent dimensions:

1. signal coverage — logs, traces, metrics, resources, events, and baggage do not all pass through the same processor;
2. container coverage — an attribute allowlist does not inspect an opaque body or URL;
3. representation coverage — raw regexes do not necessarily recognize percent-encoded or base64-wrapped values.

The default scenario creates 36 synthetic exposures across 12 records. Every strategy reports all records processed, but their actual redaction coverage differs:

Strategy	Caught	Escaped	Actual coverage
Key allowlist	7	29	19.4%
Log-only scrubber	5	31	13.9%
Regex everywhere	18	18	50%
OPA policy + audit	36	0	100%

OpenTelemetry recommends minimizing collected sensitive data and applying controls such as the Collector redaction and transform processors. It also calls out the need to consider attributes, span events, log bodies, and other telemetry fields rather than assuming one control covers every representation. See Handling sensitive data.

Why telemetry.sh helps

A redactor is an operational control. Its coverage needs telemetry of its own:

• telemetry.redaction.coverage compares actual findings against the known synthetic canary population;
• telemetry.sensitive.leak.count makes silent misses alertable;
• error logs preserve strategy, coverage, and policy identity without containing a real secret;
• policy-evaluation spans correlate rollout state with findings and export status;
• resource attributes mark the dataset as synthetic and identify the service.

The useful dashboard is not “records processed.” It is processed vs. inspected vs. contained, broken down by strategy, signal, container, encoding, and collector rollout.

Run locally

Requirements:

• OPA 1.17.0 or newer
• GNU Make
• Node.js 20 or newer

npm ci
make check
make run

Open http://127.0.0.1:3000.

If the OPA binary is not named opa, pass its path explicitly:

make check OPA=/path/to/opa
make run OPA=/path/to/opa

Useful endpoints:

GET /api/simulate
GET /api/telemetry
GET /healthz

Every UI control is also a query parameter:

/api/simulate?records=20&encodedPercent=75&rolloutPercent=60&payloadShape=trace-heavy&secretSet=token

Run with Docker

docker compose up --build

Then open http://127.0.0.1:3000. The image copies the pinned OPA 1.17.0 static binary from the official OPA image into a small Node.js runtime.

Executable policy

policy/redaction.rego is the policy used by every API request. The strategies change which telemetry surfaces and representations become policy candidates:

candidate(surface) if {
	input.strategy == "key_only"
	regex.match(`(?i)(email|token|authorization|client\.ip)`, surface.path)
}

candidate(surface) if {
	input.strategy == "policy_audit"
	surface.protected
}

inspection_value(surface) := surface.canonical if {
	input.strategy == "policy_audit"
}

The service invokes OPA directly:

opa eval \
  --stdin-input \
  --data policy/redaction.rego \
  --format json \
  data.telemetry.redaction.decision

The Rego suite is checked and tested in CI using OPA's policy testing workflow.

Architecture

browser controls
      │
      ▼
Node.js scenario generator
      │  normalized telemetry surfaces
      ▼
OPA / Rego decision ── findings ──▶ strategy comparison
      │                                  │
      └──────── executable evidence ◀────┘
                                         │
                                         ▼
                           OTLP-shaped metrics, logs, traces

The browser does not reimplement the detection rules or fabricate policy results. Changing a control triggers fresh server-side OPA evaluations.

What to try

• Set Encoded payloads to 100%. Raw regex coverage falls to zero while the normalized policy still finds every canary.
• Set Policy rollout to 50%. The recommended strategy now exposes the half-deployed control as a coverage gap.
• Switch to Trace-heavy. A log-only scrubber looks even healthier while most sensitive surfaces bypass it.
• Select one canary type to see how path-based rules interact with values embedded under generic keys.

Safety

This repository is an educational simulation, not a production redaction library. Do not use the example regexes as a complete sensitive-data classifier. In production, minimize collection, prefer allowlists, tokenize where appropriate, protect data in transit and at rest, audit the full pipeline, and test controls with organization-approved synthetic canaries.

License

MIT