Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .dumplingconf.example
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,19 @@ delete = [
# not_like / not_ilike — negation of like / ilike (prefer over regex look-around)
# regex / iregex — Rust regex crate (not PCRE); look-around / backrefs rejected at load
# not_regex / not_iregex — negation of regex / iregex
# lt / lte / gt / gte — numeric compare (values parsed as f64)
# lt / lte / gt / gte — numeric compare (values parsed as f64);
# set format = "datetime" for ISO/Postgres timestamps,
# or format = "date" for calendar dates (YYYY-MM-DD).
# Also valid on eq / neq. Unparseable cells fail closed.
# is_null / not_null — NULL check (no value needed)

# Example: keep ~18 months of orders by timestamp (plus NULL created):
# [row_filters."public.listing_order"]
# retain = [
# { column = "created", op = "gte", value = "2025-02-14", format = "datetime" },
# { column = "created", op = "is_null" },
# ]

# ---------------------------------------------------------------------------
# [[column_cases]] — conditional per-column strategy overrides
#
Expand Down
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,8 @@ Two code paths:

JSON path traversal is supported: `payload.profile.tier` (dot) and `payload__profile__tier` (Django double-underscore). Array elements are traversed by evaluating each item's fields, so list-of-dicts structures work naturally.

Comparison operators (`lt` / `lte` / `gt` / `gte`, and optionally `eq` / `neq`) accept `format = "datetime"` or `format = "date"` to compare ISO-8601 / Postgres timestamp text (or calendar dates) instead of numbers. Thresholds are validated at config load; unparseable cell values fail closed (do not match).

### Residual PII Scanning (`scan.rs`)

`ScanningWriter` wraps the output `Write` stream, intercepting bytes and passing them to `OutputScanner`. The scanner applies regex detectors line-by-line for:
Expand Down
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project follows [Semantic Versioning](https://semver.org/spec/v2.0.0.ht

## [Unreleased]

### Added

- **Datetime / date predicate comparisons**: `lt` / `lte` / `gt` / `gte` / `eq` / `neq` accept optional `format = "datetime"` (ISO-8601 / Postgres timestamp text as instants) or `format = "date"` (calendar dates). Thresholds are validated at config load; unparseable cell values fail closed ([#87](https://github.com/ababic/dumpling/issues/87)).

## [0.8.0] - 2026-08-14

### Added
Expand Down
16 changes: 15 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -328,9 +328,11 @@ Supported predicate operators:
| `not_like` / `not_ilike` | Negation of `like` / `ilike` (prefer these over negative lookahead) |
| `regex` / `iregex` | [Rust `regex`](https://docs.rs/regex/) crate (`iregex` is case-insensitive). **Not PCRE** — look-around (`(?!…)`, `(?=…)`, etc.), backreferences, and possessive quantifiers are unsupported and **rejected at config load** (fail closed). |
| `not_regex` / `not_iregex` | Negation of `regex` / `iregex` |
| `lt` / `lte` / `gt` / `gte` | Numeric compare (values parsed as numbers) |
| `lt` / `lte` / `gt` / `gte` | Numeric compare by default (values parsed as numbers). Set `format = "datetime"` to compare ISO-8601 / Postgres timestamp text as instants, or `format = "date"` for calendar dates (`YYYY-MM-DD`). Unparseable cell values fail closed (do not match). |
| `is_null` / `not_null` | No value needed |

`format = "datetime"` / `"date"` is also valid on `eq` / `neq` (instant or date equality). Thresholds are validated at config load; cell values that do not parse do not match.

Invalid or unsupported `regex` / `iregex` / `not_regex` / `not_iregex` patterns fail config load (and `dumpling lint-policy`) instead of silently matching nothing.

For “scrub unless staff domain” policies, prefer a default scrub plus a `keep` case (see Conditional cases cookbook above), or scrub via `not_ilike` / `not_iregex` cases without a default rule:
Expand Down Expand Up @@ -376,6 +378,18 @@ delete = [
]
```

Retain a recent window on timestamp columns without year-prefix `like` workarounds:

```toml
[row_filters."public.listing_order"]
retain = [
{ column = "created", op = "gte", value = "2025-02-14", format = "datetime" },
{ column = "created", op = "is_null" },
]
```

`format = "datetime"` accepts common `pg_dump` shapes (`2025-03-14 12:34:56.789+00`, `T` separators, `Z`, short offsets). Date-only thresholds are treated as midnight UTC. Use `format = "date"` to compare calendar dates only.

Row filtering works for both `INSERT ... VALUES (...)` and `COPY ... FROM stdin` rows.

---
Expand Down
4 changes: 3 additions & 1 deletion docs/src/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -379,9 +379,11 @@ salt = "${file:/run/secrets/dumpling_hmac_key}"
| `not_like` / `not_ilike` | Negation of `like` / `ilike` (prefer these over negative lookahead) |
| `regex` / `iregex` | [Rust `regex`](https://docs.rs/regex/) crate (`iregex` is case-insensitive). **Not PCRE** — look-around, backreferences, and possessive quantifiers are unsupported and **rejected at config load** (fail closed). |
| `not_regex` / `not_iregex` | Negation of `regex` / `iregex` |
| `lt` / `lte` / `gt` / `gte` | Numeric compare (values parsed as numbers) |
| `lt` / `lte` / `gt` / `gte` | Numeric compare by default. With `format = "datetime"`, compare ISO-8601 / Postgres timestamp text as instants; with `format = "date"`, compare calendar dates. Unparseable cells fail closed. |
| `is_null` / `not_null` | No value needed |

`format = "datetime"` / `"date"` also applies to `eq` / `neq`. Threshold strings are validated at config load.

Invalid or unsupported regex patterns fail config load and surface as `invalid-regex-predicate` in `dumpling lint-policy`. For “scrub unless allowlisted domain” policies, prefer a default scrub plus a `keep` case, or positive `not_ilike` / `not_iregex` cases — see the README *Conditional per-column cases* and *Row filtering* sections.

Nested JSON targeting is supported in predicate `column` values via either:
Expand Down
Loading
Loading