fix(py): hold the batch slot for unreadable inputs

.claude/rules/testing.md

@@ -68,6 +68,17 @@ longer the file being edited. Before believing any perturbation result,
 confirm the subject still contains the change: `rg -c <new symbol>` on
 the file, or a `git diff --stat` that shows what you expect.
 
+**The result *parser* is the other half of the subject.** During #1238 a
+sweep drove three perturbations of one match arm and reported zero Rust
+failures for all three, while the Python leg of the same sweep reported
+the expected four, five and eight. Nothing was stale — the driver ran
+`cargo test -q`, which prints dots rather than per-test lines, so a
+parser scanning for `... FAILED` matched nothing and every mutation read
+as "no test noticed". A uniform zero across perturbations is the same
+tell as a uniform 34: it describes the harness, not the code. Cross-check
+any parsed count against the process exit status and treat a
+disagreement in *either* direction as a harness bug, not a result.
+
 After restoring, `git status` / `git diff --stat` must show exactly
 the edits you intend — nothing extra, nothing missing.
 

CHANGELOG.md

@@ -108,6 +108,36 @@ for historical reference.
 
 ### Fixed
 
+- **(breaking)** The Python bindings' `analyze_batch` / `analyze_paths` dropped a
+  result slot for any file the read gate declines to parse — three
+  bytes or fewer, a UTF-16 BOM, or a leading window that is not valid
+  UTF-8 — even under `skip_generated=False`, which both entry points
+  documented as guaranteeing one element per input. That gate is
+  unconditional, so a batch containing one such file returned a shorter
+  list and the `zip(inputs, results)` pattern the docstrings endorse
+  silently attributed every later result to the wrong path: a
+  data-corruption class defect with no error and no `AnalysisFailure` to
+  observe. With `skip_generated=False` such a file now holds its
+  position as a `None` element — the same value single-file `analyze`
+  returns for it, and one `to_sarif` already skips — so the documented
+  `zip` is finally safe (#1238). The `skip_generated=True` default is
+  unchanged: a skipped file, generated or unreadable, still yields no
+  element. The typed surface widens to match: `analyze_batch` and
+  `analyze_paths` are now annotated
+  `list[FuncSpaceDict | AnalysisFailure | None]` in `_native.pyi`, so a
+  `mypy --strict` consumer indexing a slot without a `None` check is
+  told to add one rather than discovering it at runtime. An **untyped**
+  `skip_generated=False` consumer sees the change at runtime instead: a
+  loop that indexed every slot used to run to completion on such a batch
+  (silently mis-paired) and now raises `TypeError: 'NoneType' object is
+  not subscriptable` at the placeholder, and `len(results)` grows.
+  That is the intended direction — loud and local beats silent and
+  downstream — but it is a behaviour break, and `analyze_paths` shares
+  it, so a directory walk under `skip_generated=False` gains one element
+  per discovered file the gate declined. Widening a return union is not
+  an additive change under [STABILITY.md](./STABILITY.md); the exception
+  and its reasoning are recorded there under *Python bindings →
+  Typing*.
 - Ruby regex literals and Perl bare match literals fabricated division
   operators: the delimiter tokens under the literal wrapper were
   classified through the generic `/` operator arm, so `x = /abc/`

STABILITY.md

@@ -1035,6 +1035,32 @@ the existing values are frozen.
 
 ### Error mapping
 
+A batch slot is not always a result or a failure. `analyze_batch` /
+`analyze_paths` emit a `None` element, under `skip_generated=False`
+only, for a file the shared read gate declines to parse — three bytes
+or fewer, a UTF-16 BOM, a leading window that is not valid UTF-8, or
+(rarely) a file that shrank between the size probe and the read.
+That gate is unconditional, so before #1238 those files produced no
+element at all and the documented `zip(inputs, results)` mis-paired
+every later entry; the placeholder is what makes `skip_generated=False`
+genuinely one-element-per-input. `None` is the same value single-file
+`analyze` returns for those files, and `to_sarif` skips it. The
+`skip_generated=True` default is unchanged: a skipped file, generated
+or unreadable, still yields no element.
+
+The placeholder is a **behaviour break for existing
+`skip_generated=False` callers**, in both directions. A caller that
+looped over the results untyped — `for r in results: r["metrics"]` —
+previously ran to completion on a batch containing such a file (having
+silently mis-paired every later entry) and now raises `TypeError:
+'NoneType' object is not subscriptable` at the placeholder. A caller
+that counted `len(results)` sees a larger number. Both are the intended
+direction: the failure is now loud and local instead of silent and
+downstream, and the count is now the truth. `analyze_paths` shares the
+change, so a directory walk under `skip_generated=False` gains one
+element per discovered file the gate declined — on a tree with binary
+assets that can be a substantial fraction of the list.
+
 Per-file failures from `analyze_batch` / `analyze_paths` are
 **returned**, not raised, as `AnalysisFailure` values (not `Exception`
 subclasses: the class is deliberately not raisable; it was renamed from
@@ -1066,7 +1092,9 @@ as the library: additive in minor bumps, breaking only at a major.
 The analysis-result wire shape is also expressed as exported
 `TypedDict`s (#623): `analyze` / `analyze_source` return
 `FuncSpaceDict | None` / `FuncSpaceDict`, `analyze_batch` /
-`analyze_paths` return `list[FuncSpaceDict | AnalysisFailure]`, and the
+`analyze_paths` return `list[FuncSpaceDict | AnalysisFailure | None]`
+(the `None` slot is the read-gate placeholder described under *Error
+mapping* above, #1238), and the
 nested metric blocks (`CodeMetricsDict`, `LocDict`, `HalsteadDict`,
 `VcsDict`, …) are re-exported from the package. Like the enums, these are
 **generated** from the `big_code_analysis::wire` structs (`src/wire.rs`,
@@ -1093,6 +1121,17 @@ structs moved into `big_code_analysis::wire` and the jit shapes are
 mirrored from `src/vcs/jit.rs`, so the former `dict[str, Any]` returns are
 gone. The same additive / major-only shape contract applies.
 
+One documented exception to that contract, on the 2.x line: #1238
+widened the `analyze_batch` / `analyze_paths` return element to
+`FuncSpaceDict | AnalysisFailure | None`. Widening a *return* union is
+not additive — a `mypy --strict` consumer that indexed a slot without a
+`None` check newly fails to type-check, which is the point: the runtime
+values it would have received were already wrong. It landed in a minor
+rather than waiting for the next major because the alternative was
+leaving a silent data-corruption defect in the released line, and
+because the runtime contract the widening makes true is the one both
+entry points already documented.
+
 [strenum]: https://docs.python.org/3/library/enum.html#enum.StrEnum
 [pep561]: https://peps.python.org/pep-0561/
 

big-code-analysis-book/src/python/batch.md

@@ -2,16 +2,19 @@
 
 `bca.analyze_batch(paths)` runs the same analysis as `bca.analyze`
 over every path in an iterable and **never raises on per-file
-errors**: each result element is either an analysis `dict` or a
-`bca.AnalysisFailure` describing the failure. Results preserve input
-order, so `zip(inputs, results)` lines up by index **when no path
-is skipped**. `analyze_batch` shares `analyze`'s keyword-only
-options — `exclude_tests`, `allow_lossy_path`, `skip_generated`
-(default `True`), and `metrics` — so the two entry points are
-behaviour-preserving.
+errors**: each result element is an analysis `dict`, a
+`bca.AnalysisFailure` describing the failure, or `None`. Results
+preserve input order, so `zip(inputs, results)` lines up by index
+**when no path is skipped**. `analyze_batch` shares `analyze`'s
+keyword-only options — `exclude_tests`, `allow_lossy_path`, `skip_generated`
+(default `True`), and `metrics` — so a file is treated the same by
+both entry points. (The list *shape* matches the
+`[bca.analyze(p) for p in paths]` comprehension only under
+`skip_generated=False`; with the default the comprehension keeps a
+`None` per skipped file while the batch drops the slot.)
 
 ```python
-{{#include ../../../big-code-analysis-py/examples/batch_processing.py:18:44}}
+{{#include ../../../big-code-analysis-py/examples/batch_processing.py:18:55}}
 ```
 
 A few key contracts:
@@ -30,6 +33,15 @@ A few key contracts:
   (the pre-2.0 default). This default flipped at 2.0 so that
   switching between `analyze` and `analyze_batch` no longer
   silently changes generated-file handling.
+* A file that cannot be parsed at all still holds its slot under
+  `skip_generated=False`, as `None`. The read gate shared with the
+  CLI walker declines a file of three bytes or fewer, one carrying
+  a UTF-16 BOM, and one whose leading window is not valid UTF-8;
+  that gate is unconditional, so before #1238 those files shrank
+  the list even with the flag off and the `zip` above mis-paired
+  every later entry. `None` is the same value single-file
+  `analyze` returns for them, and `bca.to_sarif` skips it, so a
+  batch list can be passed straight through.
 
 ## Walking a directory: `analyze_paths`
 
@@ -52,10 +64,19 @@ a file directly is always analysed regardless of `exclude` — an
 explicit request overrides ignore-style rules — while `include`
 still narrows it by basename. `respect_gitignore=False` opts into
 walking ignored files. The
-result is the same `list[FuncSpaceDict | AnalysisFailure]` shape and
-never-raise contract as `analyze_batch`, and it forwards the same
-`exclude_tests` / `allow_lossy_path` / `skip_generated` / `metrics` /
-`vcs` / `vcs_per_function` kwargs.
+result is the same `list[FuncSpaceDict | AnalysisFailure | None]`
+shape and never-raise contract as `analyze_batch`, and it forwards
+the same `exclude_tests` / `allow_lossy_path` / `skip_generated` /
+`metrics` / `vcs` / `vcs_per_function` kwargs.
+
+The `None` slots reach this entry point too, under
+`skip_generated=False`, one per discovered file the read gate
+declined. There is no caller-supplied ordering to pair against here —
+results follow the walk — so they are not there for a `zip`. What
+they buy is that a file the walk found but could not analyse stays
+visible in the output instead of disappearing from it. On a tree with
+many binary assets that is a lot of `None`s; filter them with
+`[r for r in results if r is not None]` if you only want records.
 
 ## Attaching change-history metrics
 
@@ -77,7 +98,7 @@ sequential sweep. For parallelism, fan the per-file `analyze`
 call out across a thread pool:
 
 ```python
-{{#include ../../../big-code-analysis-py/examples/batch_processing.py:47:59}}
+{{#include ../../../big-code-analysis-py/examples/batch_processing.py:58:70}}
 ```
 
 PyO3's `Python::detach` releases the GIL across each file's read +

big-code-analysis-book/src/python/errors.md

@@ -179,6 +179,11 @@ def report(paths: list[str]) -> None:
             log.warning(
                 "skip %s (%s): %s", path, slot.error_kind, slot.error
             )
+        elif slot is None:
+            # Nothing to parse: three bytes or fewer, a UTF-16 BOM, or a
+            # binary leading window. The slot is held open so the zip
+            # above stays aligned (#1238) — it is not a failure.
+            log.info("skip %s: empty or binary", path)
         else:
             log.info(
                 "ok %s sloc=%s", path,

big-code-analysis-book/src/python/flat-records.md

@@ -81,4 +81,6 @@ arrows) keep their `name == "<anonymous>"` marker verbatim —
 * `flatten_spaces` raises `TypeError` if the input is not a
   mapping; callers must filter `None` returns from `bca.analyze`
   (e.g. generated files with `skip_generated=True`) before
-  passing.
+  passing — and likewise the `None` slots `analyze_batch` emits
+  under `skip_generated=False`, which mark a file the read gate
+  declined to parse.

big-code-analysis-book/src/python/sarif.md

@@ -22,10 +22,13 @@ the CLI binary.
 
 * A single `dict` returned by `bca.analyze` or
   `bca.analyze_source`.
-* Any iterable yielding such dicts or `bca.AnalysisFailure`
-  instances (the natural shape of `bca.analyze_batch`'s return
-  value). `AnalysisFailure` entries are skipped silently — they
-  represent files that could not be analyzed, not findings.
+* Any iterable yielding such dicts, `bca.AnalysisFailure`
+  instances, and/or `None` (the natural shape of
+  `bca.analyze_batch`'s return value). `AnalysisFailure` and `None`
+  entries are skipped silently — they represent files for which no
+  record was emitted, not findings.
+* A scalar `None`, the documented return of `bca.analyze` for a
+  skipped file; it yields an empty SARIF run.
 
 ## Thresholds
 

big-code-analysis-py/README.md

@@ -282,29 +282,43 @@ page for the full contract.
 
 `bca.analyze_batch(paths)` runs the same analysis as `bca.analyze`
 over every path in an iterable and **never raises on per-file
-errors**: each result element is either an analysis ``dict`` or a
-`bca.AnalysisFailure` describing the failure. Results preserve input
-order, so `zip(inputs, results)` lines up by index **when no path is
-skipped**. `analyze_batch` accepts the same keyword-only options as
-`analyze` — `exclude_tests`, `allow_lossy_path`, `skip_generated`
-(default `True`), `metrics` — so migrating between the two is
-behavior-preserving.
+errors**: each result element is an analysis `dict`, a
+`bca.AnalysisFailure` describing the failure, or `None`. Results
+preserve input order, so `zip(inputs, results)` lines up by index
+**when no path is skipped**. `analyze_batch` accepts the same
+keyword-only options as `analyze` — `exclude_tests`,
+`allow_lossy_path`, `skip_generated` (default `True`), `metrics` — so
+migrating between the two is behavior-preserving.
 
 With the default `skip_generated=True`, a generated file is *skipped*
-and produces no result element (matching single-file `analyze`, which
-returns `None`), so the result list can be shorter than the input.
+and produces no result element, so the result list can be shorter than
+the input. Note the shape difference from the comprehension it
+replaces: `[bca.analyze(p) for p in paths]` keeps a `None` per skipped
+file (staying index-aligned), while the batch drops the slot — the
+per-file *filter semantics* match, the list shape does not.
 This default flipped at 2.0 — the pre-2.0 `analyze_batch` always
 analyzed generated files (`skip_generated=False`); pass that
 explicitly to restore one-element-per-input.
 
+`skip_generated=False` really is one element per input, including for
+files that cannot be parsed at all. The read gate `analyze` shares with
+the CLI walker declines a file of three bytes or fewer, one carrying a
+UTF-16 BOM, and one whose leading window is not valid UTF-8; those hold
+their slot as `None` — the same value single-file `analyze` returns for
+them — rather than dropping out of the list. Before #1238 they dropped
+regardless of the flag, so one binary file in a corpus silently
+mis-attributed every later result to the wrong path.
+
 ```python
 import big_code_analysis as bca
 
 paths = ["src/a.py", "src/missing.py", "src/b.rs"]
-results = bca.analyze_batch(paths)
+results = bca.analyze_batch(paths, skip_generated=False)
 for path, result in zip(paths, results):
     if isinstance(result, bca.AnalysisFailure):
         print(f"skipped {path}: ({result.error_kind}) {result.error}")
+    elif result is None:
+        print(f"skipped {path}: nothing to parse (empty or binary)")
     else:
         process(result)
 ```
@@ -324,6 +338,10 @@ results = bca.analyze_batch(paths)
 # now zip(paths, results) works
 ```
 
+A `None` slot means "no record emitted": the file exists and was read,
+but there was nothing to parse. `bca.to_sarif` skips those entries, so
+a batch list can be handed to it verbatim.
+
 `bca.AnalysisFailure` is a frozen value type with `path: str`,
 `error: str`, and `error_kind: Literal["UnsupportedLanguage",
 "ParseError", "IoError"]`. It implements `__eq__`, `__hash__`,
@@ -347,8 +365,8 @@ parallel single-file calls. With the 2.0 default `skip_generated=True`,
 `analyze_batch` applies the CLI's `is_generated` walker filter, so a
 generated file is skipped and contributes no result element — the
 result list can be shorter than the input. Pass `skip_generated=False`
-to analyze every file and guarantee one `dict`-or-`AnalysisFailure`
-element per input.
+to analyze every file and guarantee one
+`dict`-or-`AnalysisFailure`-or-`None` element per input.
 
 ## Flatten to records
 
@@ -413,7 +431,9 @@ be observed in later records.
 
 `flatten_spaces` raises `TypeError` if the input is not a mapping;
 callers must filter `None` returns from `bca.analyze` (e.g. when
-`skip_generated=True` matched a generated file) before passing.
+`skip_generated=True` matched a generated file) before passing — and
+likewise the `None` slots `analyze_batch` emits under
+`skip_generated=False`.
 
 ## Errors
 

big-code-analysis-py/examples/batch_processing.py

@@ -18,8 +18,8 @@
 def run(paths: Iterable[Path]) -> dict[str, int]:
     """Analyse ``paths`` as a batch and bucket successes vs failures.
 
-    Returns a small summary dict (`ok`, `errors`, `total`) so the
-    accompanying test can assert on it without re-parsing.
+    Returns a small summary dict (`ok`, `errors`, `skipped`, `total`) so
+    the accompanying test can assert on it without re-parsing.
     """
     materialised = list(paths)
     # `skip_generated=False` guarantees one result element per input
@@ -32,16 +32,27 @@ def run(paths: Iterable[Path]) -> dict[str, int]:
 
     ok = 0
     errors = 0
+    skipped = 0
     for path, result in zip(materialised, results, strict=True):
         if isinstance(result, bca.AnalysisFailure):
             errors += 1
             print(f"  skip {path}: ({result.error_kind}) {result.error}")
+        elif result is None:
+            # The read gate declined this file; the slot is held open
+            # so the strict zip above stays aligned (#1238).
+            skipped += 1
+            print(f"  skip {path}: nothing to parse (empty or binary)")
         else:
             ok += 1
             sloc = result["metrics"]["loc"]["sloc"]
             print(f"  ok   {path}: sloc = {sloc:.0f}")
 
-    return {"ok": ok, "errors": errors, "total": len(materialised)}
+    return {
+        "ok": ok,
+        "errors": errors,
+        "skipped": skipped,
+        "total": len(materialised),
+    }
 
 
 def run_parallel(paths: Iterable[Path], *, workers: int = 4) -> list[FuncSpaceDict | None]: