Skip to content

Commit c1bf8a4

Browse files
coderabbitai[bot]CodeRabbit
andauthored
fix: apply CodeRabbit auto-fixes
Fixed 15 file(s) based on 1 failed pre-merge check. Co-authored-by: CodeRabbit <noreply@coderabbit.ai>
1 parent ad483b0 commit c1bf8a4

15 files changed

Lines changed: 1144 additions & 320 deletions

‎.gitattributes‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,6 @@ Containerfile text eol=lf
5252
# Lock files
5353
Cargo.lock text eol=lf -diff
5454
flake.lock text eol=lf -diff
55+
56+
# Test fixtures for byte detection - preserve as-is (binary)
57+
tests/fixtures/byte_detection/*.txt binary

‎CHANGES-SUMMARY.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
<!-- SPDX-License-Identifier: CC-BY-SA-4.0 -->
2+
# Byte Detection Enhancement - Changes Summary
3+
4+
## Overview
5+
6+
This change implements a dedicated leading-BOM detection system and comprehensive test coverage for all byte-level detection rules in the CI pipeline.
7+
8+
## What Was Added
9+
10+
### 1. Dedicated Leading-BOM Check
11+
- **File:** `.github/workflows/dogfood-gate.yml`
12+
- **Change:** Added separate step to detect UTF-8 BOM (EF BB BF) at file start
13+
- **Behavior:** Advisory warning (does NOT block)
14+
- **Reason:** Leading BOMs are conceptually different from mid-file invisible characters
15+
16+
### 2. Comprehensive Test Suite
17+
- **Test Script:** `tests/test_byte_detection.sh` (9 passing tests)
18+
- **Test Fixtures:** `tests/fixtures/byte_detection/` (6 test files)
19+
- Leading BOM detection
20+
- Mid-file BOM detection
21+
- NUL byte detection
22+
- Backspace character detection
23+
- Mid-file invisible Unicode detection
24+
- Clean file validation
25+
26+
### 3. Documentation
27+
- **`docs/byte-detection-rules.md`** - Complete specification of all detection rules
28+
- **`docs/BYTE-DETECTION-IMPLEMENTATION.md`** - Implementation summary
29+
- **`tests/fixtures/byte_detection/README.md`** - Test fixture documentation
30+
31+
### 4. Integration
32+
- **`.gitattributes`** - Preserve test fixtures as binary
33+
- **`Justfile`** - Added `test-byte-detection` recipe
34+
35+
## Detection Rules
36+
37+
The system now has three distinct checks:
38+
39+
### Check 1: Leading BOM (NEW)
40+
- **Pattern:** UTF-8 BOM at file start only
41+
- **Action:** Advisory warning
42+
- **Status:** Does NOT block
43+
44+
### Check 2: C0/NUL Corruption (EXISTING)
45+
- **Pattern:** Control characters and NUL bytes
46+
- **Action:** Error + CI failure
47+
- **Status:** BLOCKS the gate
48+
49+
### Check 3: Mid-file Invisible Unicode (EXISTING)
50+
- **Pattern:** Zero-width spaces, NBSP, etc.
51+
- **Action:** Advisory warning
52+
- **Status:** Does NOT block
53+
54+
## Files Changed
55+
56+
```
57+
Modified:
58+
.gitattributes (+ test fixture preservation)
59+
.github/workflows/dogfood-gate.yml (+ leading BOM step, updated summary)
60+
Justfile (+ test-byte-detection recipe)
61+
62+
Added:
63+
tests/test_byte_detection.sh (executable test script)
64+
tests/fixtures/byte_detection/ (directory)
65+
tests/fixtures/byte_detection/README.md
66+
tests/fixtures/byte_detection/with_leading_bom.txt
67+
tests/fixtures/byte_detection/with_mid_bom.txt
68+
tests/fixtures/byte_detection/with_nul_byte.txt
69+
tests/fixtures/byte_detection/with_backspace.txt
70+
tests/fixtures/byte_detection/with_mid_invisible.txt
71+
tests/fixtures/byte_detection/clean_file.txt
72+
docs/byte-detection-rules.md
73+
docs/BYTE-DETECTION-IMPLEMENTATION.md
74+
CHANGES-SUMMARY.md (this file)
75+
```
76+
77+
## Test Results
78+
79+
All tests pass (9/9):
80+
```
81+
Test Group 1: Leading BOM Detection
82+
✓ Leading BOM detected at file start
83+
✓ Clean file has no leading BOM
84+
✓ Mid-file BOM not detected as leading BOM
85+
86+
Test Group 2: C0 Control Characters (Blocking)
87+
✓ NUL byte detected
88+
✓ Backspace character detected
89+
✓ Clean file has no C0/NUL characters
90+
91+
Test Group 3: Mid-file Invisible Unicode (Advisory)
92+
✓ Zero-width space detected
93+
✓ BOM detected when in middle of file
94+
✓ Clean file has no invisible Unicode
95+
```
96+
97+
## Consistency Verification
98+
99+
✓ CI workflow patterns match test script patterns exactly
100+
✓ Test fixtures excluded from CI scanning
101+
✓ Git attributes preserve test fixtures
102+
✓ Documentation complete and consistent
103+
104+
## How to Test Locally
105+
106+
```bash
107+
# Run the full test suite
108+
./tests/test_byte_detection.sh
109+
110+
# Or using Justfile
111+
just test-byte-detection
112+
```
113+
114+
Expected output: All 9 tests pass.
115+
116+
## CI Behavior
117+
118+
On pull requests and pushes to main/master:
119+
1. Leading BOM check runs first (advisory)
120+
2. Invisible character check runs (includes C0/NUL blocking check)
121+
3. Summary shows counts for each category
122+
123+
**Blocking conditions:**
124+
- C0/NUL corruption found → CI FAILS
125+
- Leading BOM found → Advisory warning only
126+
- Invisible Unicode found → Advisory warning only
127+
128+
## Rationale
129+
130+
### Why Separate Leading-BOM Check?
131+
132+
1. **Conceptually distinct:** File-start BOM is an encoding quirk, not mid-file typography
133+
2. **Different detection:** Simple byte-level pattern vs complex PCRE
134+
3. **Clear reporting:** Separate GitHub annotation makes it obvious what was found
135+
4. **Future extensibility:** Easy to add other BOM types (UTF-16, UTF-32)
136+
137+
### Why Comprehensive Tests?
138+
139+
1. **Confidence:** Ensures patterns actually work as expected
140+
2. **Regression prevention:** Changes to patterns immediately show impact
141+
3. **Documentation:** Test fixtures serve as executable examples
142+
4. **CI-local consistency:** Same patterns used in both contexts
143+
144+
## References
145+
146+
- **Recent commits:**
147+
- `ad483b0` - fix(ci): make invisible-character PCRE locale-independent
148+
- `eaa7168` - fix(ci): enforce C0/NUL corruption in-step; warn on invisible Unicode
149+
- `a69de22` - fix(ci): the invisible-character gate never matched anything
150+
151+
- **Owner ruling (2026-08-28):**
152+
- C0/NUL corruption → BLOCKS (file corruption)
153+
- Leading BOM → Advisory (encoding quirk)
154+
- Invisible Unicode → Advisory (legitimate typography in ~2,100 files)
155+
156+
## Next Steps
157+
158+
After merging:
159+
1. Monitor CI runs for any new BOM detections
160+
2. Consider externalizing patterns to `config.ncl` or `ByteDetector.affine`
161+
3. Add support for UTF-16/UTF-32 BOM detection if needed
162+
4. Consider building standalone `empty-linter` binary
163+
164+
## Questions?
165+
166+
See:
167+
- **Full specification:** `docs/byte-detection-rules.md`
168+
- **Implementation details:** `docs/BYTE-DETECTION-IMPLEMENTATION.md`
169+
- **Test fixtures:** `tests/fixtures/byte_detection/README.md`

0 commit comments

Comments
 (0)