sed: implement \\U \\L \\u \\l \\E case-conversion escapes in s/// by dedsec-terminal · Pull Request #557 · uutils/sed

dedsec-terminal · 2026-09-15T19:32:33Z

Fixes #540.

Implements GNU s/// replacement case-conversion escapes:

\U uppercase until \L/\E, \L lowercase until \U/\E
\u/\l one-shot next-character conversion
\E ends persistent conversion and clears pending one-shot

Details:

New ReplacementPart::{Upper,Lower,UpperFirst,LowerFirst,End} in command.rs; compile_replacement parses them before parse_char_escape so \u/\U in replacements are case-conversion directives rather than \uXXXX/\UXXXXXXXX Unicode escapes (matches GNU sed behavior; README updated accordingly). Escaped delimiter still wins (e.g. sU...U\UU).
Runtime append_with_case applies persistent + one-shot to literals, &, \1..\9. Empty matches leave one-shot pending (GNU s/(b?)-/\u\1x/g carry), any produced char consumes it. Fresh state per substituted occurrence so g does not propagate.
Factored out case selection into take_case(single, persistent) helper.
UTF-8 mode is Unicode-aware with invalid-byte passthrough; byte mode is ASCII-only.

Verified against GNU sed 4.9 including the issue table (ABC DEF, ABc def) plus \U&\E, \U\1-\2, empty-group g cases, \u\l/\l\u last-wins, \U\l/\L\u combos.

Tests: cargo test --lib 385 passed, cargo test --test tests 268 passed, cargo clippy --lib clean, cargo fmt --check clean.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-16T05:08:42Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

codecov · 2026-09-16T05:09:42Z

Codecov Report

❌ Patch coverage is 97.71689% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.03%. Comparing base (f7fb0d7) to head (58b6c03).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/sed/command.rs	98.58%	8 Missing ⚠️
src/sed/processor.rs	0.00%	6 Missing ⚠️
src/sed/compiler.rs	98.80%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #557      +/-   ##
==========================================
+ Coverage   83.86%   85.03%   +1.16%     
==========================================
  Files          14       14              
  Lines        7203     7824     +621     
  Branches      424      448      +24     
==========================================
+ Hits         6041     6653     +612     
- Misses       1157     1166       +9     
  Partials        5        5

Flag	Coverage Δ
macos_latest	`86.10% <97.71%> (+1.11%)`	⬆️
ubuntu_latest	`86.25% <97.71%> (+1.07%)`	⬆️
windows_latest	`0.00% <0.00%> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codspeed · 2026-09-16T05:11:56Z

Merging this PR will improve performance by 5.87%

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 10 untouched benchmarks

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`number_fix`	1.2 s	1.2 s	+5.87%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing dedsec-terminal:fix-540-case-conversion-escapes (58b6c03) with main (0554c50)}

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

dedsec-terminal · 2026-09-16T20:19:56Z

Hi @dspinellis — friendly review request on this s/// case-conversion implementation (fixes #540; GNU suite +1, utf8-ru). Just pushed a perf follow-up (fast paths when no conversion directives; 265 tests green, fmt+clippy clean) to address the Codspeed delta. Let me know if you need anything else. Thanks!

github-actions · 2026-09-17T06:53:11Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

dedsec-terminal · 2026-09-17T18:32:55Z

Follow-up pushed (23b7709): patch coverage for the fast paths - unit tests for unmatched-group/single-literal fast paths, flag-sync contract panics, LowerFirst invalid-byte handling, compiler case-directive parsing + flag, and end-to-end substitution tests; plus removal of two dead None arms preempted by the all-None early return. Full suite green (lib + integration), fmt/clippy clean. Remaining reds are not from this change: Android jobs fail in sdkmanager setup, and CodSpeed shows -3.75% on number_fix only (genome/access_log improved; likely noise, happy to dig if you want).

github-actions · 2026-09-18T05:22:20Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-22T06:16:03Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-22T08:28:38Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-09-22T11:28:09Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 15 / FAILED: 42 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 14 / FAILED: 43 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +1
  FAILED: -1

Test improvements (1):
  + utf8-ru

…ixes uutils#540)

Addresses Codspeed -7.9pct regression: common case (no U/L/u/l/E) now plain memcpy via has_case_conversion flag; byte-mode hoists the branch; UTF-8 path validates once per segment; ASCII case uses to_ascii_* fast path. All 265 tests pass; fmt+clippy clean.

Add unit tests for the no-conversion fast paths (unmatched groups, single literals, flag-sync contracts), LowerFirst invalid-byte handling, compiler case-directive parsing and the has_case_conversion flag, plus end-to-end substitution coverage. Collapse two provably-dead None arms preempted by the all-None early return. Deduplicate GNU testsuite bot comments.

- revert unrelated GnuComment workflow modifications - update README to note \u/\U replacement behavior - consolidate case escape mapping in compiler - factor out take_case helper and shorten comments in command.rs

…placement

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

github-actions · 2026-10-01T05:16:37Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 19 / FAILED: 38 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 17 / FAILED: 40 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + subst-replacement
  + utf8-ru

github-actions · 2026-10-01T07:43:33Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 19 / FAILED: 38 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 17 / FAILED: 40 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + subst-replacement
  + utf8-ru

Copilot

Copilot review overview

🟡 Changes recommended

Fix the Unicode lowercasing mismatch and document all five directives and their semantics.

Review effort: Lite
Findings: 1 · 1

Open (2)

Use GNU locale-aware single-character lowercase mapping · New
Document all replacement-case conversion directives and semantics · New

Copilot

Copilot review overview

🟡 Changes recommended

Address malformed UTF-8 handling and POSIX Unicode escape parsing.

Review effort: Lite
Findings: 1

Open (1)

Disable Unicode replacement escapes in POSIX mode · New

Resolved since last review (2)

Use GNU locale-aware single-character lowercase mapping
Document all replacement-case conversion directives and semantics

github-actions · 2026-10-01T08:01:37Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 19 / FAILED: 38 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 17 / FAILED: 40 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + subst-replacement
  + utf8-ru

- Encapsulate ReplacementTemplate.has_case_conversion with a private field and public getter - Replace panic arm in render_parts with unreachable!() and remove artificial panic tests - Drop backslash for unrecognized escapes in replacement strings to match GNU sed - Disable GNU case-conversion escapes in replacement strings under POSIX mode - Add unit and integration regression tests for unrecognized escapes and POSIX mode

github-actions · 2026-10-01T08:22:22Z

GNU sed testsuite comparison:

Test results comparison:
  Current:   TOTAL: 65 / PASSED: 19 / FAILED: 38 / SKIPPED: 8
  Reference: TOTAL: 65 / PASSED: 17 / FAILED: 40 / SKIPPED: 8

Changes from main branch:
  TOTAL: +0
  PASSED: +2
  FAILED: -2

Test improvements (2):
  + subst-replacement
  + utf8-ru

Copilot

Copilot review overview

🟢 Approval recommended

The reviewed changes are fully covered by the supplied compatibility tests and documentation.

Review effort: Lite
Findings: None

Resolved since last review (1)

Disable Unicode replacement escapes in POSIX mode

Copilot AI lite review requested due to automatic review settings September 15, 2026 19:32

Copilot AI reviewed Sep 15, 2026

dedsec-terminal mentioned this pull request Sep 15, 2026

s/// replacement: implement the \U \L \u \l \E case-conversion escapes #540

Open

Copilot AI review requested due to automatic review settings September 16, 2026 17:26

Copilot AI reviewed Sep 16, 2026

Copilot AI review requested due to automatic review settings September 16, 2026 20:19

Copilot AI reviewed Sep 16, 2026

Copilot AI review requested due to automatic review settings September 17, 2026 18:10

Copilot AI reviewed Sep 17, 2026

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread .github/workflows/GnuComment.yml Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/compiler.rs Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/compiler.rs Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

sylvestre reviewed Sep 21, 2026

View reviewed changes

Comment thread src/sed/command.rs

Copilot AI review requested due to automatic review settings September 22, 2026 06:10

Copilot AI reviewed Sep 22, 2026

Copilot AI review requested due to automatic review settings September 22, 2026 08:03

Copilot AI reviewed Sep 22, 2026

Copilot AI review requested due to automatic review settings September 22, 2026 09:26

Copilot AI reviewed Sep 22, 2026

sylvestre reviewed Sep 22, 2026

View reviewed changes

Comment thread src/sed/command.rs Outdated

dedsec-terminal requested a review from sylvestre September 22, 2026 10:26

sylvestre reviewed Sep 24, 2026

View reviewed changes

Comment thread src/sed/command.rs

dedsec-terminal and others added 7 commits October 1, 2026 02:40

sed: implement \\U \\L \\u \\l \\E case-conversion escapes in s/// (f…

f4d2ff9

…ixes uutils#540)

style: apply cargo fmt to case-conversion code

5f5856a

refactor: address review feedback on case-conversion escapes

4420e86

- revert unrelated GnuComment workflow modifications - update README to note \u/\U replacement behavior - consolidate case escape mapping in compiler - factor out take_case helper and shorten comments in command.rs

perf(subst): avoid intermediate buffer allocations in substitution re…

3f5d89c

…placement

refactor(subst): address review feedback on case-conversion escapes

de9174f

dedsec-terminal force-pushed the fix-540-case-conversion-escapes branch from 626ce4f to 7fefe2b Compare September 30, 2026 23:57

Copilot AI lite review requested due to automatic review settings September 30, 2026 23:57

Copilot AI reviewed Sep 30, 2026

dedsec-terminal force-pushed the fix-540-case-conversion-escapes branch from 7fefe2b to ecd940c Compare October 1, 2026 07:33

Copilot AI lite review requested due to automatic review settings October 1, 2026 07:33

Copilot started reviewing on behalf of dedsec-terminal October 1, 2026 07:41 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread src/sed/command.rs

Comment thread docs/src/extensions.md Outdated

Copilot AI lite review requested due to automatic review settings October 1, 2026 07:57

dedsec-terminal force-pushed the fix-540-case-conversion-escapes branch from ecd940c to 75865ff Compare October 1, 2026 07:57

Copilot started reviewing on behalf of dedsec-terminal October 1, 2026 07:58 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Comment thread src/sed/compiler.rs

dedsec-terminal force-pushed the fix-540-case-conversion-escapes branch from 75865ff to 58b6c03 Compare October 1, 2026 08:19

Copilot AI lite review requested due to automatic review settings October 1, 2026 08:19

Copilot started reviewing on behalf of dedsec-terminal October 1, 2026 08:20 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

Conversation

dedsec-terminal commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 16, 2026

Uh oh!

codecov Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed Bot commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 5.87%

Performance Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dedsec-terminal commented Sep 16, 2026

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

dedsec-terminal commented Sep 17, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

github-actions Bot commented Oct 1, 2026

Uh oh!

github-actions Bot commented Oct 1, 2026

dedsec-terminal commented Sep 15, 2026 •

edited

Loading

codecov Bot commented Sep 16, 2026 •

edited

Loading

codspeed Bot commented Sep 16, 2026 •

edited

Loading