Skip to content

uucore: make Range::merge linear instead of quadratic - #14359

Open
haydonryan wants to merge 2 commits into
uutils:mainfrom
haydonryan:core-hr
Open

haydonryan wants to merge 2 commits into
uutils:mainfrom
haydonryan:core-hr

Conversation

@haydonryan

Copy link
Copy Markdown
Contributor

Deepseek found another optimization.

This one converts the loop from O(n^2) to O(n) for ranges.

I ran a few options for this PR, including one that was smaller in size by 96 bytes, but this is faster overall.

LLM generated below here:

Range::merge (src/uucore/src/lib/features/ranges.rs) merged overlapping
ranges with ranges.remove(j) inside a while loop. Because Vec::remove
shifts the tail, heavily-overlapping range lists were O(n²). Only caller is
cut (via Range::from_list).

Replaced with a single Vec::dedup_by pass that extends the bucket's high
on overlap. Output is unchanged — still sorted, disjoint, and adjacent ranges
are not merged. The only subtlety is that dedup_by(a, b) passes a = new
element, b = bucket, so the closure must extend b.high (the kept element),
not a.high (the dropped one).

Measurement (same harness, release)

Both versions measured in one harness, same input, realistic overlapping ranges:

ranges before (O(n²)) after (dedup_by) speedup
10 ~0 µs ~0 µs same
100 2 µs 0.3 µs 8x
1,000 94 µs 2 µs 47x
30,000 123 ms 226 µs 543x

The O(n) pass is dramatically faster at 30k+ ranges, but real cut invocations
use a handful of fields (argv caps ~30k, and a typical -f list is 1-100), so
the merge is sub-microsecond, once-at-startup work there. The value of this
change is not a user-visible speedup: it removes the O(n²) blowup (a large
overlapping field list would otherwise hang cut) and shrinks the binary by
256 B. dedup_by was also the smallest/fastest of the four O(n) variants A/B'd
(baseline, extra-Vec, in-place swap, dedup_by).

dedup_by was also the fastest of the four O(n) variants A/B'd (baseline,
extra-Vec, in-place swap, dedup_by) in every all-overlap case; the in
place variants are not measurably faster and compile to a larger binary.

Verification

  • cargo fmt clean, cargo clippy --release --bin coreutils clean (0 warnings)
  • uucore lib tests: 75/75 pass
  • cut integration tests: 103/103 pass

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/cut/bounded-memory (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/pr/bounded-memory (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/date/resolution (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/printf/printf-surprise is now being skipped but was previously passing.

@sylvestre

Copy link
Copy Markdown
Contributor

any reason why codspeed don't detect the improvements?
also, please provide a hyperfine example with before this change, after and against GNU

@anastygnome

anastygnome commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

The dedup is Indeed better, as you don't shift the queue every time

You're doing a sort which is n*logarithmic to begin with.
Now because sort in rust is adaptative, it can perform linearly in some cases which explains your benchmark results, but not in codspeed

@sylvestre TLDR : the code is still better, but not as much as the clanker said

Comment thread src/uucore/src/lib/features/ranges.rs
@haydonryan

Copy link
Copy Markdown
Contributor Author

A lot of things going on at the moment. Will get onto this this weekend. Apologies for the delay

@haydonryan
haydonryan force-pushed the core-hr branch 2 times, most recently from 0abdb08 to df42a7e Compare September 18, 2026 15:26
@haydonryan

Copy link
Copy Markdown
Contributor Author

Again, apologies for the slow replies - I have an ongoing family medical situation that is taking a lot of my focus.

any reason why codspeed don't detect the improvements? also, please provide a hyperfine example with before this change, after and against GNU

I had to test using a custom harness, as cut is the tool that exposes this but the rest of the tool dominated the testing. Not sure how codspeed works, if it's just using a test suite on the individual functions or what.

@anastygnome Your code suggestion was excellent! This is the first time i've had a PR have a code review suggestion - please let me know if I did it correctly (I also wasn't sure if I should give you attribution or how to do that on the updated line).

Ran the clanker:

Variant Binary size (B) Delta
baseline ranges.sort() 14,007,376
variant sort_unstable_by_key 14,003,608 −3,768 B

Runtime (Range::merge, median of 9 iterations, ms)

Case n baseline sort() variant by_key Speedup
random-overlap 100,000 8.19 2.39 3.4×
random-overlap 300,000 21.63 7.82 2.8×
random-overlap 1,000,000 77.47 27.15 2.9×
all-overlap 1,000,000 2.28 1.04 2.2×
disjoint 1,000,000 2.22 1.06 2.1×

Measured with an isolated mergebench harness (uucore [[example]]) calling
Range::merge directly on synthetic range lists — no process startup, argv,
parse, or I/O noise (same approach as the original coll2.md A/B).

@anastygnome

anastygnome commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

@haydonryan thanks!

If you want to add a contribution, amend your commit message, add a blank line after the title, then on the third line you can put

Co-authored-by: anastygnome <15268361+anastygnome@users.noreply.github.com>

or anyone really :)

@haydonryan

haydonryan commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! Added!

haydonryan and others added 2 commits September 18, 2026 19:26
Co-authored-by: anastygnome <15268361+anastygnome@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants