Skip to content

feat: add token frequencies - #438

Merged
cheb0 merged 7 commits into
mainfrom
0-doc-freqs
Jul 28, 2026
Merged

feat: add token frequencies#438
cheb0 merged 7 commits into
mainfrom
0-doc-freqs

Conversation

@cheb0

@cheb0 cheb0 commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

Description

Adds token frequency which can be used to optimize query performance


  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

@dkharms
dkharms force-pushed the 336-merge-source branch 3 times, most recently from 50256a3 to b017111 Compare June 26, 2026 08:54
@dkharms
dkharms force-pushed the 336-merge-source branch 3 times, most recently from 8917131 to 261846d Compare July 6, 2026 12:37
Base automatically changed from 336-merge-source to main July 6, 2026 12:40
@cheb0
cheb0 force-pushed the 0-doc-freqs branch 3 times, most recently from a8feb9e to 323d2ff Compare July 8, 2026 08:15
@codecov-commenter

codecov-commenter commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.69014% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.09%. Comparing base (e1ed059) to head (6863629).

Files with missing lines Patch % Lines
frac/sealed/token/block_loader.go 83.90% 7 Missing and 7 partials ⚠️
cmd/seq-db/seq-db.go 0.00% 10 Missing ⚠️
indexwriter/blocks.go 75.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #438      +/-   ##
==========================================
+ Coverage   70.97%   71.09%   +0.11%     
==========================================
  Files         232      232              
  Lines       18293    18404     +111     
==========================================
+ Hits        12984    13084     +100     
- Misses       4336     4340       +4     
- Partials      973      980       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Sealing_NoSort-4 25381c 6afc04
5314.00 allocs/op 6000.00 allocs/op 1.13 🔴
Sealing_WithSort-4 25381c 6afc04
5370.00 allocs/op 6074.00 allocs/op 1.13 🔴

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Sealing_NoSort-4 25381c 1d308e
5314.00 allocs/op 5977.00 allocs/op 1.12 🔴
Sealing_WithSort-4 25381c 1d308e
5370.00 allocs/op 6044.00 allocs/op 1.13 🔴

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggDeep/size=1000000-4 25381c 8e734e
4659647.00 ns/op 5193054.00 ns/op 1.11 🔴
Not/size=10000-4 25381c 8e734e
4.51 ns/op 5.02 ns/op 1.11 🔴
Not/size=1000000-4 25381c 8e734e
4.66 ns/op 5.22 ns/op 1.12 🔴
ParseESTime/es_stdlib-4 25381c 8e734e
182.30 ns/op 204.10 ns/op 1.12 🔴
ParseESTime/handwritten-4 25381c 8e734e
39.96 ns/op 44.60 ns/op 1.12 🔴
ParseESTime/rfc3339-4 25381c 8e734e
51.37 ns/op 57.38 ns/op 1.12 🔴

@eguguchkin eguguchkin added this to the v0.75.0 milestone Jul 13, 2026
Comment thread config/config.go Outdated
Comment thread consts/consts.go Outdated
Comment thread frac/sealed/token/block_loader_test.go Outdated
Comment thread packer/delta_bitpacker.go
Comment thread frac/sealed/token/block_loader.go
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggDeep/size=1000-4 e1ed05 1a3005
5152.00 ns/op 6205.00 ns/op 1.20 🔴
AggDeep/size=10000-4 e1ed05 1a3005
51317.00 ns/op 65033.00 ns/op 1.27 🔴
AggDeep/size=1000000-4 e1ed05 1a3005
5230924.00 ns/op 6451866.00 ns/op 1.23 🔴
AggWide/size=1000-4 e1ed05 1a3005
5168.00 ns/op 6262.00 ns/op 1.21 🔴
AggWide/size=10000-4 e1ed05 1a3005
50505.00 ns/op 64631.00 ns/op 1.28 🔴
AggWide/size=1000000-4 e1ed05 1a3005
529.00 B/op 638.00 B/op 1.21 🔴
5671609.00 ns/op 6926339.00 ns/op 1.22 🔴
And/size=1000-4 e1ed05 1a3005
5.05 ns/op 6.09 ns/op 1.21 🔴
And/size=10000-4 e1ed05 1a3005
5.05 ns/op 6.02 ns/op 1.19 🔴
And/size=1000000-4 e1ed05 1a3005
4.52 ns/op 5.72 ns/op 1.27 🔴
AndTree/size=1000-4 e1ed05 1a3005
4.41 ns/op 5.44 ns/op 1.23 🔴
AndTree/size=10000-4 e1ed05 1a3005
4.41 ns/op 5.46 ns/op 1.24 🔴
AndTree/size=1000000-4 e1ed05 1a3005
4.81 ns/op 6.17 ns/op 1.28 🔴
Bitmask-4 e1ed05 1a3005
223724052.00 ns/op 270037682.00 ns/op 1.21 🔴
Block_Pack-4 e1ed05 1a3005
109010.00 ns/op 141325.00 ns/op 1.30 🔴
Block_Unpack-4 e1ed05 1a3005
92335.00 ns/op 126887.00 ns/op 1.37 🔴
BucketClean-4 e1ed05 1a3005
45283.00 ns/op 57129.00 ns/op 1.26 🔴
Complex/size=1000-4 e1ed05 1a3005
5.05 ns/op 6.15 ns/op 1.22 🔴
Complex/size=10000-4 e1ed05 1a3005
5.05 ns/op 6.30 ns/op 1.25 🔴
Complex/size=1000000-4 e1ed05 1a3005
5.54 ns/op 6.88 ns/op 1.24 🔴
ESBulk-4 e1ed05 1a3005
4767.63 MB/s 4101.78 MB/s 0.86 🔴
67615.00 ns/op 78503.00 ns/op 1.16 🔴
FindSequence_Random/extra-large-4 e1ed05 1a3005
9692.29 MB/s 8672.16 MB/s 0.89 🔴
108262.00 ns/op 120913.00 ns/op 1.12 🔴
FindSequence_Random/large-4 e1ed05 1a3005
15730.57 MB/s 13733.46 MB/s 0.87 🔴
1042.00 ns/op 1193.00 ns/op 1.14 🔴
FindSequence_Random/medium-4 e1ed05 1a3005
9811.02 MB/s 8428.18 MB/s 0.86 🔴
104.60 ns/op 121.50 ns/op 1.16 🔴
FindSequence_Random/small-4 e1ed05 1a3005
6282.24 MB/s 5330.33 MB/s 0.85 🔴
40.80 ns/op 48.03 ns/op 1.18 🔴
FindSequence_Random/tiny-4 e1ed05 1a3005
2537.71 MB/s 2056.92 MB/s 0.81 🔴
25.22 ns/op 31.11 ns/op 1.23 🔴
Indexer-4 e1ed05 1a3005
271028262.00 ns/op 311100947.00 ns/op 1.15 🔴
MergeQPRs_ReusingQPR-4 e1ed05 1a3005
3722025.00 ns/op 4546939.00 ns/op 1.22 🔴
MergeSource-4 e1ed05 1a3005
1649170920.00 ns/op 1962786073.00 ns/op 1.19 🔴
MutexListAppend-4 e1ed05 1a3005
184.95 MB/s 148.64 MB/s 0.80 🔴
86631513.00 ns/op 107641518.00 ns/op 1.24 🔴
NAnd/size=1000-4 e1ed05 1a3005
4.68 ns/op 5.90 ns/op 1.26 🔴
NAnd/size=10000-4 e1ed05 1a3005
4.70 ns/op 5.81 ns/op 1.24 🔴
NAnd/size=1000000-4 e1ed05 1a3005
4.79 ns/op 5.95 ns/op 1.24 🔴
Not/size=1000-4 e1ed05 1a3005
5.03 ns/op 6.13 ns/op 1.22 🔴
Not/size=10000-4 e1ed05 1a3005
5.04 ns/op 6.05 ns/op 1.20 🔴
Not/size=1000000-4 e1ed05 1a3005
5.21 ns/op 6.81 ns/op 1.31 🔴
NotEmpty/size=1000-4 e1ed05 1a3005
5.05 ns/op 6.38 ns/op 1.26 🔴
NotEmpty/size=10000-4 e1ed05 1a3005
5.05 ns/op 6.04 ns/op 1.20 🔴
NotEmpty/size=1000000-4 e1ed05 1a3005
5.04 ns/op 6.19 ns/op 1.23 🔴
Or/size=1000-4 e1ed05 1a3005
4.64 ns/op 5.59 ns/op 1.21 🔴
Or/size=10000-4 e1ed05 1a3005
4.67 ns/op 5.86 ns/op 1.26 🔴
Or/size=1000000-4 e1ed05 1a3005
4.71 ns/op 5.95 ns/op 1.26 🔴
OrTree/size=1000-4 e1ed05 1a3005
4.69 ns/op 5.92 ns/op 1.26 🔴
OrTree/size=10000-4 e1ed05 1a3005
4.68 ns/op 5.77 ns/op 1.23 🔴
OrTree/size=1000000-4 e1ed05 1a3005
5.45 ns/op 6.83 ns/op 1.25 🔴
OrTreeNextGeq/size=1000-4 e1ed05 1a3005
4.68 ns/op 5.94 ns/op 1.27 🔴
OrTreeNextGeq/size=10000-4 e1ed05 1a3005
4.68 ns/op 5.53 ns/op 1.18 🔴
OrTreeNextGeq/size=1000000-4 e1ed05 1a3005
5.26 ns/op 6.61 ns/op 1.26 🔴
ParseESTime/es_stdlib-4 e1ed05 1a3005
198.70 ns/op 251.00 ns/op 1.26 🔴
ParseESTime/handwritten-4 e1ed05 1a3005
44.03 ns/op 53.39 ns/op 1.21 🔴
ParseESTime/rfc3339-4 e1ed05 1a3005
57.38 ns/op 68.55 ns/op 1.19 🔴
ProcessDocuments-4 e1ed05 1a3005
13324.00 B/op 16418.00 B/op 1.23 🔴
92.83 MB/s 76.60 MB/s 0.83 🔴
4444012.00 ns/op 5378833.00 ns/op 1.21 🔴
Sealing_NoSort-4 e1ed05 1a3005
1194959516.00 ns/op 1429481624.00 ns/op 1.20 🔴
Sealing_WithSort-4 e1ed05 1a3005
2294962601.00 ns/op 2860952641.00 ns/op 1.25 🔴
SeqListAppend-4 e1ed05 1a3005
116.04 MB/s 98.03 MB/s 0.84 🔴
137913962.00 ns/op 163222149.00 ns/op 1.18 🔴
SeqQLParsing-4 e1ed05 1a3005
1266.00 ns/op 1539.00 ns/op 1.22 🔴
SeqQLParsingLong-4 e1ed05 1a3005
10973.00 ns/op 13982.00 ns/op 1.27 🔴

@github-actions

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
AggDeep/size=1000-4 e1ed05 2a78f2
5152.00 ns/op 6395.00 ns/op 1.24 🔴
AggDeep/size=10000-4 e1ed05 2a78f2
51317.00 ns/op 63718.00 ns/op 1.24 🔴
AggDeep/size=1000000-4 e1ed05 2a78f2
5230924.00 ns/op 6509844.00 ns/op 1.24 🔴
AggWide/size=1000-4 e1ed05 2a78f2
5168.00 ns/op 6466.00 ns/op 1.25 🔴
AggWide/size=10000-4 e1ed05 2a78f2
50505.00 ns/op 60167.00 ns/op 1.19 🔴
AggWide/size=1000000-4 e1ed05 2a78f2
529.00 B/op 682.00 B/op 1.29 🔴
5671609.00 ns/op 7279932.00 ns/op 1.28 🔴
And/size=1000-4 e1ed05 2a78f2
5.05 ns/op 6.46 ns/op 1.28 🔴
And/size=10000-4 e1ed05 2a78f2
5.05 ns/op 6.25 ns/op 1.24 🔴
And/size=1000000-4 e1ed05 2a78f2
4.52 ns/op 6.06 ns/op 1.34 🔴
AndTree/size=1000-4 e1ed05 2a78f2
4.41 ns/op 5.48 ns/op 1.24 🔴
AndTree/size=10000-4 e1ed05 2a78f2
4.41 ns/op 5.63 ns/op 1.28 🔴
AndTree/size=1000000-4 e1ed05 2a78f2
4.81 ns/op 6.26 ns/op 1.30 🔴
Bitmask-4 e1ed05 2a78f2
223724052.00 ns/op 282562256.00 ns/op 1.26 🔴
Block_Pack-4 e1ed05 2a78f2
109010.00 ns/op 147369.00 ns/op 1.35 🔴
Block_Unpack-4 e1ed05 2a78f2
92335.00 ns/op 128265.00 ns/op 1.39 🔴
BucketClean-4 e1ed05 2a78f2
45283.00 ns/op 57377.00 ns/op 1.27 🔴
Complex/size=1000-4 e1ed05 2a78f2
5.05 ns/op 6.24 ns/op 1.24 🔴
Complex/size=10000-4 e1ed05 2a78f2
5.05 ns/op 6.37 ns/op 1.26 🔴
Complex/size=1000000-4 e1ed05 2a78f2
5.54 ns/op 6.73 ns/op 1.22 🔴
ESBulk-4 e1ed05 2a78f2
4767.63 MB/s 4035.73 MB/s 0.85 🔴
67615.00 ns/op 79787.00 ns/op 1.18 🔴
FindSequence_Random/extra-large-4 e1ed05 2a78f2
9692.29 MB/s 8648.11 MB/s 0.89 🔴
108262.00 ns/op 121249.00 ns/op 1.12 🔴
FindSequence_Random/large-4 e1ed05 2a78f2
15730.57 MB/s 13889.58 MB/s 0.88 🔴
1042.00 ns/op 1180.00 ns/op 1.13 🔴
FindSequence_Random/medium-4 e1ed05 2a78f2
9811.02 MB/s 8154.40 MB/s 0.83 🔴
104.60 ns/op 125.60 ns/op 1.20 🔴
FindSequence_Random/small-4 e1ed05 2a78f2
6282.24 MB/s 5154.49 MB/s 0.82 🔴
40.80 ns/op 49.67 ns/op 1.22 🔴
FindSequence_Random/tiny-4 e1ed05 2a78f2
2537.71 MB/s 2113.87 MB/s 0.83 🔴
25.22 ns/op 30.28 ns/op 1.20 🔴
Indexer-4 e1ed05 2a78f2
271028262.00 ns/op 310920938.00 ns/op 1.15 🔴
MergeQPRs_ReusingQPR-4 e1ed05 2a78f2
3722025.00 ns/op 4328433.00 ns/op 1.16 🔴
MergeSource-4 e1ed05 2a78f2
1649170920.00 ns/op 1994736248.00 ns/op 1.21 🔴
NAnd/size=1000-4 e1ed05 2a78f2
4.68 ns/op 6.01 ns/op 1.29 🔴
NAnd/size=10000-4 e1ed05 2a78f2
4.70 ns/op 5.94 ns/op 1.27 🔴
NAnd/size=1000000-4 e1ed05 2a78f2
4.79 ns/op 5.92 ns/op 1.24 🔴
Not/size=1000-4 e1ed05 2a78f2
5.03 ns/op 6.10 ns/op 1.21 🔴
Not/size=10000-4 e1ed05 2a78f2
5.04 ns/op 6.31 ns/op 1.25 🔴
Not/size=1000000-4 e1ed05 2a78f2
5.21 ns/op 6.23 ns/op 1.19 🔴
NotEmpty/size=1000-4 e1ed05 2a78f2
5.05 ns/op 6.42 ns/op 1.27 🔴
NotEmpty/size=10000-4 e1ed05 2a78f2
5.05 ns/op 6.31 ns/op 1.25 🔴
NotEmpty/size=1000000-4 e1ed05 2a78f2
5.04 ns/op 6.00 ns/op 1.19 🔴
Or/size=1000-4 e1ed05 2a78f2
4.64 ns/op 5.69 ns/op 1.23 🔴
Or/size=10000-4 e1ed05 2a78f2
4.67 ns/op 5.90 ns/op 1.26 🔴
Or/size=1000000-4 e1ed05 2a78f2
4.71 ns/op 6.14 ns/op 1.30 🔴
OrTree/size=1000-4 e1ed05 2a78f2
4.69 ns/op 5.79 ns/op 1.24 🔴
OrTree/size=10000-4 e1ed05 2a78f2
4.68 ns/op 5.64 ns/op 1.20 🔴
OrTree/size=1000000-4 e1ed05 2a78f2
5.45 ns/op 6.62 ns/op 1.21 🔴
OrTreeNextGeq/size=1000-4 e1ed05 2a78f2
4.68 ns/op 5.85 ns/op 1.25 🔴
OrTreeNextGeq/size=10000-4 e1ed05 2a78f2
4.68 ns/op 5.91 ns/op 1.26 🔴
OrTreeNextGeq/size=1000000-4 e1ed05 2a78f2
5.26 ns/op 6.45 ns/op 1.22 🔴
ParseESTime/es_stdlib-4 e1ed05 2a78f2
198.70 ns/op 250.00 ns/op 1.26 🔴
ParseESTime/handwritten-4 e1ed05 2a78f2
44.03 ns/op 53.30 ns/op 1.21 🔴
ParseESTime/rfc3339-4 e1ed05 2a78f2
57.38 ns/op 69.13 ns/op 1.20 🔴
ProcessDocuments-4 e1ed05 2a78f2
13324.00 B/op 16378.00 B/op 1.23 🔴
92.83 MB/s 73.19 MB/s 0.79 🔴
4444012.00 ns/op 5629274.00 ns/op 1.27 🔴
Sealing_NoSort-4 e1ed05 2a78f2
1194959516.00 ns/op 1441392037.00 ns/op 1.21 🔴
Sealing_WithSort-4 e1ed05 2a78f2
2294962601.00 ns/op 2843454000.00 ns/op 1.24 🔴
SeqListAppend-4 e1ed05 2a78f2
116.04 MB/s 101.86 MB/s 0.88 🔴
137913962.00 ns/op 157076865.00 ns/op 1.14 🔴
SeqQLParsing-4 e1ed05 2a78f2
1266.00 ns/op 1552.00 ns/op 1.23 🔴
SeqQLParsingLong-4 e1ed05 2a78f2
10973.00 ns/op 13594.00 ns/op 1.24 🔴

@cheb0
cheb0 enabled auto-merge (squash) July 28, 2026 07:09
@cheb0
cheb0 merged commit 6372ee9 into main Jul 28, 2026
7 of 8 checks passed
@cheb0
cheb0 deleted the 0-doc-freqs branch July 28, 2026 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants