Skip to content

fix: preserve edited default regex scans that require PCRE2 #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FanaticPythoner merged 2 commits into from
May 30, 2026
Merged

fix: preserve edited default regex scans that require PCRE2 #43

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7aad8a2
fix: preserve edited default regex scans that require PCRE2
FanaticPythoner May 30, 2026
19049ee
fix: avoid JS regex compilation for PCRE2-only tag scans
FanaticPythoner May 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,9 +392,9 @@ Normally, multiline support is enabled by detecting the use of `\n` in the regex
**better-todo-tree.regex.regex** (<tt>
&#x28;&#x2f;&#x2f;&#x7c;&#x23;&#x7c;&#x3c;&#x21;&#x2d;&#x2d;&#x7c;&#x3b;&#x7c;&#x2f;&#x5c;&#x5c;&#x2a;&#x7c;&#x5e;&#x7c;&#x5e;&#x5b;&#x20;&#x5c;&#x5c;&#x74;&#x5d;&#x2a;&#x28;&#x2d;&#x7c;&#x5c;&#x5c;&#x64;&#x2b;&#x2e;&#x29;&#x29;&#x5c;&#x5c;&#x73;&#x2a;&#x28;&#x24;&#x54;&#x41;&#x47;&#x53;&#x29;</tt>)</br>

This defines the regex used to locate TODOs. By default, it searches for tags in comments starting with <tt>&#47;&#47;</tt>, <tt>#</tt>, <tt>;</tt>, <tt>&lt;!--</tt> or <tt>&#47;*</tt>, and markdown todo lists. The <tt>($TAGS)</tt> placeholder is replaced by the expanded tag list. Prefix-only custom regexes that contain <tt>($TAGS)</tt> render text from the detected tag to the end of the physical line.
This defines the regex used to locate TODOs. By default, it searches for tags in comments starting with <tt>&#47;&#47;</tt>, <tt>#</tt>, <tt>;</tt>, <tt>&lt;!--</tt> or <tt>&#47;*</tt>, and markdown todo lists. The <tt>($TAGS)</tt> placeholder is replaced by the expanded tag list. Prefix-only custom regexes that contain <tt>($TAGS)</tt> render text from the detected tag to the end of the physical line. Workspace searches use PCRE2 automatically when look-around or backreference syntax requires it.

<sup>*Note: This is a [Rust regular expression](https://docs.rs/regex/1.0.0/regex)</a>, not javascript.*</sup>
<sup>*Regexes use ripgrep's Rust engine unless look-around, backreferences, or explicit ripgrep engine arguments select PCRE2.*</sup>

**better-todo-tree.regex.subTagRegex**
This is a regular expression for processing the text to the right of the tag, e.g. for extracting a sub tag, or removing unwanted characters. Anything that the regex matches will be removed from the text. If a capture group is included, the contents are extracted into a sub tag, which will be used in the tree to group similar tags. The sub tag can also be used as a placeholder in `better-todo-tree.tree.subTagClickUrl` and `better-todo-tree.tree.labelFormat`. Sub tags can also be highlighted by specifying a section in the `better-todo-tree.highlights.customHighlights` setting. To highlight the sub tag itself, set "type" to "tag-and-subTag" in custom highlights for the tag.
Expand All @@ -412,7 +412,7 @@ Set to false to allow tags to be matched regardless of case.
Empty value uses the packaged ripgrep binary. Set an absolute executable path to override it.

**better-todo-tree.ripgrep.ripgrepArgs** (`"--max-columns=1000"`)</br>
Use this to pass additional arguments to ripgrep. e.g. `"-i"` to make the search case insensitive. *Use with caution!*
Pass additional arguments to ripgrep. Engine arguments such as `--pcre2`, `-P`, and `--engine` override automatic engine selection.

**better-todo-tree.ripgrep.ripgrepMaxBuffer** (`200`)</br>
By default, the ripgrep process will have a buffer of 200KB. However, this is sometimes not enough for all the tags you might want to see. This setting can be used to increase the buffer size accordingly.
Expand Down
34 changes: 34 additions & 0 deletions artifacts/perf/issue42-regex-engine.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Issue 42 Regex Engine Benchmark

## Corpus

| Field | Value |
| --- | ---: |
| Files | 160 |
| Task lines per file | 8 |
| Expected matches | 1280 |
| Regex edit | Removed `|;` from `utils.DEFAULT_REGEX_SOURCE` |
| `rg` binary | `node_modules/@vscode/ripgrep-universal/bin/linux-x64/rg` |

## Results

| Strategy | Exit | Lookaround error | Matches | p50 ms | p95 ms | Throughput matches/s | Peak RSS MiB |
| --- | ---: | --- | ---: | ---: | ---: | ---: | ---: |
| Raw edited regex without PCRE2 | 2 | true | 0 | - | - | - | - |
| Raw edited regex with PCRE2 and workspace normalization | 0 | false | 1280 | 30.209 | 38.678 | 42370.9 | 94.12 |
| Candidate tag scan plus local normalization | 0 | false | 1280 | 22.259 | 27.029 | 57505.8 | 96.06 |

## Command

```bash
node --expose-gc scripts/perf/issue42-regex-engine.js
```

## Invariants

| Invariant | Value |
| --- | --- |
| Broken raw path fails before result parsing | `exit=2` |
| Fixed path returns all matches | `1280/1280` |
| Fixed path preserves rendered task text | Covered by `issue #42 PCRE2 markdown task payload keeps display text` |
| Candidate route avoids raw lookaround regex in `rg` | Covered by `issue #42 default-derived workspace regex uses candidate scan` |
106 changes: 106 additions & 0 deletions artifacts/perf/local-user-flow-issue42.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Runtime Benchmarks

- Baseline ref: `a6f60e0ce830c4649ac34fc05e5a1799ec91d151`
- Current source: working tree
- Node: `v25.2.0`
- Selection mode: `suite`
- Declared suite: `user-flow`
- Result-count validation: `12 rows, suite-consistent=true, all-user-flow=true`

## Machine Profile

| Category | Field | Value |
| --- | --- | --- |
| Host | Hostname | n00ne-AERO-17-YD |
| Host | OS | Ubuntu 22.04.5 LTS |
| Host | Kernel | 6.8.0-124-generic |
| Host | Architecture | x64 |
| Host | Load Average | 3.76, 3.59, 3.75 |
| Host | Available Parallelism | - |
| CPU | Model | Intel(R) Core(TM) i9-14900HX |
| CPU | Vendor | GenuineIntel |
| CPU | Topology | 16 logical CPU(s), 2 thread(s)/core, 8 core(s)/socket, 1 socket(s), 1 NUMA node(s) |
| CPU | Frequency | 800 MHz to 5,800 MHz |
| CPU | Cache | L1d 384 KiB (8 instances), L1i 256 KiB (8 instances), L2 16 MiB (8 instances), L3 36 MiB (1 instance) |
| Memory | Total RAM | 62.51 GiB (`67,119,767,552 bytes`) |
| Memory | Available At Collection | 15.95 GiB (`17,127,993,344 bytes`) |
| Memory | Online Physical RAM | 66.00 GiB (`70,866,960,384 bytes`) |
| Memory | Swap | total 120 GiB (`128,848,973,824 bytes`); free 106 GiB (`113,449,418,752 bytes`) |
| Memory | DMI / SPD | Unavailable: /sys/firmware/dmi/tables/smbios_entry_point: Permission denied /dev/mem: Permission denied |
| Storage | Root Device | nvme1n1 (Samsung SSD 9100 PRO 4TB), 3.64 TiB (`4,000,787,030,016 bytes`), transport nvme, rotational=false, readOnly=false |

## Scenario Model

| Scenario | Kind | User flow | Measurement scope | Input model |
| --- | --- | --- | --- | --- |
| open-file-default-save-rescan-visible-tree | user-flow | Save an already-open file that uses default tag scanning and redraw the visible tree. | Document save listener, document rescan, search-result replacement, and visible-tree render. | Real document text in a VS Code event harness. |
| open-file-custom-save-rescan-visible-tree | user-flow | Save an already-open file that uses custom regex scanning and redraw the visible tree. | Document save listener, custom-regex document rescan, search-result replacement, and visible-tree render. | Real document text in a VS Code event harness. |
| tree-view-cycle-visible-tree | user-flow | Cycle the tree between flat, tags-only, and tree views and redraw the visible tree each time. | View-mode commands, workspace-state mutation, and visible-tree rebuild/render. | Fixture workspace tree in a VS Code event harness. |
| tree-group-toggle-tags-view | user-flow | Toggle tag grouping on and off while in tags view and redraw the visible tree. | Grouping commands, workspace-state mutation, and visible-tree rebuild/render. | Fixture workspace tree in a VS Code event harness. |
| tree-filter-visible-tree | user-flow | Apply a tree filter and clear it again while the tree is visible. | Filter command handling, tree filtering, and visible-tree render. | Fixture workspace tree in a VS Code event harness. |
| tree-view-repeat-click-burst | user-flow | Repeatedly click the same view button while the tree state is still mutating. | Overlapping command handling and workspace-state writes. | Command burst against the extension command handlers in a VS Code event harness. |
| tree-expansion-toggle-visible-tree | user-flow | Expand and then collapse the visible tree. | Expansion commands, workspace-state mutation, and visible-tree rebuild/render. | Fixture workspace tree in a VS Code event harness. |
| workspace-default-relative-rebuild-visible-tree | user-flow | Trigger a workspace refresh with default tag scanning and rebuild the visible tree from workspace matches. | Workspace refresh orchestration, ripgrep event handling, file reads, result application, and tree rebuild/render. | Fixture ripgrep matches, fixture file contents, and fixture scan results in a VS Code event harness. |
| workspace-custom-relative-rebuild-visible-tree | user-flow | Trigger a workspace refresh with custom regex scanning and rebuild the visible tree from workspace matches. | Workspace refresh orchestration, ripgrep event handling, regex-match normalization, result application, and tree rebuild/render. | Fixture ripgrep matches, fixture file contents, and fixture normalized regex results in a VS Code event harness. |
| visible-editor-highlight-open-file | user-flow | Focus or open a visible editor and apply highlights to that editor. | Active-editor event handling, decoration creation/update, and highlight application. | Fixture scan results fed into the real highlight pipeline in a VS Code event harness. |
| visible-editor-highlight-change-open-file | user-flow | Edit a visible file and refresh its highlights. | Text-change event handling, decoration update, and highlight application. | Fixture scan results fed into the real highlight pipeline in a VS Code event harness. |
| visible-editor-custom-highlight-config-open-file | user-flow | Open a visible editor while a large custom highlight configuration is active and apply highlights. | Custom-highlight attribute lookup, decoration creation/update, and highlight application. | Fixture scan results plus a large custom-highlight config in a VS Code event harness. |

## Metric Model

| Table | Value model | Accuracy model |
| --- | --- | --- |
| Latency | Wall-clock elapsed time around each harness flow iteration, summarized as min/p50/p90/p95/max. | Exact for each sampled iteration in this run. |
| Profiled RSS Burst | Difference between the isolated scenario worker RSS measured immediately before the flow and that worker iteration's OS high-water-mark peak RSS. | Exact for the measured worker iteration, using `process.memoryUsage().rss` at flow start and `process.resourceUsage().maxRSS` for the peak. |
| Profiled Peak RSS | Highest process RSS reached by each isolated scenario worker iteration. | Exact worker-process high-water mark from `process.resourceUsage().maxRSS`. |

## Latency

| Scenario | Kind | Baseline p50 ms | Current p50 ms | Baseline p90 ms | Current p90 ms | Baseline p95 ms | Current p95 ms |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| open-file-default-save-rescan-visible-tree | user-flow | 106.37 | 2.15 | 108.62 | 2.37 | 116.03 | 2.4 |
| open-file-custom-save-rescan-visible-tree | user-flow | 93.1 | 3.18 | 94.28 | 3.39 | 97.26 | 3.48 |
| tree-view-cycle-visible-tree | user-flow | 9.85 | 0.38 | 11.41 | 0.5 | 11.46 | 0.51 |
| tree-group-toggle-tags-view | user-flow | 7.62 | 0.31 | 9.09 | 0.42 | 10.85 | 0.46 |
| tree-filter-visible-tree | user-flow | 0.42 | 0.33 | 0.53 | 0.45 | 0.6 | 0.49 |
| tree-view-repeat-click-burst | user-flow | 0.2 | 0.25 | 0.26 | 0.3 | 0.26 | 0.32 |
| tree-expansion-toggle-visible-tree | user-flow | 8.02 | 0.44 | 8.48 | 0.52 | 11.3 | 0.54 |
| workspace-default-relative-rebuild-visible-tree | user-flow | 7.36 | 1.77 | 9.19 | 2.6 | 9.36 | 2.71 |
| workspace-custom-relative-rebuild-visible-tree | user-flow | 48.07 | 4.83 | 50.77 | 5.58 | 51.2 | 5.6 |
| visible-editor-highlight-open-file | user-flow | 19.98 | 1.8 | 22.55 | 2.56 | 23.45 | 2.73 |
| visible-editor-highlight-change-open-file | user-flow | 20.4 | 1.61 | 21.04 | 1.87 | 22.2 | 1.97 |
| visible-editor-custom-highlight-config-open-file | user-flow | 1866.49 | 2.56 | 1901.63 | 3.75 | 2124.97 | 4.03 |

## Profiled RSS Burst

| Scenario | Kind | Baseline p50 MiB | Current p50 MiB | Baseline p90 MiB | Current p90 MiB | Baseline p95 MiB | Current p95 MiB | Baseline Max MiB | Current Max MiB |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| open-file-default-save-rescan-visible-tree | user-flow | 26.25 | 24.25 | 27 | 24.63 | 27.13 | 24.88 | 27.13 | 24.88 |
| open-file-custom-save-rescan-visible-tree | user-flow | 26.75 | 21 | 27.63 | 21.38 | 27.75 | 21.75 | 27.75 | 21.75 |
| tree-view-cycle-visible-tree | user-flow | 17 | 9.13 | 17.88 | 9.38 | 17.88 | 9.63 | 17.88 | 9.63 |
| tree-group-toggle-tags-view | user-flow | 16.38 | 9.13 | 16.88 | 9.5 | 17.25 | 9.5 | 17.25 | 9.5 |
| tree-filter-visible-tree | user-flow | 18.5 | 11.75 | 19 | 12.13 | 19.5 | 12.13 | 19.5 | 12.13 |
| tree-view-repeat-click-burst | user-flow | 1.25 | 1.63 | 1.38 | 1.75 | 1.5 | 1.75 | 1.5 | 1.75 |
| tree-expansion-toggle-visible-tree | user-flow | 16.75 | 9.13 | 18 | 9.25 | 18.25 | 9.63 | 18.25 | 9.63 |
| workspace-default-relative-rebuild-visible-tree | user-flow | 17.13 | 9.63 | 17.38 | 9.88 | 17.38 | 10.13 | 17.38 | 10.13 |
| workspace-custom-relative-rebuild-visible-tree | user-flow | 71.5 | 13.88 | 74.25 | 14.13 | 75.63 | 14.13 | 75.63 | 14.13 |
| visible-editor-highlight-open-file | user-flow | 48.63 | 11.88 | 49.38 | 13.13 | 49.88 | 13.25 | 49.88 | 13.25 |
| visible-editor-highlight-change-open-file | user-flow | 47.73 | 11.75 | 49.5 | 12 | 49.88 | 12.13 | 49.88 | 12.13 |
| visible-editor-custom-highlight-config-open-file | user-flow | 159.88 | 14.25 | 162.5 | 14.49 | 163.63 | 14.63 | 163.63 | 14.63 |

## Profiled Peak RSS

| Scenario | Kind | Baseline p50 RSS MiB | Current p50 RSS MiB | Baseline p90 RSS MiB | Current p90 RSS MiB | Baseline p95 RSS MiB | Current p95 RSS MiB | Baseline Max RSS MiB | Current Max RSS MiB |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| open-file-default-save-rescan-visible-tree | user-flow | 81.83 | 79.71 | 82.42 | 80.09 | 82.43 | 80.34 | 82.43 | 80.34 |
| open-file-custom-save-rescan-visible-tree | user-flow | 82.2 | 76.46 | 82.96 | 76.9 | 83.2 | 76.94 | 83.2 | 76.94 |
| tree-view-cycle-visible-tree | user-flow | 72.55 | 64.64 | 73.09 | 65.07 | 73.39 | 65.23 | 73.39 | 65.23 |
| tree-group-toggle-tags-view | user-flow | 71.89 | 64.42 | 72.26 | 64.91 | 72.29 | 65.04 | 72.29 | 65.04 |
| tree-filter-visible-tree | user-flow | 73.81 | 67.27 | 74.63 | 67.42 | 74.88 | 67.46 | 74.88 | 67.46 |
| tree-view-repeat-click-burst | user-flow | 56.72 | 57.03 | 56.81 | 57.46 | 57.16 | 57.54 | 57.16 | 57.54 |
| tree-expansion-toggle-visible-tree | user-flow | 72.12 | 64.68 | 73.64 | 64.89 | 73.68 | 64.94 | 73.68 | 64.94 |
| workspace-default-relative-rebuild-visible-tree | user-flow | 72.51 | 65.09 | 72.97 | 65.21 | 73.01 | 65.3 | 73.01 | 65.3 |
| workspace-custom-relative-rebuild-visible-tree | user-flow | 126.91 | 69.36 | 129.91 | 69.56 | 131.12 | 69.6 | 131.12 | 69.6 |
| visible-editor-highlight-open-file | user-flow | 104.12 | 67.45 | 104.94 | 68.59 | 105.26 | 68.66 | 105.26 | 68.66 |
| visible-editor-highlight-change-open-file | user-flow | 103.36 | 67.13 | 104.87 | 67.45 | 105.2 | 67.89 | 105.2 | 67.89 |
| visible-editor-custom-highlight-config-open-file | user-flow | 215.43 | 69.78 | 218.03 | 69.93 | 219.25 | 70.08 | 219.25 | 70.08 |
Loading
Loading