Is your enhancement related to a problem? Please describe
On every run the tool queries ClearlyDefined for all dependencies, regardless of whether they
were already resolved in a previous run. On a project like eclipse-che/che-dashboard (~1 421
total dependencies) this means ~1 421 API calls on every invocation — even when yarn.lock has
not changed and all licenses are already recorded in .deps/prod.md / .deps/dev.md.
At the same time, three root causes prevent previously queried results from being reused even
when they exist in the output files:
Root cause A — uncategorized yarn3 deps never written to output files.
The yarn3 lockfile parser produces prod, dev, and all arrays. Deps that appear in all
but in neither prod nor dev (graph-traversal gaps for optional/transitive packages) are
approved by the API but never written to prod.md/dev.md, so they are never cached and are
re-queried on every run.
Root cause B — transitive deps only written to EXCLUDED with --harvest.
Without --harvest, transitive unresolved deps are printed to the console but not persisted to
EXCLUDED/. On the next run they are re-queried from scratch.
Root cause C — loadResolvedCache does not read EXCLUDED files.
Even after Root cause B is fixed, transitive deps stored in EXCLUDED/prod.md /
EXCLUDED/dev.md are still re-queried because the cache loader only reads prod.md and
dev.md.
Describe the solution you'd like
Introduce a resolved-dependency cache that is read at the start of each run. Before querying
ClearlyDefined, load .deps/prod.md, .deps/dev.md, EXCLUDED/prod.md, and EXCLUDED/dev.md
into a Map<identifier, {license, cq}>. Any dependency whose Resolved CQs column is
non-empty is skipped — only truly new or previously unresolved deps are sent to the API.
Result on eclipse-che/che-dashboard:
|
API calls |
| Before |
~1 421 (all deps on every run) |
| After |
~93 (1 328 served from cache) |
--recheck bypasses the cache and re-queries everything.
Additional improvements included in the same change:
- Enriched cache-miss log — breaks down misses into
prod, dev, prod+dev buckets
and distinguishes version-bumped packages from genuinely new ones.
- Configurable POST/GET timeouts —
--post-timeout (default 30 000 ms) and
--get-timeout (default 5 000 ms) with one automatic POST retry before falling back to
individual GETs.
- Harvest scoped to direct-unresolved deps only — previously harvest was triggered for
every notfound result, including transitive deps that belong to EXCLUDED/. Harvest now
runs inside processAndGenerateDocuments() and is called only with the identifiers that end
up in problems.md.
- EXCLUDED auto-cleanup — on every run, EXCLUDED entries whose CQ is a
[clearlydefined]
link are verified against the API; approved entries are removed from the EXCLUDED file in the
same run. With --harvest, all entries including orphans and transitive dependency rows are
re-checked via the new recheckAndCleanExcluded() function.
Bugs found and fixed during implementation
Bug 1 — Generated .deps files contain hyperlinks from npm registry metadata
extractLicenseInfo in npm/bump-deps.ts and yarn3/bump-deps.ts read homepage and
repository.url from node_modules/<pkg>/package.json and stored the result in LicenseInfo.URL.
arrayToDocument then wrapped every package name that had a URL in a markdown hyperlink:
| [`qs@6.15.2`](https://github.com/ljharb/qs) | BSD-3-Clause | transitive dependency |
| [`axios@1.16.1`](https://axios-http.com) | MIT | #28196 |
These npm registry URLs are not documentation links and should not appear in the output.
Fix: removed URL from LicenseInfo, removed homepage/repository.url extraction from
all three package managers (npm, yarn, yarn3), and removed the hyperlink-rendering branch from
arrayToDocument. Package names are now always plain backtick code.
Bug 2 — EXCLUDED packages get a license sourced from node_modules
Same extractLicenseInfo path read license from node_modules/<pkg>/package.json even for
packages that are in EXCLUDED/. The value was correct by coincidence (e.g. MIT for axios)
but originated from the wrong source and would silently diverge if the node_modules field
differed from the ClearlyDefined record.
Fix: same as Bug 1 — extractLicenseInfo no longer reads any fields beyond what is needed
for the license file.
Bug 3 — removeUnusedExcludes skipped linked-name EXCLUDED rows
PackageManagerUtils.removeUnusedExcludes() matched only plain `pkg@version` rows. Linked-name rows
[`pkg@version`](url) were skipped, so entries logged as "removed" after approval stayed in
the EXCLUDED file.
Fix: updated removeUnusedExcludes regex to match both formats, and normalized both the row
identifier and the identifiersToRemove set through coordinateToIdentifier() so Yarn Berry
and ClearlyDefined coordinates compare equal to their plain name@version equivalents.
Bug 4 — loadResolvedCache stored raw identifiers as cache keys
The cache loader stored the literal backtick value as the key. A row stored as a ClearlyDefined
coordinate (npm/npmjs/-/name/version) or Yarn Berry format (name@npm:version) never matched
processor lookups that use plain name@version, causing a cache miss on every run for those
deps.
Fix: applied coordinateToIdentifier() normalization at both parse sites — 3-column
prod.md/dev.md rows and 2-column EXCLUDED rows.
Bug 5 — Stale transitive dependency EXCLUDED entries survived when a package became direct
A package previously written to EXCLUDED as transitive dependency was loaded into depsToCQ
on subsequent runs. When that package later appeared in dependencies/devDependencies, the
stillUnresolved filter skipped it (already in depsToCQ) — it was never re-queried, never
added to problems.md, and never harvested.
Fix: added an eviction pass before the stillUnresolved computation. Packages cached as
transitive dependency that are now direct are deleted from depsToCQ and their EXCLUDED rows
are removed via removeUnusedExcludes, so they flow through the normal unresolved path.
Steps to reproduce (cache miss issue)
- Run the tool against a project with a populated
.deps/prod.md.
- Run again without changing
yarn.lock.
- Observe that the same ~1 421 API calls are made both times (same "querying N dependencies"
count in the logs).
Expected behavior
On the second run (no lockfile change) the log shows:
Cache miss: 0 dependencies to query (0 version bumps, 0 new packages).
and the tool finishes without making any ClearlyDefined API calls.
Additional context
- Fixes redundant ClearlyDefined API calls.
- Prevents harvest requests being sent for transitive dependencies.
- Fixes batch POST timeout that caused consistent failures with
--harvest.
- Fixes npm audit vulnerability (qs DoS, GHSA-q8mj-m7cp-5q26).
- Auto-removes stale EXCLUDED entries approved by ClearlyDefined.
Is your enhancement related to a problem? Please describe
On every run the tool queries ClearlyDefined for all dependencies, regardless of whether they
were already resolved in a previous run. On a project like
eclipse-che/che-dashboard(~1 421total dependencies) this means ~1 421 API calls on every invocation — even when
yarn.lockhasnot changed and all licenses are already recorded in
.deps/prod.md/.deps/dev.md.At the same time, three root causes prevent previously queried results from being reused even
when they exist in the output files:
Root cause A — uncategorized yarn3 deps never written to output files.
The yarn3 lockfile parser produces
prod,dev, andallarrays. Deps that appear inallbut in neither
prodnordev(graph-traversal gaps for optional/transitive packages) areapproved by the API but never written to
prod.md/dev.md, so they are never cached and arere-queried on every run.
Root cause B — transitive deps only written to EXCLUDED with
--harvest.Without
--harvest, transitive unresolved deps are printed to the console but not persisted toEXCLUDED/. On the next run they are re-queried from scratch.Root cause C —
loadResolvedCachedoes not read EXCLUDED files.Even after Root cause B is fixed, transitive deps stored in
EXCLUDED/prod.md/EXCLUDED/dev.mdare still re-queried because the cache loader only readsprod.mdanddev.md.Describe the solution you'd like
Introduce a resolved-dependency cache that is read at the start of each run. Before querying
ClearlyDefined, load
.deps/prod.md,.deps/dev.md,EXCLUDED/prod.md, andEXCLUDED/dev.mdinto a
Map<identifier, {license, cq}>. Any dependency whose Resolved CQs column isnon-empty is skipped — only truly new or previously unresolved deps are sent to the API.
Result on
eclipse-che/che-dashboard:--recheckbypasses the cache and re-queries everything.Additional improvements included in the same change:
prod,dev,prod+devbucketsand distinguishes version-bumped packages from genuinely new ones.
--post-timeout(default 30 000 ms) and--get-timeout(default 5 000 ms) with one automatic POST retry before falling back toindividual GETs.
every
notfoundresult, including transitive deps that belong toEXCLUDED/. Harvest nowruns inside
processAndGenerateDocuments()and is called only with the identifiers that endup in
problems.md.[clearlydefined]link are verified against the API; approved entries are removed from the EXCLUDED file in the
same run. With
--harvest, all entries including orphans andtransitive dependencyrows arere-checked via the new
recheckAndCleanExcluded()function.Bugs found and fixed during implementation
Bug 1 — Generated
.depsfiles contain hyperlinks from npm registry metadataextractLicenseInfoinnpm/bump-deps.tsandyarn3/bump-deps.tsreadhomepageandrepository.urlfromnode_modules/<pkg>/package.jsonand stored the result inLicenseInfo.URL.arrayToDocumentthen wrapped every package name that had a URL in a markdown hyperlink:These npm registry URLs are not documentation links and should not appear in the output.
Fix: removed
URLfromLicenseInfo, removedhomepage/repository.urlextraction fromall three package managers (npm, yarn, yarn3), and removed the hyperlink-rendering branch from
arrayToDocument. Package names are now always plain backtick code.Bug 2 — EXCLUDED packages get a license sourced from
node_modulesSame
extractLicenseInfopath readlicensefromnode_modules/<pkg>/package.jsoneven forpackages that are in
EXCLUDED/. The value was correct by coincidence (e.g.MITfor axios)but originated from the wrong source and would silently diverge if the
node_modulesfielddiffered from the ClearlyDefined record.
Fix: same as Bug 1 —
extractLicenseInfono longer reads any fields beyond what is neededfor the license file.
Bug 3 —
removeUnusedExcludesskipped linked-name EXCLUDED rowsPackageManagerUtils.removeUnusedExcludes()matched only plain`pkg@version`rows. Linked-name rows[`pkg@version`](url)were skipped, so entries logged as "removed" after approval stayed inthe EXCLUDED file.
Fix: updated
removeUnusedExcludesregex to match both formats, and normalized both the rowidentifier and the
identifiersToRemoveset throughcoordinateToIdentifier()so Yarn Berryand ClearlyDefined coordinates compare equal to their plain
name@versionequivalents.Bug 4 —
loadResolvedCachestored raw identifiers as cache keysThe cache loader stored the literal backtick value as the key. A row stored as a ClearlyDefined
coordinate (
npm/npmjs/-/name/version) or Yarn Berry format (name@npm:version) never matchedprocessor lookups that use plain
name@version, causing a cache miss on every run for thosedeps.
Fix: applied
coordinateToIdentifier()normalization at both parse sites — 3-columnprod.md/dev.mdrows and 2-column EXCLUDED rows.Bug 5 — Stale
transitive dependencyEXCLUDED entries survived when a package became directA package previously written to EXCLUDED as
transitive dependencywas loaded intodepsToCQon subsequent runs. When that package later appeared in
dependencies/devDependencies, thestillUnresolvedfilter skipped it (already indepsToCQ) — it was never re-queried, neveradded to
problems.md, and never harvested.Fix: added an eviction pass before the
stillUnresolvedcomputation. Packages cached astransitive dependencythat are now direct are deleted fromdepsToCQand their EXCLUDED rowsare removed via
removeUnusedExcludes, so they flow through the normal unresolved path.Steps to reproduce (cache miss issue)
.deps/prod.md.yarn.lock.count in the logs).
Expected behavior
On the second run (no lockfile change) the log shows:
and the tool finishes without making any ClearlyDefined API calls.
Additional context
--harvest.