Skip to content

[dash-licenses] Cache ClearlyDefined license data to skip already-resolved dependencies #23896

Description

@olexii4

Is your enhancement related to a problem? Please describe

On every run the tool queries ClearlyDefined for all dependencies, regardless of whether they
were already resolved in a previous run. On a project like eclipse-che/che-dashboard (~1 421
total dependencies) this means ~1 421 API calls on every invocation — even when yarn.lock has
not changed and all licenses are already recorded in .deps/prod.md / .deps/dev.md.

At the same time, three root causes prevent previously queried results from being reused even
when they exist in the output files:

Root cause A — uncategorized yarn3 deps never written to output files.
The yarn3 lockfile parser produces prod, dev, and all arrays. Deps that appear in all
but in neither prod nor dev (graph-traversal gaps for optional/transitive packages) are
approved by the API but never written to prod.md/dev.md, so they are never cached and are
re-queried on every run.

Root cause B — transitive deps only written to EXCLUDED with --harvest.
Without --harvest, transitive unresolved deps are printed to the console but not persisted to
EXCLUDED/. On the next run they are re-queried from scratch.

Root cause C — loadResolvedCache does not read EXCLUDED files.
Even after Root cause B is fixed, transitive deps stored in EXCLUDED/prod.md /
EXCLUDED/dev.md are still re-queried because the cache loader only reads prod.md and
dev.md.

Describe the solution you'd like

Introduce a resolved-dependency cache that is read at the start of each run. Before querying
ClearlyDefined, load .deps/prod.md, .deps/dev.md, EXCLUDED/prod.md, and EXCLUDED/dev.md
into a Map<identifier, {license, cq}>. Any dependency whose Resolved CQs column is
non-empty is skipped — only truly new or previously unresolved deps are sent to the API.

Result on eclipse-che/che-dashboard:

API calls
Before ~1 421 (all deps on every run)
After ~93 (1 328 served from cache)

--recheck bypasses the cache and re-queries everything.

Additional improvements included in the same change:

  1. Enriched cache-miss log — breaks down misses into prod, dev, prod+dev buckets
    and distinguishes version-bumped packages from genuinely new ones.
  2. Configurable POST/GET timeouts--post-timeout (default 30 000 ms) and
    --get-timeout (default 5 000 ms) with one automatic POST retry before falling back to
    individual GETs.
  3. Harvest scoped to direct-unresolved deps only — previously harvest was triggered for
    every notfound result, including transitive deps that belong to EXCLUDED/. Harvest now
    runs inside processAndGenerateDocuments() and is called only with the identifiers that end
    up in problems.md.
  4. EXCLUDED auto-cleanup — on every run, EXCLUDED entries whose CQ is a [clearlydefined]
    link are verified against the API; approved entries are removed from the EXCLUDED file in the
    same run. With --harvest, all entries including orphans and transitive dependency rows are
    re-checked via the new recheckAndCleanExcluded() function.

Bugs found and fixed during implementation

Bug 1 — Generated .deps files contain hyperlinks from npm registry metadata

extractLicenseInfo in npm/bump-deps.ts and yarn3/bump-deps.ts read homepage and
repository.url from node_modules/<pkg>/package.json and stored the result in LicenseInfo.URL.
arrayToDocument then wrapped every package name that had a URL in a markdown hyperlink:

| [`qs@6.15.2`](https://github.com/ljharb/qs) | BSD-3-Clause | transitive dependency |
| [`axios@1.16.1`](https://axios-http.com)     | MIT           | #28196               |

These npm registry URLs are not documentation links and should not appear in the output.

Fix: removed URL from LicenseInfo, removed homepage/repository.url extraction from
all three package managers (npm, yarn, yarn3), and removed the hyperlink-rendering branch from
arrayToDocument. Package names are now always plain backtick code.

Bug 2 — EXCLUDED packages get a license sourced from node_modules

Same extractLicenseInfo path read license from node_modules/<pkg>/package.json even for
packages that are in EXCLUDED/. The value was correct by coincidence (e.g. MIT for axios)
but originated from the wrong source and would silently diverge if the node_modules field
differed from the ClearlyDefined record.

Fix: same as Bug 1 — extractLicenseInfo no longer reads any fields beyond what is needed
for the license file.

Bug 3 — removeUnusedExcludes skipped linked-name EXCLUDED rows

PackageManagerUtils.removeUnusedExcludes() matched only plain `pkg@version` rows. Linked-name rows
[`pkg@version`](url) were skipped, so entries logged as "removed" after approval stayed in
the EXCLUDED file.

Fix: updated removeUnusedExcludes regex to match both formats, and normalized both the row
identifier and the identifiersToRemove set through coordinateToIdentifier() so Yarn Berry
and ClearlyDefined coordinates compare equal to their plain name@version equivalents.

Bug 4 — loadResolvedCache stored raw identifiers as cache keys

The cache loader stored the literal backtick value as the key. A row stored as a ClearlyDefined
coordinate (npm/npmjs/-/name/version) or Yarn Berry format (name@npm:version) never matched
processor lookups that use plain name@version, causing a cache miss on every run for those
deps.

Fix: applied coordinateToIdentifier() normalization at both parse sites — 3-column
prod.md/dev.md rows and 2-column EXCLUDED rows.

Bug 5 — Stale transitive dependency EXCLUDED entries survived when a package became direct

A package previously written to EXCLUDED as transitive dependency was loaded into depsToCQ
on subsequent runs. When that package later appeared in dependencies/devDependencies, the
stillUnresolved filter skipped it (already in depsToCQ) — it was never re-queried, never
added to problems.md, and never harvested.

Fix: added an eviction pass before the stillUnresolved computation. Packages cached as
transitive dependency that are now direct are deleted from depsToCQ and their EXCLUDED rows
are removed via removeUnusedExcludes, so they flow through the normal unresolved path.


Steps to reproduce (cache miss issue)

  1. Run the tool against a project with a populated .deps/prod.md.
  2. Run again without changing yarn.lock.
  3. Observe that the same ~1 421 API calls are made both times (same "querying N dependencies"
    count in the logs).

Expected behavior

On the second run (no lockfile change) the log shows:

Cache miss: 0 dependencies to query (0 version bumps, 0 new packages).

and the tool finishes without making any ClearlyDefined API calls.

Additional context

  • Fixes redundant ClearlyDefined API calls.
  • Prevents harvest requests being sent for transitive dependencies.
  • Fixes batch POST timeout that caused consistent failures with --harvest.
  • Fixes npm audit vulnerability (qs DoS, GHSA-q8mj-m7cp-5q26).
  • Auto-removes stale EXCLUDED entries approved by ClearlyDefined.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind/enhancementA feature request - must adhere to the feature request template.severity/P2Has a minor but important impact to the usage or development of the system.team/AThis team is responsible for the Che Operator and all its operands as well as chectl and Hosted Che

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions