Skip to content

ci(gqt): build the ordinary job's engine graph once - #896

Merged
azimafroozeh merged 2 commits into
ModernRelay:mainfrom
ragnorc:claude/gqt-ordinary-one-graph
Oct 8, 2026
Merged

azimafroozeh merged 2 commits into
ModernRelay:mainfrom
ragnorc:claude/gqt-ordinary-one-graph

Conversation

@ragnorc

@ragnorc ragnorc commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem

GQT (ordinary) has timed out on pull requests that change the engine since #894. Two runs this morning were cancelled at the 60-minute budget: #871 (run 37735019477) and bench/concurrent-merges (run 37733642954).

#894 added two cargo invocations to the job, and each selects a different package:

Step Selection Main, warm cache (run 37704808113) #871, cold
refusal -p omnigraph-gqt 1.5 min (12 crates compiled) 27 min
core -p omnigraph-gqt-core 19.7 min (116 crates, tests ran 0.38 s) 20 min
parity -p omnigraph-bench 21.8 min (187 crates, test ran 5.6 s) cancelled at 11 min

Each selection resolves its own feature graph, so tokio, arrow, datafusion, lance and the engine are rebuilt in every step. The reason is that omnigraph-gqt enables the engine's test-util and depends on omnigraph-dst, and the other two crates do neither. That is the #755 mechanism. The cache rule in docs/dev/ci.md forbids it: a second cargo invocation in a job must not change the dependency graph, and no job may need its cache to fit its budget.

The cold runs were not bad luck that a rerun would avoid. rust-cache hashes every installed toolchain into the key, and that includes the runner image's preinstalled stable. Main's cache was saved from a runner with 1.99.0 (415b780c), while #871's runner had 1.98.1 (c4cc11d5). Which key a job computes therefore depends on which runner image it lands on.

Change

  • Both engine invocations of the ordinary job now select the same three packages, -p omnigraph-gqt -p omnigraph-gqt-core -p omnigraph-bench, so they resolve one feature graph and build it once.
  • The refusal step runs --test runner_dispatch. A new units step runs --lib for all three crates.
  • The units step now runs every benchmark unit test, about 20 s in Test Workspace. The fixture parity test is among them, so it still runs on cases-only diffs.
  • The seam guard step is unchanged; its crate declares no workspace crate.
  • docs/dev/ci.md describes the shared selection.

Evidence

Local run of the job's three steps from an empty target directory (RUSTFLAGS=""):

Step Result Time Crates compiled
--test runner_dispatch 10 passed 625 s 580, the whole graph
--lib gqt 35, gqt-core 176 and bench 322 passed (1 ignored) 39 s 17, all test-only dev-dependencies (assert_cmd, predicates, toml) plus the three crates' unit-test builds; no tokio, arrow, datafusion, lance or engine
seam guard 8 passed 2 s 0

With one graph, the cold cost is a single engine build. On CI that build took 27 minutes in #871's cold run, about half the 60-minute budget, which is what docs/dev/ci.md requires.

Checks: check-ci-cells.py (with --self-test), check-classify-copy.py, check-workflow-action-pins.py, check-merge-group-triggers.py --self-test, check-change-classes.py (with --self-test), check-storage-upgrade-ci.py --self-test, check-docs.py and typos.

Not in this change

  • The runner-image term in rust-cache keys means the toolchain pin does not keep caches warm across runner image updates, contrary to what docs/dev/ci.md says. Fixing that is separate.
  • No guard enforces the one-selection rule yet. This is the second time it has regressed.
  • Azurite Azure Integration breaks the same rule. Its run_cell calls select omnigraph-storage, omnigraph-azure-admission, the engine with failpoints, omnigraph-cluster, omnigraph-server and omnigraph-cli one at a time, and each selection resolves its own graph. That step took 89 minutes on main's push run and hit the 90-minute budget there and on two pull requests.
  • Storage Upgrade Compatibility runs --workspace and then three -p omnigraph-engine --features failpoints scopes, which build a third graph (20 minutes warm on main). It still fits its 90-minute budget, but docs/dev/ci.md counts only two builds for that job.

CI-only, so no changelog fragment.

GQT (ordinary) ran three cargo invocations that each selected a different
package (omnigraph-gqt, omnigraph-gqt-core, omnigraph-bench). Each
selection resolves a different feature graph, so tokio, arrow, datafusion,
lance and the engine built three times: about 20 minutes per extra graph
even with a warm cache, and past the 60-minute budget cold. Both engine
invocations now select the same three packages, so they share one graph;
the benchmark units, the fixture parity test among them, run in the units
step.

@ragnorc ragnorc left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommendation: approve, subject to the required CI checks. I found no blocking defect at e844aa150710db746ed44e8efa72ffd249dfbd0d. The inline suggestion is optional.

This PR helps the test job finish before its time limit. Previously, three commands selected different Rust packages. Cargo then rebuilt shared dependencies with different features. Both engine-related commands now select the same three packages. They share the expensive build, while separate targets select the refusal tests and library tests. The practical benefit is more time for tests when the cache is empty.

The contract here is reliable qualification of engine and corpus changes, including changes that touch only .gqt files. Hosted Linux runners must meet that contract with an empty cache. Increasing the timeout would leave the repeated work. This change removes its cause through Cargo's existing feature resolution. See the workflow and Cargo's feature rules.

Coverage remains explicit. The library command includes the benchmark fixture parity test. Both commands retain empty RUSTFLAGS. The DST refusal depends on tokio_unstable, not the engine's dst feature. The separate DST execution, DST lint, seam guard, and required-result aggregation remain in place.

The tradeoffs are acceptable:

  • Core and benchmark units now share GQT's dst, failpoints, and test-util engine features. This job no longer exercises their separate default-only engine build. The separate engine lint job still checks the engine without dst. Lint does not replace runtime coverage of that configuration.
  • Every ordinary run now executes all benchmark units, including runs for corpus-only changes. That adds test work while removing repeated dependency builds.
  • Separate commands provide separate timing and failure labels. They also require two package lists to stay aligned. The optional comment explains how one command can remove that requirement.

Liability decreases overall. The diff adds four net lines, but removes one invocation and two unnecessary engine build configurations. It adds no API, persisted state, storage rule, or runtime mechanism. Five similar changes should reuse the same package selection, rather than add five independently maintained build configurations. The remaining duplicated selection is the main maintenance cost.

Validation:

  • Reviewed the exact head in a clean isolated checkout. Checked the manifests, feature paths, refusal implementation, parity owner, and workflow conditions. The pinned Lance 11.0.0 feature declarations agree with the resolved aws and azure selection.
  • Generated Cargo build plans with the pinned toolchain and empty RUSTFLAGS. The new commands have identical recursive build-unit signatures for the engine, Lance, DataFusion, Arrow-array, and Tokio. The old commands differ. This holds for the local target and an explicit Linux target. These are planning checks, not compiled test results.
  • Local seam guard: 8 passed. Workflow guards, their applicable self-tests, changed-file spelling, and documentation checks passed. Documentation checking covered 217 Markdown files.
  • Exact-head CI run: DST execution and DST lint passed. The ordinary job was still building at review time. I did not repeat the full engine build locally. The PR's cold-build timings remain author evidence, not an independently measured time bound.

Comment thread .github/workflows/gq-logic-tests.yml
@ragnorc

ragnorc commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

@azimafroozeh could you review this one? It is a CI-only change to the ordinary job #894 added steps to. Without it, GQT (ordinary) times out on any engine PR whose runner restores no cache. #871 has now timed out twice, both times in the benchmark-parity step.

On CI this run restored main's cache (full key match). The shared selection rebuilt 116 crates in 19 minutes, because main's cache holds the three per-crate graphs but not their union. The units step then took 46 seconds and the seam guard compiled one crate. Once main saves a cache with the union graph, a warm run rebuilds only the workspace crates a PR changes; a cold run is a single engine build.

@azimafroozeh azimafroozeh self-assigned this Oct 8, 2026
@azimafroozeh
azimafroozeh added this pull request to the merge queue Oct 8, 2026
Merged via the queue into ModernRelay:main with commit 2351e26 Oct 8, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants