Skip to content

doc: RFC — auto roadmap, so the status stops being copied by hand - #157

Open
tzhouam wants to merge 1 commit into
mainfrom
doc/rfc-auto-roadmap
Open

tzhouam wants to merge 1 commit into
mainfrom
doc/rfc-auto-roadmap

Conversation

@tzhouam

@tzhouam tzhouam commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Enriches #152 ("[RFC] Auto Roadmap Generation") into a full RFC. Design only — nothing implemented.

What the measurements changed about the ask

#152 asks the agent to "auto dive the relationship of feature/rfcs/prs". Measured against vllm-project/vllm-omni over 2026-08-17 → 2026-09-11 (300 merged PRs, ~12/day), that reading does not survive:

Signal Flags Of the 8 curated in-window Verdict
PR cites the roadmap it belongs to 2 of 25 curated PRs 8% recall — the edge is not written down
PR body cites an RFC issue 22 of 300 (7%) too sparse
title keyword 40 of 300 (1.5/day) 6 15% precision
label world model 2 of 300 1 applied to 0.7% of traffic
label diffusion 87 of 300 6 6% precision

A human selected 8 of 300 merged PRs as roadmap-relevant. The taxonomy is judgment and was never written into the artifacts, so it cannot be recovered from them. What is derivable, completely and without a model, is status — and that is where the entire cost sits:

  • #7074 took 88 body edits in six days (24, 24, 26, 12 on the last four), all from one account.
  • #6494 tracks 129 PR/issue numbers, was last body-edited 2026-09-05, and five of its tracked PRs have merged since without the body changing.
  • 19 roadmap/tracking issues from 15 authors; 8 untouched for two weeks or more.

So the agent maintains and extracts; it never invents.

Design

  • Enrollment is consent — a marker the issue author places. No marker, no write, not even a comment. Eighteen of the nineteen roadmaps belong to other people.
  • The recurring lane writes a comment the bot solely authors, not the issue body. GitHub has no conditional update for an issue body, so read-check-write against a body edited 24×/day is a real TOCTOU race — it can destroy a maintainer's edit, and a splice from a cached body can restore an enrollment marker someone just deleted, defeating revocation.
  • Bootstrap is extraction under a quote gate — 19/19 roadmaps already carry headings or checklists, 12/19 a PR table. Every emitted feature, edge and attachment cites a verbatim span of the source body; a validator rejects anything that isn't there character-for-character.
  • Two lanes, two cadences — status every poll cycle with no model call; attachment proposals batched daily over ~1.5 prefiltered candidates (the keyword signal is a bad classifier but an 88%-recall prefilter).
  • The agent reports PR state; it never declares a feature done. #7074 holds E2 at partial by judgment although its PR merged.
  • Promotion in the management bot service, GitHub OAuth authorised on push permission; the HTTP surface takes a proposal id and a verdict, never roadmap content.
  • Ownership follows RFC-knowledge-intake.md with one named deviation: bootstrap runs as an IMC task kind (L2, confirm-gated) rather than SDK functions the bot orchestrates, because it is one-shot, judgment-heavy and writes into someone else's RFC.

Rollout is ordered so increment one carries 100% of the measured cost while needing no model, no bootstrap and no authentication — the increment that puts OAuth on a public host must not gate it.

Review notes

The first draft specified marker-scoped writes into the issue body guarded by a hash of the human-owned regions. The automated review gate rejected it correctly: GitHub exposes no If-Match for PATCH /issues/:n, so that guarantee cannot be delivered. §2 now moves the unattended lane off the body entirely, and the residual race on the two supervised body writes is recorded in Honest limits rather than claimed away.

Doc gates green: check_doc_links, check_doc_citations, check_spec_freshness.

Closes #152 only if the design is accepted as written; otherwise the issue stays open for the parts that change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MmTLpGtoZHBz81xj1dCo2r

vLLM-Omni runs 19 roadmap/tracking issues from 15 authors. One is current,
by hand: #7074 took 88 body edits in six days. #6494 tracks 129 numbers,
was last edited 2026-09-05, and five of its PRs have merged since.

Measured over 300 merged PRs (2026-08-17..2026-09-11, ~12/day), deriving
the roadmap from PR traffic fails: 2 of 25 curated PRs cite the roadmap
they belong to (8% recall), title keywords give 15% precision, and a human
selected 8 of 300. The taxonomy is judgment and is not recoverable from
the artifacts, because it was never written into them.

Status is. So the agent maintains and extracts, never invents: enrollment
by an author-placed marker, bootstrap gated on verbatim quotes of the
source body, status refreshed every poll cycle with no model call, and
attachment left as a proposal a maintainer accepts in the management bot
service.

The recurring lane writes a comment the bot solely authors, not the issue
body. GitHub offers no conditional update for an issue body, so a
read-check-write against a body edited 24 times a day is a real TOCTOU
race: it can destroy a maintainer's edit, and a splice from a cached body
can restore an enrollment marker someone just deleted, defeating
revocation. Only bootstrap and attachment promotion touch the body, both
human-triggered and spliced into a fresh read; the residual race is
recorded in Honest limits rather than claimed away.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MmTLpGtoZHBz81xj1dCo2r
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-12T16:45:20.832333Z fcd65c8 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fcd65c8dda

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread doc/RFC-auto-roadmap.md
Comment on lines +240 to +244
4. **The HTTP surface takes a proposal id and a verdict, never roadmap
content.** It can only apply proposals the agent already computed and
quote-validated, and every apply re-runs the quote gate before writing. A
session compromise buys an attacker "accept a proposal that was already on
offer" — not arbitrary writes into upstream issues with the bot's token.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Define a quote gate that can admit untracked PRs

When a lane-two candidate is absent from the current roadmap body—the normal reason to propose attaching it—§3's quote gate rejects the attachment because its supporting span is not present in that body. Re-running that same gate during apply therefore prevents useful proposals from ever being accepted. Define a proposal-specific gate whose allowed evidence includes the candidate PR while independently validating the destination feature against the roadmap.

Useful? React with 👍 / 👎.

Comment thread doc/RFC-auto-roadmap.md
Comment on lines +78 to +80
What the same signals are good for is **prefiltering**: the title keyword carries
88% recall (22 of the 25 curated PRs) while cutting the candidate set to 40 of
300. Useless as a classifier, exactly right as a gate in front of one.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Recompute recall from the in-window cohort

The table reports that the title keyword finds 6 of the 8 curated PRs in the measured window, which is 75% recall, not 88%. The 22/25 figure uses the broader curated set while the claimed reduction to 40/300 uses the in-window set, so combining them does not establish an 88%-recall prefilter for the daily workload; use one matched cohort or provide the corresponding candidate count for all 25 PRs.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[RFC] Auto Roadmap Generation

1 participant