fix(wix-manage): stop the events recipes steering agents into draft events - #968
fix(wix-manage): stop the events recipes steering agents into draft events#968danshw wants to merge 3 commits into
Conversation
|
| id | name | tags |
|---|---|---|
80a3c855-f399-46ed-ac25-766572c2b6c0 |
events/ticket-tiers-and-publish |
events |
Left in place it keeps running in the nightly sweep, asserting a publish-a-draft flow that this PR deliberately steers agents away from — so it would fail permanently and read as a regression caused by this change.
It must be deleted from the EvalForge project once this lands. (Compare the still-orphaned draft:wix/skills#255 and #276 entries, which are what this looks like if it is forgotten.)
The renamed scenario events/ticket-tiers-and-cancel will be created by the normal sync and carries the events tag, so no promote step is needed.
Scenario split — each one now routes directly to the recipe it coversThe previous shape had the ticket scenario asserting coverage on the management recipe while the ticket shapes had moved to the creation recipe. That made the assertion depend on the agent hopping between files — the exact hop these traces show it does not make. Split:
Its correctness judge scores grounding, not state: pass if the reported number matches what a call actually returned. The site is seeded with two events, but a bootstrap that races provisioning would then fail visibly rather than turning an otherwise-correct run red. Verified live
Everything created was deleted and the app uninstalled. Updated cleanup on mergeStill one orphan to remove, now under its final name — |
14e198b to
6c98e3f
Compare
❌ EvalForge YAML Gate: Token Budget ExceededThese scenarios exceeded their configured top-level
|
Correction — the two cleanup comments above are wrongI wrote twice that the renamed scenario would orphan The gate already tracks renames and defers the delete to merge. No manual cleanup is needed. I inferred the orphan risk from the stale Also superseded in those comments:
The PR description is up to date; read it rather than the older comments. |
831c1f6 to
acbeaa1
Compare
…vents The two slowest scenarios in the nightly sweep are both events, and the traces show the recipes causing it. `events-multiple-ticket-tiers` (143s) copied `"draft": true` from the creation example, then spent six calls discovering that a draft event cannot have tickets added, be queried, be fetched by slug, or be published without WIX_EVENTS.READ_DRAFT_EVENTS — ending at "the solution is to create the event without the draft flag". The example set that trap, so it no longer sends `draft`, and the permission and its consequences are stated where the flag was. `events-recurring` (96s) read the creation recipe, which is where a "set up a weekly event" request lands, but recurring lived in the management recipe. It spent four SearchWixAPISpec calls rediscovering `individualEventDates` and the per-occurrence `timeZoneId` that the other file already documented, including the cryptic ZoneOffset error that omitting it produces. Both point the same way: a cross-reference is not coverage. An agent that reads one recipe and sees "see the other one" searches the spec instead of following the link. So the split now follows the task rather than the API surface — everything needed to create an event, including ticket tiers and recurring occurrences, is in the creation recipe; the management recipe keeps publish, cancel, delete, clone, update and count. Also notes that 24:00 is not a valid ISO time, which is what the recurring run's first 500 was. The ticket scenario moves from publishing to cancelling, since publishing a draft was the permission trap and cancelling now covers the management recipe that tickets no longer live in.
…t covers The ticket scenario asserted coverage on the management recipe while the ticket shapes moved to the creation recipe, so it depended on the agent making a second hop between files — the exact hop the traces show it does not make, preferring to search the spec. It now asserts the creation recipe, which is where a ticketed-event request lands and where the shapes now live. That leaves the management recipe needing coverage of its own, so counting becomes its scenario: "how many events do I have" routes there directly, needs no hop, and exercises what only that recipe carries — reading pagingMetadata.total off Query Events rather than count-by-status, whose empty facets for an empty body read as though the site has none. Its site is seeded with two events. The correctness judge scores the answer on whether it is grounded in what a call actually returned, not on the number 2, so a bootstrap that races provisioning fails visibly rather than turning a working run red.
The gate compares a scenario's maxTokens against the PR run's totalTokens, which sums prompt tokens across every step — the system prompt alone is ~34.5k on the first one. At 30000 the budget was unreachable for any agentic scenario; these three used 205k, 206k and 213k. 30000 is the value 22 other scenarios carry, and it holds there only because the gate checks the budget for scenarios a PR actually touches, so theirs have never been evaluated. 300000 leaves roughly 40% headroom over what these runs used, while still catching the kind of blow-up the budget exists to catch: on the same scenario, production spent 360k.
acbeaa1 to
69e5b4a
Compare
1 participant
The two slowest scenarios in the
aria-agent-2026-08-11sweep were both events, and the traces showed the recipes causing it. Both were introduced by #942.events-multiple-ticket-tiers403 READ_DRAFT_EVENTS× 4events-ticket-setup403 READ_DRAFT_EVENTS× 3events-recurring24:00, missing per-occurrencetimeZoneIdFor contrast, what the recipes already served well:
events-create-event23s 10/10,events-update-event36s 10/10.1. The creation example set a trap
It showed
"draft": true. The agent copied it, then spent six calls discovering that a draft event cannot have tickets added, be queried, be fetched by slug, or be published withoutWIX_EVENTS.READ_DRAFT_EVENTS— ending, in its own words, at "the solution is to create the event without thedraft: trueflag."The example no longer sends
draft, and the permission and its consequences are stated where the flag used to be.2. Recurring was in the file the agent doesn't open
events-recurringread the creation recipe — where "set up a weekly event" lands — but recurring lived in the management recipe. It spent fourSearchWixAPISpeccalls rediscoveringindividualEventDatesand the per-occurrencetimeZoneId, then concluded "Each Occurrence needs…timeZoneId" — the exact line in the file it never read.The general lesson
A cross-reference is not coverage. Step 3 of the ticket trace says it plainly:
— and then it searched the API spec rather than following the link. So the split now follows the task, not the API surface:
Both stay under the 10k serve limit (9,566 / 5,037), the constraint that forced the original split. Also documents that
24:00is not a valid ISO time — the recurring run's first 500.Measured by the gate
events/ticket-tiersRoughly half the time and 43% fewer tokens — the pairwise judge's own words: "both complete the task, but PR is at least 30% more efficient."
events/create-event-with-capacity-and-descriptionpassed 3/0 on both sides.Scenarios
Each now routes directly to the recipe it covers, with no hop:
events/create-event-with-capacity-and-descriptionevents/ticket-tiersevents/count-eventsevents/ticket-tiers-and-publishis renamed, because publishing a draft was the permission trap itself.maxTokensraised 30000 → 300000. The gate compares it against the run'stotalTokens, which sums prompt tokens across every step — the system prompt alone is ~34.5k on the first one, so 30000 was unreachable for any agentic scenario. These used 205k/206k/213k. It is the value 22 other scenarios carry, and it holds there only because the gate checks the budget for scenarios a PR actually touches.Known gap, not fixed here
events/count-eventspasses both judges 10/10 but fails its coverage assertion — counting is simple enough that the agent answers correctly without opening the manage recipe. A task that genuinely needs it (cloning, where the date resetting to ~14 days out is documented only there) would cover it properly. Worth doing, but it is a scenario redesign rather than part of this fix.Testing
Every flow executed against the live Wix Events API, all clean:
rsvp.limit+shortDescriptionUPCOMING, limit 40, description storedTICKETING,20.00/60.00→CANCELEDtimeZoneIdRECURRINGpagingMetadata.total= 2;count-by-status={}Everything created was deleted and the app uninstalled. Recurring occurrences are independent events — deleting the parent leaves them, which the recipe states and cleanup confirmed.