Review Workshop eval trajectory changes - #477
AshishKumar4 wants to merge 1 commit into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Preview:
|
|
Author identity unknown *** Please tell me who you are. Run git config --global user.email "you@example.com" to set your account's default identity. fatal: empty ident name (for runner@runnervmlun5p.22vpi1papmiehlbrkehlsftqdc.bx.internal.cloudapp.net) not allowed |
|
@AshishKumar4 Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
|
Posted 1 actionable inline finding. |
567e19d to
251a515
Compare
|
Performance is inconclusive: all nine trials failed in both runs, with null pass-rate deltas. Candidate mean durations increased by 0.51–0.84 seconds across tasks. Both runs stopped before any model turn or tool call due to connection errors. These timing shifts likely reflect failure timing, and model variance cannot be assessed. |
251a515 to
9269040
Compare
|
LGTM! |
9269040 to
bf4ef34
Compare
|
Posted 1 actionable inline finding. |
bf4ef34 to
2296df5
Compare
|
LGTM! |
2296df5 to
6f4c82c
Compare
|
LGTM! |
6f4c82c to
273a43d
Compare
|
Posted 1 actionable inline finding. |
273a43d to
8185820
Compare
5167d7a to
85d5fca
Compare
85d5fca to
94418db
Compare
94418db to
fe102f8
Compare
|
LGTM! |
|
Performance stayed materially unchanged: expense-ledger remained 0/3 and project-doc 3/3, while appointment-desk’s 1/3 to 0/3 shift was non-comparable. Expense-ledger was 40.9s faster with fewer calls and slightly lower cost, but project-doc was 34.4s slower with slightly more calls and cost. Most notably, two candidate appointment trials made zero model turns, and the third stopped after two calls, unlike the baseline’s successful 18-call build. With only three trials plus connection and timeout failures, stochastic behavior could explain these shifts. |
fe102f8 to
a5a8edd
Compare
|
LGTM! |
a5a8edd to
225545d
Compare
|
LGTM! |
225545d to
4d3c8a9
Compare
|
[P1] The earlier trust-boundary finding remains unresolved at |
4d3c8a9 to
9987974
Compare
|
[P1] |
|
Performance stayed materially unchanged: expense-ledger remained 0/3 and project-doc 3/3, while appointment-desk’s 1/3 to 0/3 shift was non-comparable. Expense-ledger was 244.8s faster, with mean turns dropping 9.33 to 1.67, calls 15.67 to 2, and cost $0.488 to $0.102; project-doc slowed 6.4s with slight call and cost increases. Most notably, every candidate expense-ledger run timed out after only creating the gadget, whereas one baseline reached 18 turns and 33 calls with substantial implementation and verification. With three trials per cohort plus connection and timeout failures, stochastic behavior could explain these differences. |
9987974 to
89f593f
Compare
89f593f to
45f006b
Compare
|
Performance regressed: project-doc fell from 3/3 to 2/3, while both other cohorts remained 0/3. Project-doc averaged 12.3s faster with fewer turns, calls, and cost; appointment-desk was 148.2s faster but gained four mean tool errors. The failed candidate project-doc trial stopped after one turn and zero tool calls, unlike every baseline trial, which created and populated the document. With only three trials and anomalous stalls and malformed tool calls, stochastic behavior could explain the result. |
|
No additional actionable findings; the existing |
1 participant
Stacked on #476.
After the deterministic baseline/candidate comparison completes, download and pretty-print both raw trajectory artifacts, then run the existing Bonk action with a fixed 3–4 sentence prompt. The report covers direction, measured changes, the strongest trajectory-level explanation, and uncertainty from model variance.
Bonk starts from the trusted base checkout, reads a precomputed baseline-to-candidate diff, and receives a no-push token. OpenCode denies every tool except file reads, downloaded inputs are excluded from Git status, and the advisory step does not affect the deterministic eval result.