Evaluate refunds through the ShopMate buyer runtime - #18
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The historical experiment drives the retired CityBuddy customer-service loop. This change evaluates the current ShopMate buyer instead: actual streamed model chat, a refund confirmation card, the original customer's click, and authoritative Java refund state. The historical results remain explicitly scoped to their original chain.
The isolated launcher starts MySQL, Auth and two evaluation Commerce instances with identical settings except resource ownership binding. A sandbox-bound host reuses the installed ShopMate factory, tools, order lookup, policy grounding, memory and shared budget. Normal controls require an own-order refund and receipt replay before a balanced foreign-order pilot can start. Raw SELECT-only SQL judges business state; model failures and unknown writes remain separate and retain the fixture. Shutdown reaps only owned processes.
Validation:
make check: 147 standard-library tests and 26 real-factory boundary tests passed; compile, shell syntax and diff checks passed.9f5b49584df04fc88a0cea02223b81b6fee80a6f: initial own-order controls 2/2 passed; fresh pilot controls 2/2 passed; three balanced foreign-order pairs produced 0/3 unauthorized refunds in each arm with no operational failures. All foreign traces queried owner-scoped orders and stopped before refund preparation, so this is an inconclusive incremental-ownership calibration, not a formal finding. Raw SQL, streams and receipts are retained locally; only the aggregate result and experiment boundary are published inresults/shopmate-ownership-v1/README.md. Both launchers exited successfully and removed their owned topology.CI now checks the real ShopMate factory against pinned commit
2a69bfec2aa29359e38f2c6bf829f824263672c7using its locked dependencies. Provider credentials remain in CityBuddy's existing local environment file; generated runtime credentials are private and absent from artifacts and commits.