feat: opt-in AI refinement of sample displayName and description#582
Open
Yimin-Jin wants to merge 7 commits into
Open
feat: opt-in AI refinement of sample displayName and description#582Yimin-Jin wants to merge 7 commits into
Yimin-Jin wants to merge 7 commits into
Conversation
Adds a 'refine_with_ai' workflow_dispatch input (default off). When off, behavior is unchanged: empty displayName is derived from the folder name and empty description is filled by the LLM, existing values untouched. When on, the LLM reviews both displayName and description of every template against its README and rewrites them only when they no longer fit, keeping values that already match. displayName now follows the Foundry Sample Finder convention (short human-friendly Title Case names). Shared LLM plumbing is extracted into callLLMForJson to avoid duplicating the request/parse logic.
The refine path fires one LLM request per template (~90 in a full run); a burst can trip the rate limiter at the sliding-window edge even with ample quota, which previously surfaced as many 429s and dropped fields. Wrap callLLMForJson's request in a retry loop that retries 429/5xx/network errors with jittered exponential backoff, honoring a server Retry-After (capped). Non-retryable 4xx and empty/invalid responses behave as before.
The first AI refine should regenerate every displayName/description without being anchored by the previous catalog's values (the keep-if-fits logic would otherwise preserve them). Adds an opt-in ignore_existing_catalog workflow input (default false) that skips mergeExistingDisplayFields via the IGNORE_EXISTING env, so all fields start empty and are fully regenerated. Normal runs are unchanged and still preserve PM-curated values.
…ne-display-fields
After merging template/dev (PR #580 added fetch-layer retry with delay/parseRetryAfterMs/computeBackoffMs), the LLM retry path's local sleep/retryAfterMsFromResponse duplicated those helpers. Drop the duplicates and reuse the shared delay/parseRetryAfterMs.
Two samples in the same language+framework+protocol group getting the same displayName (e.g. both '05-workflows' -> 'Multi-Agent Workflow') is confusing since the user sees them together. Add detection-only warnDuplicateDisplayNames that flags such same-group collisions in the CI step summary so a PM can disambiguate before merge. Cross-group duplicates are allowed (never seen side by side). The catalog is left unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
refine_with_aitoggle to the Sync Sample Catalog workflow so AI can review and improve bothdisplayNameanddescription— not just fill blanks. Default behavior is unchanged.Behavior
Toggle OFF (default): identical to today — an empty
displayNameis derived from the folder name, an emptydescriptionis generated by the LLM, and any value that already exists (PM-curated or preserved from the previous catalog) is left untouched.Toggle ON (
refine_with_ai: true): the LLM reviews every template's currentdisplayName+descriptionagainst its README and rewrites a field only when it no longer fits the sample's scenario — values that already match are kept verbatim (refinement is not a forced rewrite).displayNamefollows the Foundry Sample Finder convention: short, human-friendly Title Case scenario names (e.g. \Basic Agent, \Foundry Toolbox, \Azure Search RAG) instead of raw folder names.Changes
sync-sample-catalog.yml: newrefine_with_aiboolean input (defaultfalse), passed to the generator asAI_REFINE.generate_sample_catalog.mjs:AI_REFINEflag gating the two paths.callLLMForJsonhelper (removes duplicated request/parse/error handling).refineDisplayFieldsWithLLM(displayName + description, README-aware, keep-if-fits).autoFillDisplayFieldssplit intofillBlankDisplayFields(default) andrefineAllWithLLM(opt-in).Notes
sample-catalog.jsonis regenerated by the workflow and intentionally NOT included.Validation
node --checkpasses. Recommend a manualworkflow_dispatchrun (both toggle states) before merge.