Mission/open goal Description
Built a semantic theme-discovery and retrieval-quality audit over Zava's
multilingual customer feedback (English, Spanish, and French reviews and
support chats). The goal was to use precomputed embeddings and the
find_similar_docs_by_doc_id vector tool to surface semantically related
feedback across languages, then audit how trustworthy those matches
actually are — producing an analysis grounded in real documents rather
than a polished demo.
Harness and model
GitHub Copilot Chat (Agent mode) in VS Code Codespaces, powered by GPT-5.1
Turn-by-turn journey
-
Prompt: Explore the SQL database to find tables related to customer reviews and support chats.
Agent response or action: Ran describe_entities to list all entities, then read_records to sample rows from SupportChat, SupportTicket, and Doc tables.
Result: Identified Docs as the main feedback corpus containing reviews and support-chat text with precomputed embeddings.
-
Prompt: Pick diverse seed documents and run the vector tool to find similar documents.
Agent response or action: Selected 6 seeds (reviews + support chats, spanning English/Spanish/French) and ran find_similar_docs_by_doc_id for each.
Result: Returned top-5 nearest neighbors with cosine distances; smart-fabric connectivity issues matched correctly across all three languages.
-
Prompt: Label each neighbor as true match / near miss / false positive and build a precision@k audit notebook.
Agent response or action: Created zava_semantic_retrieval_audit.ipynb, labeled all 30 neighbor pairs, and computed precision@k per seed.
Result: Overall precision@k of 0.89 across 6 seeds, with a summary noting the 83-document corpus is too small to support broad claims.
Completion
Bonus work
Computed an honest precision@k metric (0.89 average) and explicitly
documented the limitations of the 83-document corpus rather than
overclaiming retrieval quality — included a model-card style note on
what the system is good for and where it falls short.
Mission/open goal Description
Built a semantic theme-discovery and retrieval-quality audit over Zava's
multilingual customer feedback (English, Spanish, and French reviews and
support chats). The goal was to use precomputed embeddings and the
find_similar_docs_by_doc_id vector tool to surface semantically related
feedback across languages, then audit how trustworthy those matches
actually are — producing an analysis grounded in real documents rather
than a polished demo.
Harness and model
GitHub Copilot Chat (Agent mode) in VS Code Codespaces, powered by GPT-5.1
Turn-by-turn journey
Prompt: Explore the SQL database to find tables related to customer reviews and support chats.
Agent response or action: Ran describe_entities to list all entities, then read_records to sample rows from SupportChat, SupportTicket, and Doc tables.
Result: Identified Docs as the main feedback corpus containing reviews and support-chat text with precomputed embeddings.
Prompt: Pick diverse seed documents and run the vector tool to find similar documents.
Agent response or action: Selected 6 seeds (reviews + support chats, spanning English/Spanish/French) and ran find_similar_docs_by_doc_id for each.
Result: Returned top-5 nearest neighbors with cosine distances; smart-fabric connectivity issues matched correctly across all three languages.
Prompt: Label each neighbor as true match / near miss / false positive and build a precision@k audit notebook.
Agent response or action: Created zava_semantic_retrieval_audit.ipynb, labeled all 30 neighbor pairs, and computed precision@k per seed.
Result: Overall precision@k of 0.89 across 6 seeds, with a summary noting the 83-document corpus is too small to support broad claims.
Completion
Bonus work
Computed an honest precision@k metric (0.89 average) and explicitly
documented the limitations of the 83-document corpus rather than
overclaiming retrieval quality — included a model-card style note on
what the system is good for and where it falls short.