Principal Data Analyst Mission: Investigating Zava's Hidden Product-Quality Crisis - From Category-Level Risk to SKU-Level Evidence

[Wasimrajaanwar]

Mission/open goal Description

I chose the Principal Data Analyst mission to investigate whether Zava had an underlying product-quality issue that wasn't immediately visible from sales data alone. Rather than assuming where the problem existed, I approached the investigation in stages—starting with category-level sales and support metrics to identify the highest-risk area, then drilling into customer support conversations to understand recurring issues, and finally using vector similarity search to determine whether those complaints represented isolated incidents or a consistent pattern across multiple customers.
The objective was to combine quantitative measures such as sales performance, support burden, and customer satisfaction with qualitative evidence from support chats and customer reviews to identify the strongest evidence-backed product-quality risk and provide a practical recommendation for Zava's leadership team.

Harness and model

GitHub Copilot Chat (Agent Mode) using Auto model selection. During the analysis, Copilot automatically changed requests to GPT-5.5 and MAI-Code-1-Flash depending on the task.

Turn-by-turn journey

Turn 1:
Prompt: I'm investigating a hidden product-quality issue in the Promptathon database. Before we start analysing anything, first discover which SQL MCP tools and database entities are available to you. Please use only live SQL MCP tool calls and don't rely on the repository documentation. Once you've discovered the available tools and entities, tell me which ones are most relevant for investigating a product-quality issue that combines sales impact, support burden, customer satisfaction, and recurring customer complaints. Briefly explain how you plan to approach the investigation, but don't start analyzing the data or drawing conclusions yet. We'll build the evidence step by step.

Agent response or action:
The agent queried the live SQL MCP service to discover the available tools and database entities instead of relying on the repository. It identified the core operations (describe_entities, read_records, aggregate_records, and FindSimilarDocsByDocId) along with the nine entities in the Promptathon database. It then mapped each entity to its role in the investigation and outlined an evidence-first approach without attempting to analyse the data.

Result: The discovery phase completed cleanly with no errors or repository lookups. One thing that stood out was that the agent identified the document similarity capability as an important part of the investigation without being prompted explicitly, suggesting it understood the purpose of the available tools rather than simply matching keywords. It also proposed a five-step investigation that closely aligned with the approach I had planned, giving a solid foundation before moving into the data analysis.

Turn 2
Prompt: Let's start by identifying which product categories deserve a closer look. Aggregate SalesOrderLines by ProductCategory and report the order count, quantity sold, and total revenue for each category. Then use RelatedSKU to connect SupportTickets with Products so the tickets are grouped by the correct product category instead of the support issue category.

For each product category, show:
Order count
Quantity sold
Total revenue
Support ticket count
Average SatisfactionScore
Number of High and Critical priority tickets

Compare the sales performance with the support burden and identify the three product categories that look the most concerning. Explain why each one stands out and which category you think we should investigate next. Use only live SQL MCP tool calls and don't rely on the repository files.

Follow-up Prompt: Before we move on, confirm that Premium is the strongest candidate across the complete product catalog. Were any product categories excluded because they had no linked support tickets or insufficient data? Also, is comparing B2B directly with the consumer product categories appropriate, given the obvious differences in purchasing behaviour and revenue?

Agent response or action:
The agent initially attempted to retrieve the data through the SQL MCP entity layer but encountered entity-name resolution issues. Rather than stopping, it switched to a direct database connection and rebuilt the analysis using SQL queries against the underlying tables. During the first pass, the joins returned only a subset of product categories, so the agent validated the category mappings and reran the aggregation to ensure every category with valid ticket linkage was included. After the quantitative analysis, I challenged one of its assumptions by asking it to verify that Premium really was the strongest candidate and to explain whether B2B should be compared directly with the consumer categories. The agent confirmed that no product categories had been excluded and explained why B2B should be treated as a separate business line or normalized before comparison.

Result: The final results were internally consistent and matched an independent manual validation of the underlying data, increasing confidence in the findings. Premium emerged as the strongest B2C product-quality risk with 19 support tickets and an average SatisfactionScore of 2.79, while Elite recorded 14 tickets and a higher satisfaction score of 3.71. Although B2B generated the highest revenue, it had no linked support tickets and was ultimately treated as a contextual benchmark rather than a direct comparison because of its fundamentally different sales model. Based on the quantitative evidence, Premium was selected for deeper qualitative investigation in the next stage.

Turn 3
Prompt: Based on the previous analysis, Premium emerged as the strongest B2C product category for further investigation after comparing all categories on sales impact, support burden, and customer satisfaction. Now let's understand what's driving those results. Read the SupportTicket records linked to Premium products by joining RelatedSKU to the Product entity. Summarize the ticket-level findings rather than listing every record unless a specific ticket is needed as evidence. Include the TicketId, RelatedSKU, ProductName, Priority, Status, SatisfactionScore, and any available customer segment, sales channel, country, and service language information. Then read the corresponding SupportChat transcripts for those tickets. Parse the MessagesJson field and focus only on messages sent by the customer. Group similar complaints together to identify recurring themes instead of listing every individual message.
For each recurring complaint theme, provide:
The number of tickets where the theme appears. Two or three short customer excerpts that best represent the issue. Whether the complaints are concentrated around a specific SKU or spread across multiple Premium products. Also determine whether the complaints cluster around a particular sales channel, customer segment, country, or service language. If no meaningful pattern exists, state that explicitly rather than inferring one. Finally, summarize the three to five strongest complaint themes and conclude whether the evidence points to a single defective SKU, multiple affected SKUs, or a broader Premium category quality issue. Explain your reasoning using the evidence gathered. Use only live SQL MCP tool calls and do not rely on repository files.

Prompt 3 Follow up - after clarification from agent: Proceed with the consolidation. Summarize the findings using the complaint counts and representative customer excerpts. Based on the evidence, determine whether the issues are concentrated around a single SKU, multiple related SKUs, or represent a broader Premium category quality issue. Explain your reasoning. There's no need to include every TicketId unless a specific ticket is important as supporting evidence.

Agent response or action:
The agent retrieved the support tickets linked to Premium products by joining them through RelatedSKU to the Product table, then pulled the corresponding support chat transcripts. It parsed the MessagesJson field to isolate only customer messages before grouping similar conversations into recurring complaint themes instead of listing every ticket individually. After presenting the initial findings, I asked it to consolidate the results into a higher-level summary focused on complaint patterns, representative customer excerpts, and whether the issues clustered around particular SKUs, customer groups, sales channels, or regions.

Result: This was the point where the investigation started to move beyond numbers and reveal the underlying customer experience. Rather than presenting a long list of tickets, the agent consolidated the conversations into four clear complaint themes: shipping and order status (6 tickets), returns and sizing (4), connectivity and app issues (3), and billing or unexpected charges (2).
One of the most useful findings was that the complaints weren't limited to a single defective product. Although ZCPTM-SS-M-BW appeared most frequently and featured across multiple complaint types—including shipping, connectivity, and billing—it wasn't the only SKU involved. Several other Premium products also appeared throughout the support conversations, suggesting a broader issue affecting the Premium smart fabric product experience rather than a single manufacturing defect.
The analysis also highlighted an important distinction between operational issues and product-quality issues. Shipping and order-status enquiries accounted for the largest number of tickets, but they appeared to reflect fulfilment and post-purchase communication rather than defects in the product itself. By contrast, the connectivity and "not working as expected" complaints, while fewer in number, provided much stronger evidence of a genuine product or user-experience problem.
The complaints were spread across multiple countries, sales channels, and languages, with no meaningful geographic or channel clustering. Most came from customers in the New segment, although the available evidence wasn't sufficient to explain why that group was overrepresented. Taken together, the findings identified ZCPTM-SS-M-BW as the strongest candidate for deeper investigation while indicating that the wider Premium product line also deserved attention.

Turn 4.
Prompt: From the previous investigation, ZCPTM-SS-M-BW appears to be the Premium SKU showing up most often across different complaint types, especially the connectivity and "not working as expected" issues. I want to see whether those complaints are isolated cases or whether similar conversations and reviews exist elsewhere in the data. Search the Docs table for documents related to this SKU and the Premium category. Find the document that best represents the connectivity or product-performance issues we identified earlier, rather than routine shipping or billing enquiries. Show me the DocId, SourceType, Title, RelatedSKU, TagsJson, and a short excerpt from the document so I can see why it was chosen. Then use that document to run find_similar_docs_by_doc_id with TopN = 10. For each similar document, show the DocId, SourceType, RelatedSKU (if available), cosine distance, and a short excerpt highlighting the complaint.

Once you've done that, tell me whether these similar documents reinforce the same pattern we've seen so far. I'm interested in whether the complaints keep pointing to the same SKU, spread across several Premium SKUs, or cluster around a particular customer segment, sales channel, country, or language. If the documents are only textually similar but don't represent the same underlying issue, make that clear as well.

Use only live SQL MCP tool calls and don't rely on the repository files.

Agent response or action:
The agent searched the Docs table for documents related to ZCPTM-SS-M-BW and the Premium product line before selecting the review that best represented the connectivity problems identified in the previous step. It then used that document as the source for find_similar_docs_by_doc_id and retrieved the ten closest matches. For each result, it returned the document metadata, similarity score, and a short excerpt before comparing the documents to determine whether they described the same underlying issue or were simply textually similar.

Result: Up to this point, the investigation suggested that Premium products were generating more support activity, but it wasn't clear whether the complaints were all connected or simply a collection of unrelated issues. The vector similarity search answered that question.

Starting from a single negative review for ZCPTM-SS-M-BW, the search surfaced several highly similar reviews describing the same failure pattern: the smart-fabric features stopped working after the garment had been washed, causing the product to lose its connection with the Zava app. The closest matches consistently referenced the same SKU, while equivalent complaints also appeared in multiple languages, showing that customers in different markets were reporting the same experience.

The lower-ranked results told a different story. Although they were textually similar, many were general support conversations, washing-care questions, or complaints relating to other products rather than the same defect. The agent correctly distinguished these from the core evidence instead of treating every similar document as confirmation.

This was the strongest validation point in the investigation. It confirmed that the recurring connectivity failures weren't isolated customer experiences but a consistent pattern centred on ZCPTM-SS-M-BW, providing much stronger evidence than the support-ticket analysis alone.

Turn 5.
Prompt: We've now worked through the full investigation, so I'd like you to pull everything together into a concise executive brief for Zava's leadership team, so I'd like you to pull everything together into a concise executive brief for Zava's leadership team. Use the evidence we've gathered throughout this investigation instead of running new analysis unless something essential is missing.
Please structure the brief as follows:
1. Risk cluster

• Product or category
• Order type and sales channel
• Customer segment or B2B client pattern (if any)
  2. Evidence
  Include the key findings from the investigation:
• Revenue and quantity sold
• Support ticket count
• Average satisfaction score
• Priority and status breakdown
• Review and support document count - Vector similarity findings
  3. Complaint themes
  Summarise the main recurring issues and include 3–5 short customer excerpts that best represent them.
  4. Recommendation
  Based on all of the evidence, recommend the first action Zava should take and explain why.
  5. Confidence and limitations
  Explain how confident you are in the findings, what the evidence strongly supports, and where the available data leaves uncertainty. Keep the brief concise, evidence-based, and suitable for a leadership audience. Write the brief in plain, direct language suitable for a non-technical leadership audience — avoid SQL terminology, entity names, and technical jargon in the final output. If one finding clearly stands out as the highest-confidence product-quality issue, make that the primary focus of the executive brief and support it with the strongest evidence collected during the investigation. Use only the evidence collected during this investigation and don't rely on repository files.

Agent response or action:
The agent recognised that the investigation was complete and reused the evidence gathered throughout the previous steps instead of performing new analysis. It consolidated the quantitative findings, customer support evidence, and vector similarity results into a concise executive brief written for a non-technical leadership audience. Throughout the report, it clearly separated observations, interpretations, recommendations, and limitations while avoiding unsupported conclusions.

Result: The final output transformed the investigation into a leadership-ready recommendation. It identified the Premium product line—and particularly ZCPTM-SS-M-BW—as the highest-risk area, supported that conclusion with sales, support, customer, and document evidence, and highlighted wash-related connectivity failures as the recurring defect pattern. The brief concluded with a focused recommendation to investigate the SKU's wash-cycle durability and app connectivity while acknowledging that the available evidence identifies the problem but does not establish its underlying engineering or manufacturing cause.

Executive brief: Premium smart-fabric product issue
1. Risk cluster
Product or category: Premium smart-fabric apparel, with the clearest concentration on the ZCPTM-SS-M-BW short-sleeve top.
Order type and sales channel: This is primarily a consumer product issue rather than a B2B client issue. The complaints appear across ordinary customer purchase flows and do not point to a single channel as the main problem.
Customer segment pattern: The evidence points more to a product-quality issue than to a specific customer segment. The strongest signal is customer frustration with the product’s connected features, not a concentrated B2B or account-specific pattern.

2. Evidence
Revenue and quantity sold: The Premium category generated about $190,553.87 in revenue from 2,207 units across 893 orders.
Support ticket count: The Premium category generated 19 support tickets in the reviewed evidence.
Average satisfaction score: Average satisfaction for Premium was about 2.79 out of 5, which is materially weak for a premium-priced product.
Priority and status breakdown: The evidence captured 2 high-priority and 2 critical tickets. In the support conversations reviewed, most were marked as resolved, but at least one was escalated, suggesting the issue was real and persistent rather than a one-off.
Review and support document count: The strongest evidence came from 14 documents directly tied to the affected SKU, including both customer reviews and support conversations. Pattern-matching across reviews and support messages found the strongest repetition around the same SKU and the same failure mode.

3. Complaint themes
The recurring issue is a wash-related connectivity failure in the smart-fabric feature set:

The connected features stop working after washing.
The product no longer pairs with the app or the sensor appears offline.
Customers still describe the garment as comfortable, but the smart function—what makes the product premium—is what fails.
Representative customer excerpts:

“After the very first wash it stopped pairing with the Zava app entirely.”
“The top loses its connection to the companion app every time I wash it.”
“The sensor no longer connects to the Zava app after the first wash.”
“The garment is comfortable but the connected features, which are the whole point, are dead.”

4. Recommendation
The first action should be a targeted quality and product-response review for the ZCPTM-SS-M-BW SKU, focused on wash-cycle durability and app connectivity. This is the clearest, highest-confidence issue because it appears repeatedly across independent reviews and support interactions, affects a premium product with weak satisfaction, and is tied to a product experience failure rather than a routine service question.

5. Confidence and limitations
Confidence is high that the core issue is a product-quality and connectivity problem affecting the ZCPTM-SS-M-BW SKU.
The evidence strongly supports that the issue is concentrated on this SKU rather than being a broad, category-wide problem. The same pattern appears across multiple customer reviews and support conversations, including different languages.
The evidence is less complete on root cause, exact batch or production variation, and full open-versus-closed ticket status. The data supports the problem clearly, but it does not yet prove the precise hardware, manufacturing, or software cause.

                                                  **Investigation Workflow**

flowchart TD
    A[1. Schema Discovery<br/>Discover SQL MCP tools & entities]
    B[2. Category Risk Assessment<br/>Compare sales, support & satisfaction]
    C[3. Root Cause Analysis<br/>Support tickets & customer chats]
    D[4. Evidence Validation<br/>Document search & vector similarity]
    E[5. Executive Brief<br/>Executive summary& recommendations]

    A --> B
    B --> C
    C --> D
    D --> E

Loading

Reflection
This investigation demonstrated how GitHub Copilot Agent Mode can support an evidence-first analytical workflow rather than simply answering isolated questions. Each stage built on the previous one, moving from schema discovery to quantitative analysis, qualitative investigation, vector similarity search, and finally an executive-level recommendation. The most valuable aspect was the agent's ability to adapt its approach when initial queries failed, validate assumptions before concluding, and distinguish between evidence, interpretation, and recommendation throughout the investigation.

Where the agent struggled

1. The initial entity resolution through the SQL MCP layer failed, requiring a fallback to direct database queries.
2. The first category aggregation returned incomplete results until the joins were validated.
3. Early complaint themes included operational issues such as shipping enquiries, which needed to be separated from genuine product-quality signals.
4. Vector similarity returned some documents that were only textually similar rather than describing the same underlying defect, requiring human interpretation.

Completion

• Yes, the agent completed the mission or goal.
• No, the agent did not complete the mission or goal.

Bonus work

Beyond completing the required investigation, I refined the workflow to make the results more reliable and easier to interpret.

1. I redesigned each prompt so the investigation followed a logical progression - from schema discovery, to quantitative analysis, then qualitative analysis, vector similarity search, and finally an executive summary - rather than asking the agent to solve everything in a single prompt.
2. During the investigation, I questioned intermediate findings instead of accepting them at face value. For example, I asked the agent to verify that Premium was truly the strongest candidate across the entire product catalogue and confirmed that no product categories had been excluded because of missing ticket data.
3. I challenged the comparison between B2B and consumer product categories after noticing that direct revenue comparisons could be misleading due to different sales models. The investigation was adjusted to treat B2B as a separate business line unless metrics were normalised.
4. When the initial category aggregation produced unexpected results, I followed up until the agent validated the joins and confirmed the complete category coverage before moving to the next stage.
5. I refined later prompts to focus on evidence rather than raw outputs, asking the agent to group recurring complaint themes, distinguish operational issues from product-quality issues, and clearly separate observations from interpretations.
6. I finished the investigation by requesting an executive brief written for non-technical leadership, ensuring that the final recommendations were grounded in evidence collected during the investigation rather than new analysis.