Mission/open goal Description
I chose the "Bring Your Own" mission.
Star ratings are lying to you.
Every ITSM tool on Capterra sits between 4.4 and 4.6 stars. They all look the same. But when you actually read what customers wrote — not what they rated — a different story emerges. Some of those "happy" customers sound genuinely frustrated. They just didn't say it in the stars.
I built the Liar Score: a measure of how much a tool's star rating is hiding about real customer feelings. Using 4,899 real Capterra reviews across 6 ITSM tools, I asked one question: who is hiding the most?
The answer surprised me. ServiceNow and Jira — two of the most enterprise-trusted tools in the market — have the highest Liar Scores. Nearly 1 in 5 of their "satisfied" customers sounds frustrated in the text.
Freshdesk, on the other hand, is the most honest tool in the dataset. What customers rate and what they write actually align.
This is the Bring Your Own mission. I brought my own data, my own question, and built something I'd actually want to use.
Harness and model
GitHub Copilot Chat — Agent Mode, Claude Sonnet
Turn-by-turn journey
Turn 1 — Loading the data into SQL
Why: The contest requires everything to run through SQL MCP tools, not just local Python scripts. I needed my dataset living in the database before any analysis could begin.
What I asked: Load data/capterra_reviews.csv into PromptathonDb as a CapterraReviews table via SQL MCP.
Result: 4,899 reviews loaded successfully across 6 ITSM tools. The agent also ran a data quality check and flagged that 34% of cons_text fields were empty — expected, since many reviewers skip the cons section.
Turn 2 — Testing the core hypothesis
Why: Before building anything, I wanted to verify my hunch was real — do high-rated reviews actually contain frustrated language?
What I asked: Pull a sample of 4-5 star reviews and look for signs of real frustration in the text.
Result: Confirmed. The pattern exists. Reviewers giving 5 stars still wrote things like "slow loading, very annoying" and "login process is cumbersome." The hypothesis was worth building on.
Turn 3 — Scoring sentiment with VADER
Why: I needed a defensible, scalable way to measure the gap between star ratings and text sentiment across all 4,899 reviews — not just a sample.
What I asked: Install vaderSentiment, score every review using combined pros+cons+overall text, store sentiment_compound and sentiment_label back in the database.
Result: All reviews scored. Key finding: 437 reviews with 4-5 stars had non-positive sentiment — that became the Liar Score population.
Turn 4 — Calculating the Liar Score per tool
Why: I wanted to know which tool was hiding the most frustration, not just that the pattern existed globally.
What I asked: Calculate Liar Score rate per tool — OverallRating >= 4 but sentiment != positive — ranked worst to best.
Result: ServiceNow 18.18% , Jira 16.77% , Zoho Desk 8.43% , Zendesk 7.25% , OTRS 5.62% , Freshdesk 4.98% . All tools had nearly identical star averages (4.4-4.6) — impossible to distinguish by rating alone, but the Liar Score separated them clearly.
Turn 5 — Classifying what they're actually complaining about
Why: Knowing that 18% of ServiceNow reviews hide frustration is interesting, but not actionable. I needed to know WHY they're frustrated.
What I asked: For each tool's flagged reviews, classify the complaint into: Support Quality, Pricing, Features, Ease of Use, Performance, Integration.
Result: Support Quality dominated across all 6 tools. Specific insight: Zendesk — whose core product IS customer support — had the most hidden Support Quality complaints (48). Zoho Desk had the most diverse frustration profile (Support + Pricing + Features).
Turn 6 — Adding semantic search
Why: The minimum bar required either two tables or a vector/semantic search step. I chose semantic search because it adds real value — finding the same complaint expressed differently across tools.
What I asked: Build a semantic search function using sentence-transformers so I can find similar hidden complaints across tools using natural language.
Result: Working semantic search. Query "slow ticket routing" returns similar complaints from multiple tools even when worded completely differently.
Turn 7 — Building the Red Flag Detector
Why: I wanted to turn the analysis into a reusable tool, not just a report. Anyone should be able to paste a review and instantly know if it's hiding something.
What I asked: Build a function that takes any review text and returns " This review is hiding something" or " This review is genuine" with confidence score and complaint aspect detected.
Result: Working detector. Example: "Great tool overall, but the support team never responds" → hiding something, Support Quality, confidence 0.88.
Turn 8 — Building the Recommendation Engine
Why: The logical conclusion of the analysis — if you know which tool hides the most complaints in each category, you can recommend the right tool based on what someone actually cares about.
What I asked: Build a function that takes a user priority (Support Quality, Pricing, Integration...) and ranks all 6 tools from best to worst based on hidden complaint patterns.
Result: Working engine. Example — if Integration is your priority: best choice is ServiceNow (0% hidden integration complaints), avoid Jira (11.6%).
Dead end documented:
Initially tried BERT (nlptown/bert-base-multilingual-uncased-sentiment) for sentiment scoring — more powerful than VADER but crashed the Codespace repeatedly due to memory limits. Switched to VADER which proved reliable and still produced meaningful, defensible results.
Full journey documented here: https://github.com/rajaechouiref/sql-ai-promptathon/blob/main/JOURNEY.md
Completion
Bonus work
Beyond the core analysis, I pushed the project further in three directions:
1. The Red Flag Detector
Instead of just reporting which tools have high Liar Scores, I built a tool anyone can use right now: paste any review text and get an instant verdict — " This review is hiding something" or " This review is genuine" — with the confidence level and the complaint category detected. This turns a static analysis into a reusable product.
2. The Tool Recommendation Engine
I built a recommendation system that goes beyond ranking tools by Liar Score. Given a user's priority — Support Quality, Pricing, Integration, etc. — it ranks all 6 tools from best to worst based on hidden complaint patterns, not visible ratings. The insight: ServiceNow looks average on paper but is actually the best choice if Pricing is your concern (0% hidden pricing complaints). Jira looks good on paper but has the most hidden Integration complaints.
Mission/open goal Description
I chose the "Bring Your Own" mission.
Star ratings are lying to you.
Every ITSM tool on Capterra sits between 4.4 and 4.6 stars. They all look the same. But when you actually read what customers wrote — not what they rated — a different story emerges. Some of those "happy" customers sound genuinely frustrated. They just didn't say it in the stars.
I built the Liar Score: a measure of how much a tool's star rating is hiding about real customer feelings. Using 4,899 real Capterra reviews across 6 ITSM tools, I asked one question: who is hiding the most?
The answer surprised me. ServiceNow and Jira — two of the most enterprise-trusted tools in the market — have the highest Liar Scores. Nearly 1 in 5 of their "satisfied" customers sounds frustrated in the text.
Freshdesk, on the other hand, is the most honest tool in the dataset. What customers rate and what they write actually align.
This is the Bring Your Own mission. I brought my own data, my own question, and built something I'd actually want to use.
Harness and model
GitHub Copilot Chat — Agent Mode, Claude Sonnet
Turn-by-turn journey
Turn 1 — Loading the data into SQL
Why: The contest requires everything to run through SQL MCP tools, not just local Python scripts. I needed my dataset living in the database before any analysis could begin.
What I asked: Load data/capterra_reviews.csv into PromptathonDb as a CapterraReviews table via SQL MCP.
Result: 4,899 reviews loaded successfully across 6 ITSM tools. The agent also ran a data quality check and flagged that 34% of cons_text fields were empty — expected, since many reviewers skip the cons section.
Turn 2 — Testing the core hypothesis
Why: Before building anything, I wanted to verify my hunch was real — do high-rated reviews actually contain frustrated language?
What I asked: Pull a sample of 4-5 star reviews and look for signs of real frustration in the text.
Result: Confirmed. The pattern exists. Reviewers giving 5 stars still wrote things like "slow loading, very annoying" and "login process is cumbersome." The hypothesis was worth building on.
Turn 3 — Scoring sentiment with VADER
Why: I needed a defensible, scalable way to measure the gap between star ratings and text sentiment across all 4,899 reviews — not just a sample.
What I asked: Install vaderSentiment, score every review using combined pros+cons+overall text, store sentiment_compound and sentiment_label back in the database.
Result: All reviews scored. Key finding: 437 reviews with 4-5 stars had non-positive sentiment — that became the Liar Score population.
Turn 4 — Calculating the Liar Score per tool
Why: I wanted to know which tool was hiding the most frustration, not just that the pattern existed globally.
What I asked: Calculate Liar Score rate per tool — OverallRating >= 4 but sentiment != positive — ranked worst to best.
Result: ServiceNow 18.18% , Jira 16.77% , Zoho Desk 8.43% , Zendesk 7.25% , OTRS 5.62% , Freshdesk 4.98% . All tools had nearly identical star averages (4.4-4.6) — impossible to distinguish by rating alone, but the Liar Score separated them clearly.
Turn 5 — Classifying what they're actually complaining about
Why: Knowing that 18% of ServiceNow reviews hide frustration is interesting, but not actionable. I needed to know WHY they're frustrated.
What I asked: For each tool's flagged reviews, classify the complaint into: Support Quality, Pricing, Features, Ease of Use, Performance, Integration.
Result: Support Quality dominated across all 6 tools. Specific insight: Zendesk — whose core product IS customer support — had the most hidden Support Quality complaints (48). Zoho Desk had the most diverse frustration profile (Support + Pricing + Features).
Turn 6 — Adding semantic search
Why: The minimum bar required either two tables or a vector/semantic search step. I chose semantic search because it adds real value — finding the same complaint expressed differently across tools.
What I asked: Build a semantic search function using sentence-transformers so I can find similar hidden complaints across tools using natural language.
Result: Working semantic search. Query "slow ticket routing" returns similar complaints from multiple tools even when worded completely differently.
Turn 7 — Building the Red Flag Detector
Why: I wanted to turn the analysis into a reusable tool, not just a report. Anyone should be able to paste a review and instantly know if it's hiding something.
What I asked: Build a function that takes any review text and returns " This review is hiding something" or " This review is genuine" with confidence score and complaint aspect detected.
Result: Working detector. Example: "Great tool overall, but the support team never responds" → hiding something, Support Quality, confidence 0.88.
Turn 8 — Building the Recommendation Engine
Why: The logical conclusion of the analysis — if you know which tool hides the most complaints in each category, you can recommend the right tool based on what someone actually cares about.
What I asked: Build a function that takes a user priority (Support Quality, Pricing, Integration...) and ranks all 6 tools from best to worst based on hidden complaint patterns.
Result: Working engine. Example — if Integration is your priority: best choice is ServiceNow (0% hidden integration complaints), avoid Jira (11.6%).
Dead end documented:
Initially tried BERT (nlptown/bert-base-multilingual-uncased-sentiment) for sentiment scoring — more powerful than VADER but crashed the Codespace repeatedly due to memory limits. Switched to VADER which proved reliable and still produced meaningful, defensible results.
Full journey documented here: https://github.com/rajaechouiref/sql-ai-promptathon/blob/main/JOURNEY.md
Completion
Bonus work
Beyond the core analysis, I pushed the project further in three directions:
1. The Red Flag Detector
Instead of just reporting which tools have high Liar Scores, I built a tool anyone can use right now: paste any review text and get an instant verdict — " This review is hiding something" or " This review is genuine" — with the confidence level and the complaint category detected. This turns a static analysis into a reusable product.
2. The Tool Recommendation Engine
I built a recommendation system that goes beyond ranking tools by Liar Score. Given a user's priority — Support Quality, Pricing, Integration, etc. — it ranks all 6 tools from best to worst based on hidden complaint patterns, not visible ratings. The insight: ServiceNow looks average on paper but is actually the best choice if Pricing is your concern (0% hidden pricing complaints). Jira looks good on paper but has the most hidden Integration complaints.