Add Language AI Evaluation 101 post

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: Boris Dev

docs/boris_dev_resume.md

@@ -127,7 +127,7 @@ Urban Mapping provided geospatial analytics to Tableau.
 | **Papers & code** | **Non-tech fun** |
 |---|---|
 | LLM-based taxonomy (topic modeling): [bertopic-easy](https://github.com/borisdev/bertopic-easy) | Climbed Cotopaxi (21,000 ft) |
-| [Language AI Evaluation 101: Know your user](https://medium.com/@boris.dev/why-did-your-language-ai-feature-fail-66a280954287) | Bodyboarded Mexpipe |
+| [Language AI Evaluation 101: Know your user](language-ai-evaluation-101.md) | Bodyboarded Mexpipe |
 | [Langchain PR: Causal Program-aided Language (CPAL)](https://github.com/hwchase17/langchain/pull/6255) — see Harrison Chase's [tweet](https://twitter.com/LangChainAI/status/1678797225013440514) | Taught with students in Medellín, Colombia to make ClusterPy (open-source geo clustering library) |
 | [Work papers](https://docs.google.com/document/d/1pMID97O4hHkK8ok7cwLH4Y4KpsgQSPUAXtYrscwcyb4/edit) | Taught kids snowboarding as an instructor |
 | [Academic papers](https://scholar.google.com/citations?hl=en&user=Nk4jOl0AAAAJ&view_op=list_works&gmla=AKKJWFcXmp1czN7ENwhvDx7hvgEHHD9lR1FLROPUvMco2ptysbNAe0Cdya8R9DZUmePAtMN53t2N97S_t5xA4NF-) | Counseled severely emotionally disturbed children |

docs/index.md

@@ -6,5 +6,6 @@
 | 2 | [What Is Knowledge Engineering, Really?](knowledge-engineering.md) | A working definition built around elicitation, evaluation, and 0→1 modeling in messy domains |
 | 3 | [Fine-Tuning LLMs Will Restructure Your Data Science Team](fine-tuning-restructures-data-science.md) | How fine-tuning replaces annotation pipelines and the NN-optimization role; the new "fine-tuning analyst" |
 | 4 | [Why Domain-Specific Language AI Features Fail](domain-specific-language-ai.md) | The customer-discovery process for niche language AI, and why a Lean Startup approach is required |
-| 5 | [Hyper-Local Community Funding: A DAO Alternative to CDFIs](hyperlocal-community-funding.md) | Local digital tokens and DAOs as a delivery mechanism for under-served-neighborhood capital |
-| 6 | [CV: Knowledge Engineering in Messy Domains](boris_dev_resume.md) | The IR-compile pattern across clinical trials, legal billing, maritime construction, narrative gaming, and geographic inequality |
+| 5 | [Language AI Evaluation 101: Know Your User](language-ai-evaluation-101.md) | Why simplistic Ground Truth produces misleading accuracy metrics; cognitive empathy as the iteration loop |
+| 6 | [Hyper-Local Community Funding: A DAO Alternative to CDFIs](hyperlocal-community-funding.md) | Local digital tokens and DAOs as a delivery mechanism for under-served-neighborhood capital |
+| 7 | [CV: Knowledge Engineering in Messy Domains](boris_dev_resume.md) | The IR-compile pattern across clinical trials, legal billing, maritime construction, narrative gaming, and geographic inequality |

docs/language-ai-evaluation-101.md

@@ -0,0 +1,38 @@
+# Language AI Evaluation 101: Know Your User
+
+*Originally published on [Medium](https://medium.com/@boris.dev/why-did-your-language-ai-feature-fail-66a280954287), Mar 20, 2023. Co-authors: Robert McKee and Shawn Larson.*
+
+## Background
+
+A language AI application can be a liability to your customer if it makes just a few terrible judgements on high-risk matters.
+
+## The evaluation problem
+
+"Most AI applications launch with bad predictions." Why? The cause is an overly simplistic evaluation test dataset that leads to exaggeratedly positive evaluation metrics. By "overly simplistic" we mean that the evaluation dataset, or Ground Truth, does not contain examples that robustly represent high-stakes, nuanced customer semantic judgements. Why is this Ground Truth so challenging to build? Because it requires both customer feedback and data science.
+
+## Illustration
+
+Imagine a Mom whose children text her hundreds of questions every day. Your job is to build her a language AI app that reduces her toil. You start by building a Ground Truth dataset of example correct responses. This Ground Truth is key to evaluating your future AI's prediction performance.
+
+**Iteration 1's Ground Truth.** The problem with this initial set of examples is that all of them depend on rationales that are overly simplistic. After training the model on these examples, evaluation against a test set drawn from the same simplistic distribution yields a 100% accuracy metric. That metric is misleading.
+
+To make a more rigorous evaluation, we need to discover more nuanced examples — questions whose correct response depends on context, priority, or risk that the simplistic rationales don't capture.
+
+**Iteration 2's Ground Truth.** Adding nuanced examples depends on *cognitive empathy* — sitting with the customer and discovering the implicit principles she actually uses. When we re-evaluate the same model against this richer test set, the accuracy drops to 0%.
+
+The lesson is twofold:
+
+1. **Your quantitative evaluation will be misleading when your Ground Truth is overly simplistic.** Same model, two test sets, two opposite verdicts.
+2. **Building Ground Truth is iterative, not a checkoff.** It is a continuous-improvement process driven by customer dialog, not a one-time annotation pass.
+
+In summary: *without cognitive empathy for your customer, your language AI performance metrics are meaningless.*
+
+## Connection to the IR-compile pattern
+
+This is step 1 of [the pattern](boris_dev_resume.md#common-pattern) — decomposing human expertise into an IR — applied specifically to evaluation. The "nuanced rationales" you discover through cognitive empathy *are* the primitive concepts of the domain IR. Without that decomposition, the eval dataset is just surface-text matching, and the metric measures nothing real.
+
+## References
+
+1. *Human-in-the-Loop Machine Learning* by Robert Munro, O'Reilly Media, 2021.
+2. *The One Practice That Is Separating The AI Successes From The Failures.* Forbes, Aug 14, 2022.
+3. *Evaluation.* LangChain Blog, March 14, 2023.

mkdocs.yml

@@ -42,5 +42,6 @@ nav:
   - "What Is Knowledge Engineering, Really?": knowledge-engineering.md
   - "Fine-Tuning LLMs Will Restructure Your Data Science Team": fine-tuning-restructures-data-science.md
   - "Why Domain-Specific Language AI Features Fail": domain-specific-language-ai.md
+  - "Language AI Evaluation 101: Know Your User": language-ai-evaluation-101.md
   - "Hyper-Local Community Funding: A DAO Alternative to CDFIs": hyperlocal-community-funding.md
   - "CV: Knowledge Engineering in Messy Domains": boris_dev_resume.md