Companion artifacts for the paper Auditable Multi-Pass AI Model Card and Datasheet Augmentation (Neelesh Karthikeyan, Beth Plale; School of Computer and Data Sciences, University of Oregon).
This repository holds the low-level material referenced from the paper's Artifact Availability section: the exact prompt templates, the full method descriptions, the aggregate evaluation results, and the evaluation corpus manifests.
| File | Contents | Paper section it supports |
|---|---|---|
methods.md |
Full M1/M2/M3 method descriptions, generator/judge configuration, and the production deployment decisions | Approach (Sec. II), Effectiveness (Sec. IV) |
prompts.md |
The five M1–M3 prompt templates, verbatim | Approach (Sec. II) |
aggregate-results.md |
Aggregate completeness / confidence / accuracy per method, the per-field datasheet comparison, and the trust-tier distribution across all three datasheet corpora | Effectiveness (Sec. IV) |
corpus.md |
Evaluation corpus manifests: 20 HuggingFace records, an unfiltered 10-dataset HuggingFace sample (manifest pending), and 16 NCAR GDEX AI-Ready datasets | Effectiveness (Sec. IV), three-corpus datasheet study |
- Generator: Llama 4-17B, temperature 0.2, served through LiteLLM on TACC Tapis.
- Judge: Qwen3-32B, temperature 0.0,
enable_thinking: False, drawn from a different model family than the generator to reduce self-enhancement bias. - Corpus: 20 HuggingFace records (10 model cards, 10 datasheets; stratified
most-downloaded vs. sparse) for the M1/M2/M3 method comparison, plus two
further datasheet corpora — an unfiltered random HuggingFace sample and 16
NCAR GDEX AI-Ready datasets — evaluated with the deployed M1+M3 pipeline to
span the range of metadata quality. See
corpus.md. - Judge calls: all filled fields for a card are scored in a single batched call — 60 calls in total (3 methods × 20 records) for the method comparison.
Citation for the paper will be added upon publication.
This work funded in part under a grant from the National Science Foundation OAC 2112606.