Audit benchmark methodology: identify 7 critical integrity gaps

[Claude]

The repository claims "honest anti-mimicry evaluation" but benchmark methodology has fundamental gaps: proxy metrics without ground-truth validation, no baseline comparisons, missing robustness tests, and unvalidated claims presented as facts in user-facing documentation.

Issues Created

Created 7 detailed technical issues in benchmark-audit-issues/:

Issue 1: Proxy Metrics Don't Reflect Real Protection

• Protection score (0.65×style + 0.35×embedding) uses arbitrary weights, single ResNet18 extractor
• No calibration against actual mimicry outcomes
• Thresholds (Strong≥45) are uncalibrated

Issue 2: Missing Ground-Truth Validation

• Complete LoRA/DreamBooth benchmark infrastructure exists but zero published validation results
• All claims based on proxy metrics
• Cannot verify protection survives actual mimicry training

Issue 3: Weak Robustness Testing

• Only 4 transforms tested (identity, blur, 2×resize)
• Missing: JPEG compression, crop, CLIP/VAE preprocessing, purification defenses
• Resize-restore ≠ JPEG artifacts

Issue 4: Dataset Split Methodology Risks

• No train/val/test separation or enforcement
• Anti-DreamBooth set_C is "holdout metadata" (not actually held out)
• Overfitting and cherry-picking not prevented

Issue 5: Insufficient Reproducibility Documentation

• Published results cannot be independently verified
• Missing: dataset provenance, model versions, random seeds, environment specs
• No result archives or reproduction scripts

Issue 6: No Baseline Comparisons

• Zero comparisons against existing methods (Anti-DreamBooth, Mist, Glaze)
• No unprotected or naive baselines (Gaussian noise, JPEG, blur)
• Cannot assess if AuraLock represents progress over SOTA

Issue 7: Misleading Benchmark Framing

• Unvalidated claims in README presented as facts with definitive language
• Critical disclaimer buried after impressive numbers (line 60 vs lines 52-58)
• Claims "honest evaluation" while using marketing presentation patterns

Structure

Each issue includes:

• Technical problem with code evidence and line numbers
• User deception scenarios
• Proposed fixes with acceptance criteria
• Implementation phases

Summary index in benchmark-audit-issues/README.md with severity assessment and recommendations.

Immediate Actions Needed

1. Add prominent validation status disclaimers to README
2. Run minimal ground-truth validation (10 subjects × 5 profiles, ~$50 on Colab)
3. Implement JPEG compression testing (most critical robustness gap)

Example: Current vs Honest Framing

Current (line 54):

| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |

Should be:

⚠️ All scores are proxy metrics only. Real protection effectiveness not validated.

| Profile | Proxy Score (unvalidated) | PSNR | SSIM | Validation Status |
| `fortress` | `53.2` | `29.08` | `0.7858` | ❌ No ground-truth validation |