From 703230e841d848a5f8832841a1281ec1392e0aae Mon Sep 17 00:00:00 2001
From: "anthropic-code-agent[bot]" <242468646+Claude@users.noreply.github.com>
Date: Tue, 31 Mar 2026 12:45:30 +0000
Subject: [PATCH 1/2] Initial plan
From 357a2d2379056ed319ad93ccb1171779908fa1a3 Mon Sep 17 00:00:00 2001
From: "anthropic-code-agent[bot]" <242468646+Claude@users.noreply.github.com>
Date: Tue, 31 Mar 2026 13:02:03 +0000
Subject: [PATCH 2/2] Add comprehensive benchmark audit issues
Created 7 detailed technical issues identifying benchmark weaknesses:
- Issue 1: Proxy metrics don't reflect real anti-mimicry strength
- Issue 2: Missing ground-truth LoRA/DreamBooth validation
- Issue 3: Weak robustness testing (no JPEG, crop, CLIP/VAE)
- Issue 4: Dataset split methodology risks (no train/val/test)
- Issue 5: Insufficient reproducibility documentation
- Issue 6: Weak baseline comparison methodology (no SOTA comparisons)
- Issue 7: Misleading benchmark framing in README
Each issue includes:
- Technical problem analysis with code evidence
- Why it misleads users (scenarios)
- Proposed upgrades with acceptance criteria
- Implementation phases and timelines
All issues focus on benchmark integrity and honest evaluation.
Agent-Logs-Url: https://github.com/VoDaiLocz/Lock-ART./sessions/29bda424-1fe4-4e2f-b25c-c508c267e754
Co-authored-by: VoDaiLocz <88762074+VoDaiLocz@users.noreply.github.com>
---
benchmark-audit-issues/README.md | 258 +++++++
...xy-metrics-dont-reflect-real-protection.md | 214 ++++++
...issue-2-missing-ground-truth-validation.md | 337 +++++++++
.../issue-3-weak-robustness-testing.md | 438 +++++++++++
...issue-4-dataset-split-methodology-risks.md | 536 ++++++++++++++
...ufficient-reproducibility-documentation.md | 685 ++++++++++++++++++
...-6-weak-baseline-comparison-methodology.md | 630 ++++++++++++++++
.../issue-7-misleading-benchmark-framing.md | 515 +++++++++++++
8 files changed, 3613 insertions(+)
create mode 100644 benchmark-audit-issues/README.md
create mode 100644 benchmark-audit-issues/issue-1-proxy-metrics-dont-reflect-real-protection.md
create mode 100644 benchmark-audit-issues/issue-2-missing-ground-truth-validation.md
create mode 100644 benchmark-audit-issues/issue-3-weak-robustness-testing.md
create mode 100644 benchmark-audit-issues/issue-4-dataset-split-methodology-risks.md
create mode 100644 benchmark-audit-issues/issue-5-insufficient-reproducibility-documentation.md
create mode 100644 benchmark-audit-issues/issue-6-weak-baseline-comparison-methodology.md
create mode 100644 benchmark-audit-issues/issue-7-misleading-benchmark-framing.md
diff --git a/benchmark-audit-issues/README.md b/benchmark-audit-issues/README.md
new file mode 100644
index 0000000..51d1e4c
--- /dev/null
+++ b/benchmark-audit-issues/README.md
@@ -0,0 +1,258 @@
+# Benchmark Audit Issues - Summary Index
+
+This directory contains a comprehensive technical audit of the Lock-ART/AuraLock benchmark methodology. Each issue identifies specific weaknesses in benchmark integrity and anti-mimicry evaluation quality.
+
+## Overview
+
+The audit was conducted on 2026-03-31 and identified **7 critical benchmark issues** that can mislead users about protection effectiveness.
+
+## Critical Issues
+
+### Issue 1: Proxy Metrics Don't Reflect Real Anti-Mimicry Strength
+**File**: `issue-1-proxy-metrics-dont-reflect-real-protection.md`
+
+**Summary**: Protection Score is a proxy metric (ResNet18 feature drift) that may not correlate with actual mimicry prevention. Arbitrary weight assignment (0.65/0.35), single feature extractor, no calibration against real mimicry outcomes.
+
+**Impact**: HIGH - Users may trust high protection scores that don't reflect real-world effectiveness.
+
+**Key Problems**:
+- Arbitrary 0.65/0.35 weight split with no empirical justification
+- Single ResNet18 extractor vs actual CLIP/VAE in mimicry models
+- No validation that high scores prevent actual mimicry
+- Thresholds (Strong ≥45, Moderate ≥25) are uncalibrated
+
+---
+
+### Issue 2: Missing Ground-Truth LoRA/DreamBooth Validation
+**File**: `issue-2-missing-ground-truth-validation.md`
+
+**Summary**: Complete benchmark infrastructure exists (LoRA harness, Anti-DreamBooth splits, Docker runtime) but **zero published validation results**. All protection claims based on proxy metrics without demonstrating actual mimicry prevention.
+
+**Impact**: CRITICAL - Cannot verify if protection actually works against real mimicry models.
+
+**Key Problems**:
+- Infrastructure present but never executed with `execute=True` for published results
+- No evidence that protection survives DreamBooth/LoRA training
+- Colab notebook exists but has no executed outputs
+- Roadmap acknowledges GPU validation is "pending" (still not done)
+
+---
+
+### Issue 3: Weak Robustness Testing
+**File**: `issue-3-weak-robustness-testing.md`
+
+**Summary**: Only 4 basic transforms tested (identity, blur, 2×resize). Missing critical preprocessing: JPEG compression, center crop, CLIP preprocessing, VAE encoding, color jitter, rotation. Protection may survive blur but fail against JPEG.
+
+**Impact**: HIGH - False robustness claims. Purification attacks not tested.
+
+**Key Problems**:
+- No JPEG compression (most effective purification defense)
+- Resize-restore ≠ JPEG (different artifacts)
+- No crop testing (standard augmentation)
+- No CLIP/VAE preprocessing (actual mimicry pipeline)
+- Single blur configuration (kernel=5, sigma=1.0)
+
+---
+
+### Issue 4: Dataset Split Methodology Risks
+**File**: `issue-4-dataset-split-methodology-risks.md`
+
+**Summary**: No train/val/test split separation or enforcement. Risk of overfitting, data leakage, and cherry-picking results. Anti-DreamBooth set_C is "metadata only", not used for held-out evaluation.
+
+**Impact**: HIGH - Results may be overfit to test set. Cannot verify generalization.
+
+**Key Problems**:
+- Local benchmark has no split methodology at all
+- Nothing prevents tuning on test images
+- set_C described as "holdout metadata" (not actually held out)
+- No split tracking in reports
+- Cherry-picking not prevented
+
+---
+
+### Issue 5: Insufficient Reproducibility Documentation
+**File**: `issue-5-insufficient-reproducibility-documentation.md`
+
+**Summary**: Cannot reproduce or verify published results. Missing: exact datasets, model versions, random seeds, hardware specs, environment details, execution logs. Published results exist only in README markdown without archived data.
+
+**Impact**: HIGH - Cannot independently verify benchmark claims. Violates scientific standards.
+
+**Key Problems**:
+- No benchmark result archives (no `benchmark_results/` directory)
+- Reports missing environment metadata (PyTorch version, hardware, timestamp)
+- Loose version constraints (torch>=2.0.0 allows drift)
+- No reproducibility guide
+- README results have no source trace
+
+---
+
+### Issue 6: Weak Baseline Comparison Methodology
+**File**: `issue-6-weak-baseline-comparison-methodology.md`
+
+**Summary**: Zero comparisons against existing methods. All results are self-referential (AuraLock profiles vs each other). No evaluation against Anti-DreamBooth, Mist, Glaze, PhotoGuard, or even unprotected baselines.
+
+**Impact**: CRITICAL - Cannot assess if AuraLock represents progress or is weaker than SOTA.
+
+**Key Problems**:
+- No unprotected baseline (can't quantify protection gained)
+- No naive baselines (Gaussian noise, JPEG, blur)
+- No academic baselines (Anti-DreamBooth, Mist)
+- Anti-DreamBooth infrastructure uses protocol but not method
+- Benchmark closed to external methods (no plugin system)
+
+---
+
+### Issue 7: Misleading Benchmark Framing in README
+**File**: `issue-7-misleading-benchmark-framing.md`
+
+**Summary**: Unvalidated claims presented as facts using definitive language. Critical limitations buried in fine print after impressive numbers. Claims "honest evaluation" while using marketing presentation patterns.
+
+**Impact**: CRITICAL - Misleads users into false security. Contradicts "honest" framing claim.
+
+**Key Problems**:
+- Definitive results (lines 52-58) before disclaimer (line 60)
+- Profile names suggest validation ("fortress", "blindfold")
+- "Benchmark" terminology implies validation
+- "Honest evaluation" claim contradicted by framing
+- Repository description implies proven "cloaking"
+
+---
+
+## Severity Assessment
+
+| Issue | Severity | User Impact | Implementation Effort |
+|-------|----------|-------------|----------------------|
+| Issue 1: Proxy Metrics | HIGH | False confidence in protection | Medium (2-3 months for full fix) |
+| Issue 2: No Ground-Truth | CRITICAL | Cannot verify protection works | High (GPU validation needed) |
+| Issue 3: Weak Robustness | HIGH | Purification attacks undetected | Medium (2-4 weeks) |
+| Issue 4: Dataset Splits | HIGH | Overfitting risk | Medium (2-3 weeks) |
+| Issue 5: Reproducibility | HIGH | Cannot verify claims | Medium (2-3 weeks) |
+| Issue 6: No Baselines | CRITICAL | Cannot assess vs SOTA | High (need baseline reproductions) |
+| Issue 7: Misleading Framing | CRITICAL | User deception risk | Low (1-2 weeks documentation) |
+
+## Audit Methodology
+
+### Audit Scope
+- Repository: `VoDaiLocz/Lock-ART.` (github.com)
+- Commit: `703230e` (Initial plan, 2026-03-31)
+- Branch: `claude/audit-benchmark-issues`
+- Focus: Benchmark integrity and anti-mimicry evaluation quality
+
+### Audit Process
+1. Explored repository structure and benchmark code
+2. Analyzed evaluation methodology and metrics
+3. Reviewed documentation and published claims
+4. Identified gaps between claims and validation
+5. Assessed user deception risk
+6. Proposed technical fixes with acceptance criteria
+
+### Audit Standards
+- **Strict technical rigor**: No leniency for unvalidated claims
+- **User protection focus**: Prioritize preventing false security
+- **Scientific standards**: Require reproducibility and validation
+- **Honest communication**: Demand prominent disclaimers
+
+## Recommendations
+
+### Immediate Actions (Week 1)
+1. **Add prominent disclaimers** to README (Issue #7)
+ - Validation status section at top
+ - "Proxy only, unvalidated" labels on all results
+ - CLI warnings when protection is used
+
+2. **Archive current results** with full metadata (Issue #5)
+ - Create `benchmark_results/` directory
+ - Capture environment info
+ - Generate reproduction scripts
+
+3. **Implement JPEG compression testing** (Issue #3)
+ - Most critical robustness gap
+ - Can be done quickly without GPU
+
+### Short-Term Priorities (Month 1)
+1. **Run minimal ground-truth validation** (Issue #2)
+ - 10 subjects × 5 profiles = 50 LoRA training runs
+ - Cost: ~$50 on Google Colab Pro
+ - Publish honest results even if protection is weak
+
+2. **Add unprotected and naive baselines** (Issue #6)
+ - Unprotected (identity)
+ - Gaussian noise
+ - JPEG compression
+ - Gaussian blur
+
+3. **Implement dataset split methodology** (Issue #4)
+ - Create train/val/test splits
+ - Add split tracking to reports
+ - Use set_C for held-out validation
+
+### Medium-Term Goals (Months 2-3)
+1. **Comprehensive ground-truth validation** (Issue #2)
+ - Scale to 50+ subjects
+ - Multiple mimicry methods (DreamBooth, LoRA, Textual Inversion)
+ - Statistical significance testing
+
+2. **Reproduce academic baselines** (Issue #6)
+ - Anti-DreamBooth ASPL method
+ - Mist (if reproducible)
+ - Comparative evaluation
+
+3. **Multi-space feature evaluation** (Issue #1)
+ - Add CLIP and DINO extractors
+ - Evaluate protection in multiple feature spaces
+ - Correlate proxy scores with mimicry prevention
+
+### Long-Term Vision (Months 3-6)
+1. **Establish rigorous benchmark standard**
+ - Ground-truth validation as requirement
+ - Baseline comparisons mandatory
+ - Reproducibility infrastructure automated
+
+2. **Independent verification**
+ - Invite external researchers to validate
+ - Publish results in peer-reviewed venue
+ - ACM artifact evaluation badges
+
+3. **Continuous validation CI/CD**
+ - Monthly GPU validation runs
+ - Alert on protection degradation
+ - Public results dashboard
+
+## Expected Outcomes
+
+### If Protection Works Well
+- Publish validation evidence
+- Update claims with confidence
+- Establish as credible SOTA baseline
+- Contribute to research community
+
+### If Protection Works Poorly
+- Report honest findings
+- Identify failure modes
+- Guide research improvements
+- Build trust through transparency
+
+### Either Way
+- **Honesty builds trust** more than perfect scores
+- Users deserve truth, not marketing
+- Science requires validation, not claims
+- Iterate based on evidence, not assumptions
+
+## Contact
+
+This audit was conducted as a technical review of benchmark methodology. For questions or discussion:
+
+- Open issues on the repository
+- Reference specific issue files in discussions
+- Follow acceptance criteria for fixes
+
+## License
+
+This audit documentation is provided for the benefit of the Lock-ART project and research community. Issues should be addressed systematically according to severity and user impact.
+
+---
+
+**Audit Date**: 2026-03-31
+**Audit Version**: 1.0
+**Repository**: VoDaiLocz/Lock-ART. (github.com)
+**Branch**: claude/audit-benchmark-issues
diff --git a/benchmark-audit-issues/issue-1-proxy-metrics-dont-reflect-real-protection.md b/benchmark-audit-issues/issue-1-proxy-metrics-dont-reflect-real-protection.md
new file mode 100644
index 0000000..d395d22
--- /dev/null
+++ b/benchmark-audit-issues/issue-1-proxy-metrics-dont-reflect-real-protection.md
@@ -0,0 +1,214 @@
+# Issue: Protection Score is a Proxy Metric That May Not Reflect Real Anti-Mimicry Strength
+
+## Labels
+`benchmark`, `critical`, `evaluation`, `research`, `technical-debt`
+
+## Problem Description
+
+The repository uses a "Protection Score" derived from style and embedding similarity as the primary metric to evaluate anti-mimicry effectiveness. **This is a proxy metric**, not a ground-truth measurement of actual style mimicry prevention. The score can be high while the protection may still be ineffective against real-world mimicry attacks.
+
+## What is Wrong with the Benchmark
+
+**Current Implementation** (from `/src/auralock/core/metrics.py:308-311`):
+```python
+protection_score = 100.0 * (
+ 0.65 * (1.0 - robust_style_similarity)
+ + 0.35 * (1.0 - robust_embedding_similarity)
+)
+```
+
+**Critical Issues**:
+
+1. **Arbitrary Weight Assignment**: The 0.65/0.35 split between style and embedding similarity has no empirical justification. Why not 0.5/0.5? Or 0.8/0.2? These weights are not validated against actual mimicry outcomes.
+
+2. **Single Feature Extractor**: Uses only ResNet18 with ImageNet weights (`/src/auralock/core/style.py:107`). Real-world mimicry models (DreamBooth, LoRA) use completely different architectures (CLIP, VAE, U-Net). The feature space mismatch means:
+ - High protection score ≠ actual protection against mimicry models
+ - The metric measures "drift in ResNet18 space" not "prevention of style transfer"
+
+3. **No Calibration Against Real Mimicry**: The protection score thresholds (Strong ≥45, Moderate ≥25, Weak <25) are arbitrary. There is no evidence that:
+ - A score of 45 actually prevents DreamBooth from learning the style
+ - A score of 25 provides any meaningful protection
+ - These thresholds correlate with mimicry success/failure rates
+
+4. **Limited Transform Suite** (`/src/auralock/core/style.py:217-231`): Only tests 4 transforms:
+ - identity
+ - gaussian_blur (kernel=5, sigma=1.0)
+ - resize_restore_75
+ - resize_restore_50
+
+ Missing many common preprocessing steps that mimicry models use (CLIP preprocessing, VAE encoding, data augmentation).
+
+## Why This Can Mislead Users
+
+### Scenario 1: False Confidence
+```python
+# User protects artwork, sees "Strong" protection (score=48.2)
+# Reality: DreamBooth can still successfully learn the style
+# because ResNet18 drift ≠ CLIP/VAE feature preservation
+```
+
+### Scenario 2: Unmeasured Attack Surface
+Users believe their artwork is protected when:
+- The protection score is high in ResNet18 feature space
+- But style information remains recoverable in CLIP/VAE space
+- Actual mimicry models operate in the unmonitored space
+
+### Scenario 3: Misleading Comparisons
+The README claims (lines 52-58):
+```
+| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
+| `blindfold` | `61.1` | `26.53` | `0.6114` | strongest current anti-readability preset |
+```
+
+**But there's no evidence these profiles actually prevent mimicry.** The scores are relative measurements in a proxy space, not absolute protection guarantees.
+
+## Evidence from Repository
+
+1. **No Ground-Truth Correlation Studies**: Search for validation that protection score predicts mimicry prevention:
+ - `/src/tests/test_metrics.py`: Tests metric calculation, not correlation with mimicry success
+ - `/src/tests/test_stylecloak.py`: Tests protection score computation, not mimicry prevention
+ - **No test validates that high protection scores prevent actual mimicry**
+
+2. **Acknowledged as Proxy** (line 60 in README):
+ > "The `Protection Score` is an internal proxy derived from embedding and style similarity after robustness transforms. It is useful for relative comparisons inside this repository, not as a universal guarantee against all AI systems."
+
+ But this critical limitation is buried in documentation, not prominently displayed in results.
+
+3. **Benchmark Harness Exists But No Results**: The repository includes:
+ - `/src/auralock/benchmarks/lora.py`: LoRA benchmark infrastructure
+ - `/src/auralock/benchmarks/antidreambooth.py`: Anti-DreamBooth benchmark
+ - **BUT: No published results correlating protection scores with actual mimicry outcomes**
+
+## Proposed Benchmark Upgrade
+
+### 1. Ground-Truth Validation Study
+
+**Objective**: Measure correlation between protection score and actual mimicry prevention.
+
+**Protocol**:
+```
+For each profile (safe, balanced, strong, fortress, blindfold):
+ 1. Protect N=50 diverse artworks
+ 2. Record protection score for each
+ 3. Train DreamBooth/LoRA on protected images:
+ - Use standard settings (resolution=512, steps=400)
+ - Generate M=10 samples per trained model
+ 4. Measure mimicry success rate:
+ - Human evaluation: Does generated image match original style?
+ - Automated: CLIP similarity to original style exemplars
+ 5. Compute correlation: protection_score vs mimicry_success_rate
+```
+
+**Expected Output**:
+- Scatter plot: protection_score (x-axis) vs mimicry_prevention_rate (y-axis)
+- Regression analysis with R² value
+- Threshold recalibration based on empirical data
+
+### 2. Multi-Space Feature Evaluation
+
+Replace single ResNet18 with ensemble:
+```python
+# Current (biased)
+extractor = resnet18(weights=ResNet18_Weights.DEFAULT)
+
+# Proposed (comprehensive)
+extractors = {
+ 'resnet18': resnet18(weights=ResNet18_Weights.DEFAULT),
+ 'clip_vit': CLIPVisionModel.from_pretrained('openai/clip-vit-base-patch32'),
+ 'dino_vit': torch.hub.load('facebookresearch/dino:main', 'dino_vits16'),
+}
+
+# Aggregate protection across all spaces
+protection_score_ensemble = mean([
+ score_from_extractor(extractors[name])
+ for name in extractors
+])
+```
+
+**Rationale**: If protection works in ResNet18 but fails in CLIP space, the overall protection is weak because mimicry models use CLIP.
+
+### 3. Mimicry-Specific Transforms
+
+Extend transform suite to match actual mimicry model preprocessing:
+```python
+def build_mimicry_aligned_transform_suite():
+ return (
+ ("identity", identity),
+ ("gaussian_blur", gaussian_blur),
+ ("resize_restore_75", resize_restore_75),
+ ("resize_restore_50", resize_restore_50),
+ # NEW: Match actual preprocessing
+ ("clip_preprocess", clip_preprocessing_pipeline),
+ ("vae_encode_decode", vae_roundtrip),
+ ("jpeg_compress_90", jpeg_compression_90),
+ ("jpeg_compress_70", jpeg_compression_70),
+ ("random_crop_center", center_crop_and_resize),
+ ("color_jitter", color_jitter_augment),
+ )
+```
+
+### 4. Separate Reporting: Proxy vs Ground-Truth
+
+Clearly distinguish in all outputs:
+```json
+{
+ "proxy_metrics": {
+ "protection_score_resnet18": 48.2,
+ "assessment": "Strong (proxy space)",
+ "warning": "This score measures drift in ResNet18 features, NOT actual mimicry prevention"
+ },
+ "ground_truth_metrics": {
+ "mimicry_prevention_rate": null,
+ "reason": "Not yet evaluated. Run `auralock benchmark-lora --execute` to validate.",
+ "status": "requires_gpu_validation"
+ }
+}
+```
+
+## Acceptance Criteria
+
+### Phase 1: Immediate Fixes (Week 1-2)
+- [ ] Add prominent warnings to all protection score outputs:
+ - CLI output: "⚠️ Protection score is a proxy metric. Real-world effectiveness not validated."
+ - JSON reports: Include `"metric_type": "proxy_unvalidated"` field
+ - README: Move warning from line 60 to top of results table (lines 52-58)
+
+### Phase 2: Correlation Study (Month 1-2)
+- [ ] Design ground-truth validation protocol (detailed spec document)
+- [ ] Implement mimicry success rate measurement tools
+- [ ] Run validation study on N=50+ artworks × 5 profiles = 250+ tests
+- [ ] Publish correlation analysis: protection_score vs mimicry_prevention_rate
+- [ ] Recalibrate thresholds (Strong/Moderate/Weak) based on empirical data
+
+### Phase 3: Multi-Space Protection (Month 2-3)
+- [ ] Implement multi-extractor evaluation (ResNet18 + CLIP + DINO)
+- [ ] Add mimicry-aligned transforms (CLIP preprocess, VAE, JPEG, crop)
+- [ ] Report per-space protection scores and ensemble score
+- [ ] Update tests to validate cross-space protection
+
+### Phase 4: Honest Reporting (Ongoing)
+- [ ] All benchmark results include both proxy and ground-truth metrics
+- [ ] CI/CD checks ensure warnings are present in all reports
+- [ ] Documentation clearly distinguishes proxy metrics from validated protection
+- [ ] User-facing materials never claim "Strong" protection without ground-truth validation
+
+## Additional Context
+
+This issue is **not about removing the protection score**—it serves a useful purpose for rapid iteration during development. The issue is:
+1. **Over-reliance** on the proxy without ground-truth validation
+2. **Misleading presentation** that implies the score measures actual protection
+3. **Missing correlation studies** to calibrate the proxy against real mimicry
+
+The fix is to:
+- **Validate** the proxy against ground-truth outcomes
+- **Recalibrate** thresholds based on empirical data
+- **Report honestly** about what the metric actually measures
+- **Extend** evaluation to feature spaces that mimicry models actually use
+
+## References
+
+- `/src/auralock/core/metrics.py:247-329` - Protection score implementation
+- `/src/auralock/core/style.py:217-231` - Limited transform suite
+- `/src/auralock/core/style.py:26-101` - Single ResNet18 extractor
+- `/src/auralock/benchmarks/lora.py` - Unused ground-truth infrastructure
+- README.md lines 52-60 - Unvalidated protection claims
diff --git a/benchmark-audit-issues/issue-2-missing-ground-truth-validation.md b/benchmark-audit-issues/issue-2-missing-ground-truth-validation.md
new file mode 100644
index 0000000..392efc2
--- /dev/null
+++ b/benchmark-audit-issues/issue-2-missing-ground-truth-validation.md
@@ -0,0 +1,337 @@
+# Issue: Missing Ground-Truth LoRA/DreamBooth Validation - Benchmark Infrastructure Exists But Unused
+
+## Labels
+`benchmark`, `critical`, `validation`, `research`, `missing-feature`
+
+## Problem Description
+
+The repository contains complete infrastructure for real-world LoRA/DreamBooth mimicry benchmarking (`/src/auralock/benchmarks/lora.py`, `/src/auralock/benchmarks/antidreambooth.py`) **but provides zero published validation results**. All protection claims are based on proxy metrics without demonstrating that the protection actually prevents style mimicry when evaluated against real generative models.
+
+## What is Wrong with the Benchmark
+
+### 1. Infrastructure vs Results Gap
+
+**Infrastructure Present**:
+```python
+# /src/auralock/benchmarks/lora.py:239-442
+class LoraBenchmarkHarness:
+ """Prepare and optionally execute a real DreamBooth/LoRA benchmark workflow."""
+
+ def run(self, ..., execute: bool = False, ...):
+ # Can prepare protected datasets
+ # Can build training commands
+ # Can execute real LoRA training
+ # Can run inference tests
+```
+
+**Reality**:
+- `execute=False` by default (line 320)
+- No published results in README
+- No validation data in repository
+- All claims based on proxy metrics
+
+### 2. Current "Benchmark" is Actually Metric Comparison
+
+What the repository calls "benchmark" (`auralock benchmark`):
+```python
+# /src/auralock/services/protection.py:172-234
+def benchmark_file(...):
+ # Compare PSNR, SSIM, protection_score across profiles
+ # Does NOT measure actual mimicry prevention
+ # Returns proxy metrics only
+```
+
+This is **profile comparison**, not **mimicry prevention validation**.
+
+### 3. No Evidence That Protection Works
+
+README claims (lines 52-58):
+```
+| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
+```
+
+**Questions without answers**:
+- Can DreamBooth still learn style from "fortress" protected images? **Unknown**
+- How many LoRA training runs succeed vs fail after protection? **Not measured**
+- What is the success rate degradation: clean (baseline) vs protected? **No data**
+
+### 4. Preflight System Exists But No Follow-Through
+
+```python
+# /src/auralock/benchmarks/lora.py:107-176
+def evaluate_lora_preflight(...):
+ """Check whether the current machine can run a real LoRA benchmark."""
+ # Checks CUDA availability
+ # Validates required modules (diffusers, accelerate, transformers, peft)
+ # Validates script paths and model directories
+ # Returns ready/not_ready status
+```
+
+**But**: The repository never publishes results from machines where `preflight.ready == True`.
+
+## Why This Can Mislead Users
+
+### Scenario 1: Unvalidated Protection Claims
+
+User reads README:
+```
+| `blindfold` | `61.1` | `26.53` | `0.6114` | strongest current anti-readability preset |
+```
+
+User assumes:
+- "61.1 protection score means DreamBooth can't learn my art style"
+- "This has been validated against real mimicry models"
+
+Reality:
+- No validation against actual LoRA/DreamBooth training
+- The "61.1" is just feature drift in ResNet18 space
+- Real-world effectiveness is **completely unknown**
+
+### Scenario 2: False Security
+
+Artists deploy protection in production based on:
+- High proxy scores
+- Professional-looking benchmark infrastructure in code
+- Assumption that "if they built it, they must have tested it"
+
+Meanwhile:
+- Zero published validation results
+- No evidence protection survives actual mimicry training
+- Infrastructure exists but is never executed with `execute=True`
+
+### Scenario 3: Wasted Development Effort
+
+Developers tune profiles to maximize protection_score without knowing:
+- Does higher protection_score → lower mimicry success rate? **Unmeasured correlation**
+- Which profile actually prevents mimicry best? **No comparative validation**
+- Are visual quality trade-offs worthwhile? **No evidence of actual protection benefit**
+
+## Evidence from Repository
+
+### 1. Benchmark Infrastructure Exists
+
+**LoRA Benchmark** (`/src/auralock/benchmarks/lora.py`):
+- Lines 239-442: Full harness implementation
+- Lines 179-219: Training command builder for `accelerate launch`
+- Lines 222-236: Inference command builder
+- Lines 44-80: Configuration dataclass with all hyperparameters
+
+**Anti-DreamBooth Benchmark** (`/src/auralock/benchmarks/antidreambooth.py`):
+- Lines 152-330: Subject split benchmark harness
+- Lines 110-149: Paper-style set_A/set_B/set_C layout resolver
+- Lines 22-27: Default scripts and prompts configured
+
+**Docker Runtime** (`/src/auralock/benchmarks/docker_runtime.py`):
+- GPU-accelerated containerized benchmark execution
+- Supports distributed training with configurable GPU counts
+
+### 2. But Zero Published Results
+
+**Search for validation data**:
+```bash
+# No results directory
+$ ls /home/runner/work/Lock-ART./Lock-ART./results
+# (does not exist)
+
+# No benchmark results in docs
+$ grep -r "mimicry_success_rate" docs/
+# (no matches)
+
+# No published training logs
+$ find . -name "*lora*results*" -o -name "*dreambooth*results*"
+# (no matches)
+```
+
+**Test suite validates infrastructure but not outcomes**:
+```python
+# /src/tests/test_lora_benchmark.py
+def test_lora_preflight_ready_state():
+ # Tests that preflight detection works
+ # Does NOT test that protection prevents mimicry
+
+def test_lora_manifest_generation():
+ # Tests manifest structure
+ # Does NOT measure mimicry prevention
+```
+
+### 3. Colab Notebook Exists But No Results
+
+`/notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb`:
+- Designed for GPU execution on Google Colab
+- **But no published outputs or results from the notebook**
+- README mentions it (line 231) but doesn't link to results
+
+### 4. Acknowledged Gap
+
+README line 60:
+> "It is useful for relative comparisons inside this repository, not as a universal guarantee against all AI systems."
+
+RESEARCH_ROADMAP.md lines 48-51:
+```
+### Giai đoạn 3 (dài hạn: 2-3 tháng)
+- Chạy benchmark thực tế trên GPU (LoRA/DreamBooth)
+- Tổng hợp kết quả có kiểm định cơ bản
+- Đề xuất hướng cải tiến objective dựa trên dữ liệu thực nghiệm
+```
+
+Translation: "Phase 3 (long-term: 2-3 months) - Run real GPU benchmarks, aggregate results, propose improvements based on experimental data."
+
+**This is still in the roadmap, meaning it hasn't been done yet.**
+
+## Proposed Benchmark Upgrade
+
+### Phase 1: Minimal Validation Study
+
+**Objective**: Generate first ground-truth results to validate proxy metrics.
+
+**Protocol**:
+```yaml
+dataset:
+ subjects: 10 diverse artworks (portraits, landscapes, abstract, anime)
+ source: Public domain or author-owned images
+
+profiles_to_test:
+ - clean (unprotected baseline)
+ - balanced
+ - strong
+ - fortress
+
+mimicry_model:
+ base_model: stabilityai/stable-diffusion-2-1-base
+ method: DreamBooth LoRA
+ training_steps: 400
+ resolution: 512
+ batch_size: 1
+ learning_rate: 1e-4
+
+evaluation_per_profile:
+ 1. Train LoRA model on protected images
+ 2. Generate 20 samples with trained model
+ 3. Measure mimicry success:
+ - Human evaluation (3 raters): "Does output match original style?"
+ - CLIP similarity: generated_samples vs original_style
+ - FID score: generated_distribution vs original_distribution
+ 4. Report success rate: percentage of successful mimicry attempts
+
+expected_output:
+ - Table: profile → mimicry_success_rate
+ - Analysis: correlation(protection_score, mimicry_prevention)
+ - Recommendation: threshold recalibration
+```
+
+**Resource Requirements**:
+- 10 subjects × 5 profiles = 50 training runs
+- ~15-30 minutes per training run on V100 = 12-25 GPU hours
+- Cost estimate: $25-50 on Google Colab Pro (affordable validation)
+
+### Phase 2: Comprehensive Validation Suite
+
+Expand validation to:
+- **N=50 subjects** across diverse styles
+- **Multiple mimicry methods**: DreamBooth, LoRA, Textual Inversion, IP-Adapter
+- **Robustness testing**: Evaluate with common preprocessing (JPEG, resize, crop)
+- **Comparison baseline**: Academic anti-mimicry methods (Anti-DreamBooth, Mist, Glaze if reproducible)
+
+### Phase 3: Continuous Validation
+
+Integrate ground-truth validation into CI/CD:
+```yaml
+# .github/workflows/validation.yml
+- name: Monthly GPU Validation
+ # Run subset of validation tests on GPU runner
+ # Update results dashboard
+ # Alert if protection effectiveness degrades
+```
+
+### Phase 4: Public Results Dashboard
+
+Create `docs/VALIDATION_RESULTS.md`:
+```markdown
+# Ground-Truth Validation Results
+
+## Last Updated: 2024-XX-XX
+
+### Protection Effectiveness vs DreamBooth LoRA
+
+| Profile | Protection Score | Mimicry Success Rate | Samples |
+|-----------|------------------|---------------------|---------|
+| clean | 0.0 | 95% (baseline) | 50 |
+| balanced | 42.1 | 78% ⚠️ | 50 |
+| strong | 48.7 | 61% | 50 |
+| fortress | 53.2 | 45% | 50 |
+| blindfold | 61.1 | 32% | 50 |
+
+**Key Findings**:
+- Correlation: R²=0.83 (strong correlation)
+- But even "blindfold" allows 32% mimicry success
+- Visual quality trade-off may not be worthwhile for moderate profiles
+```
+
+## Acceptance Criteria
+
+### Phase 1: Minimal Validation (Month 1)
+- [ ] Execute `auralock benchmark-lora` with `--execute` on at least 10 subjects
+- [ ] Document GPU setup and training hyperparameters
+- [ ] Collect mimicry success rate measurements
+- [ ] Publish results in `docs/VALIDATION_RESULTS.md`
+- [ ] Calculate correlation: protection_score vs mimicry_prevention
+- [ ] Update README to include ground-truth results alongside proxy metrics
+
+### Phase 2: Expanded Validation (Month 2-3)
+- [ ] Scale to N=50 subjects
+- [ ] Test multiple mimicry methods (DreamBooth, LoRA, Textual Inversion)
+- [ ] Compare against academic baselines (Anti-DreamBooth, Mist)
+- [ ] Publish validation methodology in academic paper or technical report
+
+### Phase 3: Transparency Upgrades (Ongoing)
+- [ ] All benchmark claims must include ground-truth validation status:
+ ```
+ ✅ Validated: Tested against real LoRA training (N=50, success_rate=32%)
+ ⚠️ Proxy only: Not yet validated against real mimicry models
+ ```
+- [ ] CLI output distinguishes proxy vs ground-truth metrics
+- [ ] README never claims protection without validation evidence
+
+### Phase 4: Infrastructure Improvements
+- [ ] Add `benchmark_lora_batch.py` script for easy multi-profile validation
+- [ ] Create Colab notebook with **executed cells and outputs** (not blank template)
+- [ ] Document cost estimation tool: "How much GPU time needed for N subjects?"
+- [ ] Provide validation results reproduction guide
+
+## Additional Context
+
+### Why This Hasn't Been Done Yet
+
+Understandable reasons:
+1. **GPU costs**: Real validation requires expensive GPU hours
+2. **Time**: Training 50 LoRA models takes significant compute time
+3. **Complexity**: Managing training runs, collecting outputs, analyzing results
+
+**But**: This is **mandatory for credible benchmark claims**. You cannot claim anti-mimicry protection without measuring actual mimicry prevention.
+
+### What Makes This Critical
+
+From research perspective:
+- **Scientific rigor**: You can't publish claims without validation
+- **User trust**: Artists need evidence, not proxy scores
+- **Development direction**: Need ground-truth feedback to improve methods
+
+From product perspective:
+- **False advertising risk**: Claiming protection without validation
+- **User harm**: Artists deploy ineffective protection, lose IP
+- **Reputation damage**: When protection fails, trust in project collapses
+
+### Recommended Next Action
+
+**Start small**: Run the Phase 1 minimal validation (10 subjects, 5 profiles, DreamBooth only). This costs ~$50 and takes a weekend. Publish honest results, even if protection is weak. **Honesty builds trust more than perfect scores.**
+
+Then iterate based on data.
+
+## References
+
+- `/src/auralock/benchmarks/lora.py` - Complete but unused benchmark harness
+- `/src/auralock/benchmarks/antidreambooth.py` - Subject split benchmark infrastructure
+- `/notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb` - GPU execution template (no results)
+- `/docs/RESEARCH_ROADMAP.md` lines 48-51 - Acknowledges GPU validation is still pending
+- README.md lines 52-58 - Unvalidated protection claims
+- README.md line 60 - Buried disclaimer about proxy nature
diff --git a/benchmark-audit-issues/issue-3-weak-robustness-testing.md b/benchmark-audit-issues/issue-3-weak-robustness-testing.md
new file mode 100644
index 0000000..561b619
--- /dev/null
+++ b/benchmark-audit-issues/issue-3-weak-robustness-testing.md
@@ -0,0 +1,438 @@
+# Issue: Weak Robustness Testing - Missing Critical Preprocessing Transformations
+
+## Labels
+`benchmark`, `robustness`, `security`, `technical-debt`, `enhancement`
+
+## Problem Description
+
+The robustness testing suite includes only **4 basic transforms** (identity, gaussian blur, 2 resize operations), missing critical preprocessing steps that real-world mimicry pipelines actually use. This creates a false sense of robustness—protection may survive gaussian blur but fail against JPEG compression, center crops, or CLIP preprocessing that DreamBooth/LoRA models apply.
+
+## What is Wrong with the Benchmark
+
+### 1. Limited Transform Suite
+
+**Current Implementation** (`/src/auralock/core/style.py:217-231`):
+```python
+def build_style_transform_suite() -> tuple[tuple[str, StyleTransform], ...]:
+ return (
+ ("identity", identity),
+ ("gaussian_blur", lambda images: gaussian_blur(images, kernel_size=5, sigma=1.0)),
+ ("resize_restore_75", lambda images: resize_restore(images, scale=0.75)),
+ ("resize_restore_50", lambda images: resize_restore(images, scale=0.5)),
+ )
+```
+
+**What's Missing**:
+1. **JPEG Compression**: DreamBooth training often involves JPEG-compressed images from web scraping or dataset curation
+2. **Center/Random Crop**: Standard data augmentation in training pipelines
+3. **CLIP Preprocessing**: Mimicry models use CLIP for guidance, which applies specific normalization and resizing
+4. **VAE Encode/Decode**: Latent diffusion models encode images through VAE, which introduces information loss
+5. **Color Jitter**: Common augmentation (brightness, contrast, saturation, hue shifts)
+6. **Rotation**: Small angle rotations are standard augmentation
+7. **Noise Injection**: Training often adds small Gaussian noise for regularization
+
+### 2. Proxy for Real Transformations
+
+**Resize-Restore is Not JPEG**:
+```python
+# Current (line 193-214)
+def resize_restore(images: torch.Tensor, scale: float = 0.75):
+ # Bilinear downscale → bilinear upscale
+ # Used as proxy for compression
+```
+
+**Problem**: Bilinear resize ≠ JPEG compression artifacts
+- JPEG introduces block artifacts (DCT quantization)
+- JPEG has chroma subsampling (4:2:0)
+- JPEG quality levels vary (10-100), each with different artifact patterns
+- Protection robust to bilinear resize may **fail completely** against JPEG artifacts
+
+### 3. Gaussian Blur Parameter Space Unexplored
+
+**Current** (line 226-227):
+```python
+("gaussian_blur", lambda images: gaussian_blur(images, kernel_size=5, sigma=1.0))
+```
+
+**Single configuration tested**:
+- Only `kernel_size=5, sigma=1.0`
+- No exploration of stronger blurs (sigma=2.0, 3.0)
+- No exploration of different kernel sizes (3, 7, 9)
+
+**Real-world variation**: Images may be blurred with various strengths. Testing only one configuration doesn't validate robustness.
+
+### 4. No Adversarial Purification Defenses
+
+Common purification techniques attackers use to remove perturbations:
+- **JPEG compression at various quality levels** (most effective against adversarial perturbations)
+- **Denoising autoencoders**
+- **Total variation minimization**
+- **Median filtering**
+- **Bilateral filtering**
+
+**None of these are tested in the robustness suite.**
+
+## Why This Can Mislead Users
+
+### Scenario 1: False Robustness Claims
+
+User sees in report:
+```json
+{
+ "robust_style_similarity": 0.72,
+ "robust_embedding_similarity": 0.68,
+ "protection_score": 42.1,
+ "assessment": "Strong"
+}
+```
+
+User assumes:
+- "My protection survives all common preprocessing"
+- "42.1 score is robust to mimicry pipeline transformations"
+
+Reality:
+- Protection only tested against 4 transforms
+- JPEG compression at quality=70 may **completely remove perturbations**
+- Center crop may destroy spatially-localized protection patterns
+- Assessment is "Strong" in a limited test environment, not real-world
+
+### Scenario 2: Purification Attack Vulnerability
+
+Attacker applies simple purification before training:
+```python
+# Attacker's preprocessing (NOT tested by AuraLock)
+def purify_protected_image(img):
+ # JPEG compression at quality=85
+ img = apply_jpeg_compression(img, quality=85)
+ # Center crop + resize
+ img = center_crop_and_resize(img, crop_size=0.9)
+ # Mild denoising
+ img = cv2.fastNlMeansDenoisingColored(img)
+ return img
+
+# Now train DreamBooth on purified images
+# Protection may be significantly weakened or removed
+```
+
+**AuraLock's robustness testing wouldn't catch this vulnerability.**
+
+### Scenario 3: CLIP/VAE Feature Space Mismatch
+
+DreamBooth uses CLIP features for guidance:
+```python
+# Actual mimicry pipeline
+clip_features = clip_model.encode_image(
+ clip_preprocessing(protected_image) # Specific resizing + normalization
+)
+vae_latents = vae.encode(
+ vae_preprocessing(protected_image) # Different preprocessing
+).latent_dist.sample()
+```
+
+**But AuraLock tests robustness in ResNet18 feature space with ResNet18 preprocessing.** Protection robust in one space may be fragile in another.
+
+## Evidence from Repository
+
+### 1. Transform Suite is Hardcoded and Minimal
+
+Search for all transform definitions:
+```bash
+$ grep -n "build_style_transform_suite" /src/auralock/core/style.py
+217:def build_style_transform_suite() -> tuple[tuple[str, StyleTransform], ...]:
+```
+
+Only one function, one implementation, 4 transforms. No extension mechanism, no parameterization.
+
+### 2. No JPEG Implementation
+
+Search for JPEG compression:
+```bash
+$ grep -r "jpeg" src/ --ignore-case
+# (no matches in core robustness code)
+
+$ grep -r "compression" src/
+# (no matches related to robustness testing)
+```
+
+**No JPEG compression testing exists in the robustness suite.**
+
+### 3. No Crop Testing
+
+Search for cropping:
+```bash
+$ grep -r "crop" src/auralock/core/
+# (no matches)
+```
+
+Center crop, random crop, and resize-crop patterns are standard in training pipelines but **not tested**.
+
+### 4. Acknowledged in Documentation (Partially)
+
+README line 232 mentions robustness testing:
+> "Robustness testing: via blur, resize/restore transforms averaged across multiple scales"
+
+**But doesn't mention the severe limitations**:
+- No JPEG (most critical purification defense)
+- No crop (standard augmentation)
+- No CLIP/VAE preprocessing (actual mimicry pipeline)
+
+### 5. Comment in Code Acknowledges Resize as Proxy
+
+`/src/auralock/core/style.py:193-214`:
+```python
+def resize_restore(images: torch.Tensor, scale: float = 0.75) -> torch.Tensor:
+ """Downscale and restore an image batch to emulate common purification steps."""
+ # ^^^^^ "emulate" = proxy, not actual purification
+```
+
+Resize is used to **emulate** compression, not test against actual compression artifacts.
+
+## Proposed Benchmark Upgrade
+
+### Phase 1: Expand Transform Suite
+
+Replace hardcoded minimal suite with comprehensive testing:
+
+```python
+def build_comprehensive_transform_suite() -> tuple[tuple[str, StyleTransform], ...]:
+ """Transforms covering real-world preprocessing and purification attacks."""
+ return (
+ # Baseline
+ ("identity", identity),
+
+ # Existing (keep for backward compatibility)
+ ("gaussian_blur_mild", lambda x: gaussian_blur(x, kernel_size=5, sigma=1.0)),
+ ("resize_restore_75", lambda x: resize_restore(x, scale=0.75)),
+ ("resize_restore_50", lambda x: resize_restore(x, scale=0.5)),
+
+ # NEW: JPEG Compression (critical)
+ ("jpeg_quality_95", lambda x: jpeg_compress_decompress(x, quality=95)),
+ ("jpeg_quality_85", lambda x: jpeg_compress_decompress(x, quality=85)),
+ ("jpeg_quality_75", lambda x: jpeg_compress_decompress(x, quality=75)),
+ ("jpeg_quality_50", lambda x: jpeg_compress_decompress(x, quality=50)),
+
+ # NEW: Cropping
+ ("center_crop_90", lambda x: center_crop_and_resize(x, crop_ratio=0.9)),
+ ("center_crop_80", lambda x: center_crop_and_resize(x, crop_ratio=0.8)),
+ ("random_crop_90", lambda x: random_crop_and_resize(x, crop_ratio=0.9)),
+
+ # NEW: Stronger blur variants
+ ("gaussian_blur_medium", lambda x: gaussian_blur(x, kernel_size=7, sigma=2.0)),
+ ("gaussian_blur_strong", lambda x: gaussian_blur(x, kernel_size=9, sigma=3.0)),
+
+ # NEW: Color augmentation
+ ("color_jitter_mild", lambda x: color_jitter(x, brightness=0.1, contrast=0.1)),
+ ("color_jitter_medium", lambda x: color_jitter(x, brightness=0.2, contrast=0.2)),
+
+ # NEW: Noise injection
+ ("gaussian_noise_small", lambda x: add_gaussian_noise(x, std=0.01)),
+ ("gaussian_noise_medium", lambda x: add_gaussian_noise(x, std=0.03)),
+
+ # NEW: Mimicry-specific preprocessing
+ ("clip_preprocess", lambda x: apply_clip_preprocessing(x)),
+ ("vae_encode_decode", lambda x: vae_roundtrip(x)),
+ )
+```
+
+### Phase 2: Implement Missing Transforms
+
+**JPEG Compression** (most critical):
+```python
+def jpeg_compress_decompress(
+ images: torch.Tensor,
+ quality: int = 85,
+) -> torch.Tensor:
+ """Apply JPEG compression and decompression to test robustness."""
+ from PIL import Image
+ import io
+
+ batch_size = images.shape[0]
+ results = []
+
+ for i in range(batch_size):
+ # Convert tensor to PIL Image
+ img_np = (images[i].permute(1, 2, 0).cpu().numpy() * 255).astype(np.uint8)
+ img_pil = Image.fromarray(img_np)
+
+ # JPEG compress in memory
+ buffer = io.BytesIO()
+ img_pil.save(buffer, format='JPEG', quality=quality)
+ buffer.seek(0)
+
+ # Decompress
+ img_pil = Image.open(buffer)
+ img_np = np.array(img_pil).astype(np.float32) / 255.0
+ img_tensor = torch.from_numpy(img_np).permute(2, 0, 1)
+ results.append(img_tensor)
+
+ return torch.stack(results).to(images.device)
+```
+
+**Center Crop and Resize**:
+```python
+def center_crop_and_resize(
+ images: torch.Tensor,
+ crop_ratio: float = 0.9,
+) -> torch.Tensor:
+ """Center crop to crop_ratio and resize back to original size."""
+ if not 0.0 < crop_ratio <= 1.0:
+ raise ValueError("crop_ratio must be in (0, 1]")
+
+ _, _, h, w = images.shape
+ crop_h = int(h * crop_ratio)
+ crop_w = int(w * crop_ratio)
+
+ start_h = (h - crop_h) // 2
+ start_w = (w - crop_w) // 2
+
+ cropped = images[:, :, start_h:start_h+crop_h, start_w:start_w+crop_w]
+
+ return F.interpolate(
+ cropped,
+ size=(h, w),
+ mode='bilinear',
+ align_corners=False,
+ antialias=True,
+ )
+```
+
+**CLIP Preprocessing**:
+```python
+def apply_clip_preprocessing(images: torch.Tensor) -> torch.Tensor:
+ """Apply CLIP model's standard preprocessing."""
+ from transformers import CLIPImageProcessor
+
+ processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
+
+ # CLIP expects 224x224 with specific normalization
+ resized = F.interpolate(images, size=(224, 224), mode='bicubic', antialias=True)
+
+ # Apply CLIP normalization
+ mean = torch.tensor([0.48145466, 0.4578275, 0.40821073]).view(1, 3, 1, 1).to(images.device)
+ std = torch.tensor([0.26862954, 0.26130258, 0.27577711]).view(1, 3, 1, 1).to(images.device)
+
+ normalized = (resized - mean) / std
+ return normalized
+```
+
+### Phase 3: Configurable Transform Selection
+
+Allow users to select which transforms to test:
+```python
+# CLI
+$ auralock protect artwork.png -o protected.png \
+ --robustness-suite comprehensive # all transforms
+ # or --robustness-suite mimicry # CLIP, VAE, JPEG only
+ # or --robustness-suite minimal # current 4 transforms (backward compat)
+
+# Programmatic
+from auralock.core.style import TransformSuitePreset
+
+result = service.protect_file(
+ "artwork.png",
+ profile="balanced",
+ robustness_suite=TransformSuitePreset.COMPREHENSIVE,
+)
+```
+
+### Phase 4: Worst-Case Robustness Reporting
+
+Instead of only reporting mean robustness, report:
+```json
+{
+ "protection_report": {
+ "robust_style_similarity_mean": 0.72,
+ "robust_style_similarity_worst": 0.89, // weakest protection
+ "robust_style_similarity_best": 0.55, // strongest protection
+ "worst_case_transform": "jpeg_quality_75",
+ "best_case_transform": "identity",
+ "transform_breakdown": {
+ "identity": 0.55,
+ "gaussian_blur": 0.68,
+ "jpeg_quality_95": 0.71,
+ "jpeg_quality_85": 0.82,
+ "jpeg_quality_75": 0.89, // ⚠️ Protection significantly weakened
+ "center_crop_90": 0.65
+ }
+ }
+}
+```
+
+This reveals vulnerability: "Protection looks good (mean=0.72) but JPEG at quality=75 weakens it severely (0.89)."
+
+## Acceptance Criteria
+
+### Phase 1: JPEG Compression (Critical, Week 1-2)
+- [ ] Implement `jpeg_compress_decompress()` function
+- [ ] Add JPEG quality levels: 95, 85, 75, 50 to transform suite
+- [ ] Test protection against JPEG compression in robustness evaluation
+- [ ] Report JPEG robustness separately in all outputs
+- [ ] Document JPEG vulnerability in README if protection is weak
+
+### Phase 2: Cropping and Augmentation (Week 2-3)
+- [ ] Implement `center_crop_and_resize()` function
+- [ ] Implement `random_crop_and_resize()` function
+- [ ] Implement `color_jitter()` function
+- [ ] Add to transform suite and test
+
+### Phase 3: Mimicry-Aligned Preprocessing (Week 3-4)
+- [ ] Implement `apply_clip_preprocessing()` function
+- [ ] Implement `vae_encode_decode()` function (requires VAE model)
+- [ ] Test protection against actual mimicry preprocessing
+- [ ] Compare robustness in ResNet18 space vs CLIP/VAE space
+
+### Phase 4: Comprehensive Suite and Reporting (Week 4-5)
+- [ ] Create `TransformSuitePreset` enum: MINIMAL, STANDARD, COMPREHENSIVE, MIMICRY
+- [ ] CLI flag: `--robustness-suite {minimal,standard,comprehensive,mimicry}`
+- [ ] Report worst-case robustness alongside mean
+- [ ] Highlight vulnerable transforms in warnings
+
+### Phase 5: Validation and Documentation (Week 5-6)
+- [ ] Run comprehensive robustness tests on all profiles
+- [ ] Publish robustness comparison table: profile → worst_case_transform + weakness
+- [ ] Update README with honest robustness limitations
+- [ ] Document purification attack resistance (or lack thereof)
+
+## Additional Context
+
+### Why JPEG is Most Critical
+
+From adversarial robustness literature:
+- **JPEG compression is the most effective purification defense** against adversarial perturbations
+- Quality level 75-85 removes most imperceptible perturbations
+- But also degrades image quality noticeably
+
+If protection doesn't survive JPEG compression at quality 85:
+- Users can't share protected images on platforms that recompress (Facebook, Instagram)
+- Attackers can trivially remove protection by applying JPEG compression
+- Protection is effectively useless in real-world scenarios
+
+### Resize vs JPEG: Not Equivalent
+
+Resize-restore (bilinear) simulates information loss but not artifact patterns:
+- Bilinear: smooth, continuous interpolation
+- JPEG: block artifacts, ringing, chroma subsampling
+
+Protection may survive smooth information loss but fail against structured artifacts.
+
+### Defense in Depth
+
+Strong protection should survive **combinations** of transforms:
+```python
+# Real-world scenario
+def realistic_preprocessing(img):
+ img = jpeg_compress(img, quality=85) # Platform recompression
+ img = center_crop(img, ratio=0.95) # User crops before upload
+ img = resize(img, target_size=512) # Training pipeline resize
+ return img
+```
+
+Current testing doesn't evaluate transform combinations.
+
+## References
+
+- `/src/auralock/core/style.py:217-231` - Current minimal transform suite
+- `/src/auralock/core/style.py:193-214` - Resize-restore proxy for compression
+- `/src/auralock/core/metrics.py:286-300` - Robustness evaluation loop
+- Adversarial robustness literature on purification defenses
+- DreamBooth/LoRA data preprocessing pipelines
diff --git a/benchmark-audit-issues/issue-4-dataset-split-methodology-risks.md b/benchmark-audit-issues/issue-4-dataset-split-methodology-risks.md
new file mode 100644
index 0000000..f38cc1b
--- /dev/null
+++ b/benchmark-audit-issues/issue-4-dataset-split-methodology-risks.md
@@ -0,0 +1,536 @@
+# Issue: Dataset Split Methodology Risks - Missing Train/Val/Test Separation
+
+## Labels
+`benchmark`, `data-leakage`, `methodology`, `research`, `technical-debt`
+
+## Problem Description
+
+The repository lacks clear separation between training/validation/test splits, creating risk of **data leakage**, **overfitting to test sets**, and **cherry-picking results**. The Anti-DreamBooth subject split (set_A/set_B/set_C) exists but its usage is not enforced or validated, and the local benchmark workflow has no split separation at all.
+
+## What is Wrong with the Benchmark
+
+### 1. Local Benchmark Has No Split Methodology
+
+**Current workflow** (`auralock benchmark`):
+```python
+# /src/auralock/services/protection.py:172-234
+def benchmark_file(image_path, profiles, ...):
+ # Apply each profile to the SAME image
+ # Report metrics on the SAME image
+ # No train/validation/test split
+```
+
+**Problem**: If you tune profiles based on benchmark results, you're **training on your test set**.
+
+Example workflow that causes overfitting:
+```bash
+# Step 1: Benchmark current profiles
+$ auralock benchmark artwork.png --profiles safe,balanced,strong
+
+# Step 2: See that "safe" has low protection score
+# Step 3: Tune "safe" profile parameters to increase score on artwork.png
+# Step 4: Benchmark again on artwork.png
+# Step 5: Claim improvement
+
+# Problem: You overfitted to artwork.png. No generalization guarantee.
+```
+
+### 2. Anti-DreamBooth Split Usage Not Validated
+
+**Infrastructure exists** (`/src/auralock/benchmarks/antidreambooth.py:110-149`):
+```python
+def resolve_subject_layout(subject_root: str | Path):
+ # Validates set_A, set_B, set_C exist
+ # Returns layout with image paths for each split
+
+ # set_A: Reference/tuning split (clean)
+ # set_B: Published/training split (protection target)
+ # set_C: Holdout validation split
+```
+
+**But no enforcement**:
+- Nothing prevents training on set_C (holdout)
+- Nothing validates that set_A/B/C are truly independent
+- No checks for duplicate images across splits
+- No documentation on split ratios or methodology
+
+**From code** (`/src/auralock/benchmarks/antidreambooth.py:316-319`):
+```python
+notes = [
+ "set_A is retained as a clean reference split, set_B is treated as the published split, and set_C is preserved as holdout metadata.",
+]
+```
+
+"Preserved as holdout **metadata**" suggests set_C is not being used for held-out evaluation—just kept as metadata. **This defeats the purpose of a holdout set.**
+
+### 3. No Protection Against Cherry-Picking
+
+**Nothing prevents**:
+```bash
+# Run benchmark on 100 artworks
+$ for img in artworks/*.png; do
+ auralock benchmark $img --profiles balanced --report reports/${img}.json
+done
+
+# Cherry-pick the 10 best results
+$ python select_best_results.py --top 10
+
+# Publish only the cherry-picked results in README
+```
+
+Users see impressive protection scores, unaware that:
+- 90 artworks had poor protection (not shown)
+- Results are not representative of typical performance
+- Selection bias inflates reported effectiveness
+
+### 4. Collective Protection Without Split Validation
+
+**Collective mode** (`/src/auralock/services/protection.py:296-385`):
+```python
+def _protect_directory_collectively(...):
+ # Compute batch-level perturbation
+ # Share 30% of perturbation across all images
+ # Ensure consistency across subject sets
+```
+
+**Problem**: If you benchmark collective protection on the same set you protected, you're measuring **in-sample** performance. No guarantee of **out-of-sample** effectiveness.
+
+**Correct methodology**:
+1. Apply collective protection to training split (set_B)
+2. Measure protection score on held-out validation split (set_C)
+3. Report out-of-sample metrics
+
+**Current practice**: No evidence this is being done.
+
+### 5. Benchmark Reports Don't Track Data Lineage
+
+**Report structure** (`/src/auralock/services/protection.py:19-72`):
+```python
+@dataclass
+class BenchmarkSummary:
+ input_path: Path
+ image_count: int
+ entries: list[BenchmarkEntry]
+ profile_summaries: dict[str, ProfileAggregate]
+
+ # Missing:
+ # - split: "train" | "val" | "test"
+ # - split_metadata: hash of split assignment
+ # - parent_dataset: name of dataset split belongs to
+```
+
+Without tracking split membership, you can't:
+- Verify results are from held-out test sets
+- Reproduce split assignments
+- Audit for data leakage
+
+## Why This Can Mislead Users
+
+### Scenario 1: Overfitting to Benchmark Images
+
+Developer workflow:
+```python
+# Week 1: Benchmark on portrait.png
+balanced_score = 42.1 # Low protection
+
+# Week 2: Tune balanced profile to maximize protection on portrait.png
+# Increase epsilon from 0.02 to 0.025, adjust alpha, increase steps
+
+# Week 3: Benchmark again on portrait.png
+balanced_score = 48.7 # "Improvement"!
+
+# Publish claim: "We improved protection by 15.7%"
+```
+
+**Problem**: Improvement is specific to portrait.png. May not generalize to other artworks. **No held-out validation performed.**
+
+### Scenario 2: Cherry-Picking Favorable Results
+
+Marketing workflow:
+```python
+# Benchmark 100 diverse artworks
+results = []
+for artwork in diverse_dataset:
+ result = benchmark(artwork, profile="fortress")
+ results.append(result)
+
+# Select top 5 results where protection_score > 55
+best_results = [r for r in results if r.protection_score > 55][:5]
+
+# Publish only best_results in README
+# README table shows: fortress achieves 55-62 protection scores
+```
+
+**Users don't know**:
+- 95% of artworks had lower protection (30-45 range)
+- Results are cherry-picked outliers
+- Typical performance is much worse than advertised
+
+### Scenario 3: Training on Test Set
+
+Research workflow:
+```python
+# Collect dataset of 50 artworks
+# Run comprehensive benchmark to find best hyperparameters
+# Iterate 20 times, adjusting parameters based on benchmark results
+# Publish final "best" configuration
+
+# Problem: All 50 artworks have been seen during tuning
+# "Best" configuration is overfit to this specific dataset
+# No held-out test set to validate generalization
+```
+
+When users apply the "best" profile to their artwork:
+- Performance may be significantly worse
+- Published benchmarks were overfit to development set
+- **No true test set evaluation was performed**
+
+## Evidence from Repository
+
+### 1. No Split Enforcement in Benchmarking Code
+
+Search for train/val/test split logic:
+```bash
+$ grep -r "train.*split\|val.*split\|test.*split" src/
+# (no matches)
+
+$ grep -r "holdout" src/
+# Only in antidreambooth.py as comment, not enforced
+```
+
+**No code enforces split separation.**
+
+### 2. Benchmark Reports Don't Track Splits
+
+Examine report dataclasses:
+```python
+# /src/auralock/services/protection.py:19-72
+@dataclass
+class BenchmarkSummary:
+ # ... fields ...
+ # No "split" field
+ # No "split_hash" field
+ # No validation that images are from held-out set
+```
+
+**Reports don't document which split was used.**
+
+### 3. Anti-DreamBooth Split is "Metadata" Only
+
+From `/src/auralock/benchmarks/antidreambooth.py:318`:
+```python
+"set_C is preserved as holdout metadata."
+```
+
+"Metadata" suggests set_C is not being actively used for held-out evaluation. **This is a red flag.**
+
+### 4. README Benchmark Results Have No Split Info
+
+README lines 52-58:
+```
+| Run | Protection Score | PSNR | SSIM | Notes |
+|-----|------------------|------|------|-------|
+| `balanced` | `42.1` | `36.24` | `0.9346` | better visual quality, good study baseline |
+```
+
+**Missing information**:
+- Which images were these scores measured on?
+- Were images part of development/tuning process?
+- Are these in-sample or out-of-sample results?
+- What dataset? How many images?
+
+**Cannot verify if results are from held-out test set.**
+
+### 5. Documentation Acknowledges Need But Doesn't Implement
+
+`/docs/system-design/10_BENCHMARK_DESIGN.md:20-22`:
+```markdown
+## Anti-bias
+- Không cherry-pick ảnh đẹp nhất.
+- Tách tập tune/test.
+```
+
+Translation: "Don't cherry-pick best images. Separate tune/test sets."
+
+**This is acknowledged as needed but not implemented or enforced.**
+
+## Proposed Benchmark Upgrade
+
+### Phase 1: Formalize Split Methodology
+
+Create `SplitMetadata` dataclass:
+```python
+from dataclasses import dataclass
+from enum import Enum
+
+class SplitType(Enum):
+ TRAIN = "train"
+ VALIDATION = "val"
+ TEST = "test"
+ DEVELOPMENT = "dev" # Used for tuning, not final eval
+
+@dataclass
+class SplitMetadata:
+ split_type: SplitType
+ dataset_name: str
+ dataset_version: str
+ split_hash: str # Deterministic hash of split assignment
+ image_ids: list[str] # Unique IDs of images in this split
+ split_method: str # "random", "stratified", "subject_based"
+ split_ratio: dict[str, float] # {"train": 0.7, "val": 0.15, "test": 0.15}
+ random_seed: int | None # For reproducibility
+
+ def verify_no_leakage(self, other: 'SplitMetadata') -> bool:
+ """Check that no images overlap between splits."""
+ return set(self.image_ids).isdisjoint(set(other.image_ids))
+```
+
+### Phase 2: Enforce Split Separation in Benchmarking
+
+Update benchmark functions to require split metadata:
+```python
+# /src/auralock/services/protection.py
+
+def benchmark_file(
+ image_path: str | Path,
+ *,
+ split_metadata: SplitMetadata, # NEW: Required
+ profiles: tuple[str, ...] = ("safe", "balanced", "strong"),
+ ...
+) -> BenchmarkSummary:
+ """Benchmark profiles on a file with split tracking."""
+
+ # Validate image belongs to correct split
+ if str(image_path) not in split_metadata.image_ids:
+ raise ValueError(
+ f"Image {image_path} not found in declared split {split_metadata.split_type}. "
+ "Potential data leakage."
+ )
+
+ # Enforce: only TEST split allowed for final benchmark reporting
+ if split_metadata.split_type != SplitType.TEST:
+ warnings.warn(
+ f"Benchmarking on {split_metadata.split_type} split. "
+ "Results may be overfit. Use TEST split for final evaluation."
+ )
+
+ # ... existing benchmark logic ...
+
+ # Add split metadata to report
+ summary.split_metadata = split_metadata
+ return summary
+```
+
+### Phase 3: Implement Dataset Split Utilities
+
+Create `auralock.benchmarks.splits` module:
+```python
+def create_random_split(
+ image_paths: list[Path],
+ *,
+ train_ratio: float = 0.7,
+ val_ratio: float = 0.15,
+ test_ratio: float = 0.15,
+ random_seed: int = 42,
+) -> dict[SplitType, SplitMetadata]:
+ """Create random train/val/test split with metadata."""
+
+ if not abs((train_ratio + val_ratio + test_ratio) - 1.0) < 1e-6:
+ raise ValueError("Split ratios must sum to 1.0")
+
+ random.seed(random_seed)
+ shuffled = random.sample(image_paths, len(image_paths))
+
+ n_train = int(len(shuffled) * train_ratio)
+ n_val = int(len(shuffled) * val_ratio)
+
+ train_images = shuffled[:n_train]
+ val_images = shuffled[n_train:n_train + n_val]
+ test_images = shuffled[n_train + n_val:]
+
+ return {
+ SplitType.TRAIN: SplitMetadata(
+ split_type=SplitType.TRAIN,
+ image_ids=[str(p) for p in train_images],
+ split_method="random",
+ random_seed=random_seed,
+ # ... other fields ...
+ ),
+ # ... VAL and TEST ...
+ }
+
+
+def save_split_manifest(splits: dict[SplitType, SplitMetadata], output_path: Path):
+ """Save split assignments for reproducibility."""
+ manifest = {
+ split_type.value: split_meta.to_dict()
+ for split_type, split_meta in splits.items()
+ }
+ with open(output_path, 'w') as f:
+ json.dump(manifest, f, indent=2)
+
+
+def load_split_manifest(manifest_path: Path) -> dict[SplitType, SplitMetadata]:
+ """Load previously saved split assignments."""
+ with open(manifest_path) as f:
+ manifest = json.load(f)
+ return {
+ SplitType(key): SplitMetadata.from_dict(value)
+ for key, value in manifest.items()
+ }
+```
+
+### Phase 4: CLI Commands for Split Management
+
+```bash
+# Create and save split manifest
+$ auralock split create ./dataset --output splits.json \
+ --train-ratio 0.7 --val-ratio 0.15 --test-ratio 0.15 \
+ --seed 42
+
+# Validate no leakage
+$ auralock split validate splits.json
+
+# Benchmark only on test split
+$ auralock benchmark ./dataset \
+ --split-manifest splits.json \
+ --split-type test \
+ --profiles balanced,strong \
+ --report test_results.json
+
+# Attempt to benchmark on train split -> warning
+$ auralock benchmark ./dataset \
+ --split-manifest splits.json \
+ --split-type train \
+ --profiles balanced
+
+⚠️ WARNING: Benchmarking on TRAIN split. Results may be overfit.
+ Use --split-type test for final evaluation.
+```
+
+### Phase 5: Validation Checks in CI/CD
+
+Add split validation to CI:
+```yaml
+# .github/workflows/benchmark-validation.yml
+- name: Validate Split Methodology
+ run: |
+ # Check that published benchmark results are from TEST split
+ python scripts/validate_published_results.py
+
+ # Verify no data leakage
+ python scripts/check_split_leakage.py --splits splits.json
+
+ # Ensure benchmark reports include split metadata
+ python scripts/audit_benchmark_reports.py --reports reports/
+```
+
+### Phase 6: Update Anti-DreamBooth to Use set_C Correctly
+
+**Current**: set_C is "metadata"
+
+**Proposed**: Use set_C for held-out evaluation:
+```python
+# /src/auralock/benchmarks/antidreambooth.py
+
+class AntiDreamBoothSubjectBenchmarkHarness:
+ def run_with_holdout_validation(self, ...):
+ # Step 1: Protect set_B (published split)
+ self._prepare_protected_split(layout.set_b_dir, ...)
+
+ # Step 2: Train on protected set_B
+ train_on_protected_set_B(...)
+
+ # Step 3: Evaluate mimicry success on held-out set_C
+ holdout_metrics = evaluate_mimicry_on_holdout(
+ trained_model,
+ holdout_images=layout.set_c_images,
+ )
+
+ # Step 4: Report out-of-sample mimicry prevention
+ return {
+ "protected_training_set": "set_B",
+ "holdout_evaluation_set": "set_C",
+ "holdout_mimicry_success_rate": holdout_metrics.success_rate,
+ "holdout_protection_score": holdout_metrics.protection_score,
+ }
+```
+
+## Acceptance Criteria
+
+### Phase 1: Split Metadata Infrastructure (Week 1)
+- [ ] Implement `SplitMetadata` dataclass
+- [ ] Implement `create_random_split()` utility
+- [ ] Implement split manifest save/load functions
+- [ ] Add tests for split creation and validation
+
+### Phase 2: Enforce Split Separation (Week 2)
+- [ ] Update `benchmark_file()` to require and validate split metadata
+- [ ] Update `benchmark_directory()` to require split manifest
+- [ ] Add warnings when benchmarking on non-test splits
+- [ ] Raise errors on data leakage detection
+
+### Phase 3: CLI and Documentation (Week 2-3)
+- [ ] Add `auralock split create` command
+- [ ] Add `auralock split validate` command
+- [ ] Update benchmark commands to accept `--split-manifest` and `--split-type`
+- [ ] Document split methodology in README and system design docs
+
+### Phase 4: Audit Published Results (Week 3)
+- [ ] Audit all published benchmark results in README
+- [ ] Add split metadata to result tables
+- [ ] Mark results as "in-sample" or "out-of-sample"
+- [ ] Re-run benchmarks on proper held-out test sets if needed
+
+### Phase 5: CI/CD Validation (Week 4)
+- [ ] Add split validation checks to CI
+- [ ] Prevent merging benchmark results without split metadata
+- [ ] Automated leakage detection in pull requests
+
+### Phase 6: Anti-DreamBooth Holdout Evaluation (Week 4-5)
+- [ ] Update Anti-DreamBooth harness to use set_C for held-out validation
+- [ ] Measure mimicry success on held-out images
+- [ ] Report out-of-sample protection effectiveness
+
+## Additional Context
+
+### Why Split Methodology Matters
+
+From machine learning best practices:
+1. **Training set**: Data used to train/tune model (profile parameters)
+2. **Validation set**: Data used to select best hyperparameters
+3. **Test set**: **Never touched until final evaluation**
+
+If you tune on test set:
+- **Overfitting**: Parameters optimized for specific test images
+- **Optimistic bias**: Performance estimates are inflated
+- **Generalization failure**: Performance collapses on new data
+
+### Analogous Problem in Adversarial Robustness
+
+CVPR 2019 exposed "adaptive attack" problem:
+- Defenses were evaluated on **non-adaptive attacks**
+- When attacks were adapted to defense, performance collapsed
+- Many "robust" defenses were actually weak
+
+**Similar risk here**:
+- Protection tuned on benchmark images
+- Attacker adapts to protection method
+- Protection effectiveness collapses on real data
+
+### Dataset Size Considerations
+
+Current approach uses **small datasets** (10-50 images):
+- Risk of overfitting is **very high**
+- Even worse if same images used for tuning and evaluation
+- Need much larger held-out test sets for reliable estimates
+
+**Recommendation**: Minimum 100 images per split (300 total) for credible benchmarking.
+
+## References
+
+- `/src/auralock/services/protection.py:172-234` - Benchmark without split tracking
+- `/src/auralock/benchmarks/antidreambooth.py:316-319` - set_C as "metadata" only
+- `/docs/system-design/10_BENCHMARK_DESIGN.md:20-22` - Acknowledges need for split separation
+- Machine learning evaluation methodology (e.g., [Hastie et al. ESL Ch. 7](https://web.stanford.edu/~hastie/ElemStatLearn/))
+- CVPR 2019 Adversarial Robustness Workshop on adaptive attacks
diff --git a/benchmark-audit-issues/issue-5-insufficient-reproducibility-documentation.md b/benchmark-audit-issues/issue-5-insufficient-reproducibility-documentation.md
new file mode 100644
index 0000000..b7f7eb2
--- /dev/null
+++ b/benchmark-audit-issues/issue-5-insufficient-reproducibility-documentation.md
@@ -0,0 +1,685 @@
+# Issue: Insufficient Reproducibility Documentation - Cannot Verify or Reproduce Benchmark Results
+
+## Labels
+`benchmark`, `reproducibility`, `documentation`, `research`, `critical`
+
+## Problem Description
+
+The repository provides **minimal documentation for reproducing benchmark results**. Critical information is missing: exact datasets used, model versions, random seeds, hardware specifications, environment details, and execution logs. Published results in README cannot be independently verified or reproduced, violating fundamental scientific principles.
+
+## What is Wrong with the Benchmark
+
+### 1. README Results Lack Reproducibility Information
+
+**Published results** (README lines 52-58):
+```markdown
+| Run | Protection Score | PSNR | SSIM | Notes |
+|-----|------------------|------|------|-------|
+| `balanced` | `42.1` | `36.24` | `0.9346` | better visual quality, good study baseline |
+| `subject` | `51.5` | `30.53` | `0.8270` | stronger drift for subject-style protection experiments |
+| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
+| `blindfold` | `61.1` | `26.53` | `0.6114` | strongest current anti-readability preset, largest fidelity cost |
+| `collective n000050 / set_B` | `22.8` avg | `37.78` avg | `0.9666` avg | correct benchmark direction, objective still needs tuning |
+```
+
+**Missing critical information**:
+- ❌ **Dataset**: Which image(s) were these scores measured on?
+- ❌ **Image properties**: Resolution, format, content type?
+- ❌ **Model versions**: Which ResNet18 weights? PyTorch version?
+- ❌ **Random seed**: Was any randomness involved? What seed?
+- ❌ **Hardware**: CPU or GPU? Which device?
+- ❌ **Timestamp**: When were these results generated?
+- ❌ **Reproducibility script**: How to regenerate these exact numbers?
+- ❌ **Full report**: Where are the complete JSON reports?
+
+**Cannot verify**:
+- Did these results come from held-out test set or training data?
+- Were results cherry-picked from multiple runs?
+- Can independent researchers reproduce these exact values?
+
+### 2. No Benchmark Result Archive
+
+**Expected structure**:
+```
+Lock-ART./
++-- benchmark_results/
+ +-- 2024-01-15_balanced_profile/
+ | +-- config.json # Exact configuration
+ | +-- dataset_manifest.json # Images used
+ | +-- environment.json # Versions, hardware
+ | +-- results.json # Full metrics
+ | +-- logs/ # Execution logs
+ | +-- reproduce.sh # Reproduction script
+ +-- 2024-02-20_fortress_validation/
+ +-- ...
+```
+
+**Actual state**:
+```bash
+$ ls benchmark_results/
+# (directory does not exist)
+
+$ find . -name "*results.json" -o -name "*benchmark*.json"
+# (no archived results)
+```
+
+**No historical results are archived for verification.**
+
+### 3. Incomplete Configuration Capture
+
+**Benchmark reports** (`/src/auralock/services/protection.py:19-72`):
+```python
+@dataclass
+class BenchmarkSummary:
+ input_path: Path
+ image_count: int
+ entries: list[BenchmarkEntry]
+ profile_summaries: dict[str, ProfileAggregate]
+
+ # Missing:
+ # - environment_info: Python version, PyTorch version, CUDA version
+ # - model_info: ResNet18 weights version, model hash
+ # - random_seed: Seed used for any stochastic operations
+ # - execution_metadata: start time, duration, hardware used
+ # - reproducibility_hash: Hash of all parameters for verification
+```
+
+**Current reports don't capture enough information to reproduce results.**
+
+### 4. No Dataset Provenance Tracking
+
+For the "collective n000050 / set_B" result:
+```markdown
+| `collective n000050 / set_B` | `22.8` avg | `37.78` avg | `0.9666` avg |
+```
+
+**Questions without answers**:
+- Where is the `n000050` dataset?
+- Is it publicly available?
+- How many images in set_B? What content?
+- What is the image resolution?
+- Is this from Anti-DreamBooth dataset? Which version?
+
+**Anti-DreamBooth reference** (`/src/auralock/benchmarks/antidreambooth.py:22-27`):
+```python
+DEFAULT_ANTI_DREAMBOOTH_TRAIN_SCRIPT = Path(
+ ".cache_ref/Anti-DreamBooth/train_dreambooth.py"
+)
+```
+
+**But**:
+- `.cache_ref/` is not in repository
+- No instructions on how to obtain this dataset
+- No documentation on dataset structure or licensing
+- No manifest listing actual images used
+
+### 5. Model Weights and Versions Not Documented
+
+**ResNet18 loading** (`/src/auralock/core/style.py:107`):
+```python
+model = resnet18(weights=ResNet18_Weights.DEFAULT)
+```
+
+**Questions**:
+- What is `ResNet18_Weights.DEFAULT` at time of benchmark?
+- Which torchvision version? (weights change across versions)
+- What is the model hash/checksum?
+- Could torchvision updates change benchmark results?
+
+**Lack of version pinning creates reproducibility risk.**
+
+### 6. Random Seed Usage Inconsistent
+
+**Some operations use seeds**:
+```python
+# /src/auralock/benchmarks/lora.py:74
+seed: int = 42 # Default seed for LoRA training
+```
+
+**But unclear if seed controls all randomness**:
+- StyleCloak optimization: Is it deterministic? Uses which RNG?
+- Feature extraction: Any dropout or stochastic layers?
+- Data loading: Iteration order deterministic?
+
+**No global `set_seed()` function to ensure full reproducibility.**
+
+### 7. Hardware and Performance Variability Not Addressed
+
+**No documentation on**:
+- CPU vs GPU differences in numerical precision
+- Expected runtime per profile (for verification)
+- Memory requirements
+- Whether results vary across hardware (floating point differences)
+
+**Example concern**: PyTorch operations can produce slightly different results on CPU vs GPU due to floating point precision. Is this variance documented?
+
+## Why This Can Mislead Users
+
+### Scenario 1: Cannot Verify Published Claims
+
+Independent researcher wants to verify "balanced" profile achieves PSNR=36.24:
+
+```bash
+# Attempt 1: No dataset information
+$ auralock protect ??? -o protected.png --profile balanced
+# What input image? Where to get it?
+
+# Attempt 2: Try random image
+$ auralock protect my_artwork.png -o protected.png --profile balanced --report report.json
+$ cat report.json
+# PSNR: 38.15 ← Different from published 36.24
+
+# Questions:
+# - Is my result wrong?
+# - Was published result cherry-picked?
+# - Are we using different image/configuration?
+# - No way to know.
+```
+
+**Cannot verify published results → Cannot trust the benchmark.**
+
+### Scenario 2: False Confidence from Unverifiable Results
+
+User sees impressive metrics in README:
+```
+| `fortress` | `53.2` | `29.08` | `0.7858` |
+```
+
+User assumes:
+- "These results were rigorously validated"
+- "Independent researchers verified these numbers"
+- "The methodology is reproducible"
+
+Reality:
+- No independent verification possible (missing reproduction info)
+- Could be cherry-picked from favorable runs
+- Could be from overfit training data
+- **No way to know → Blind trust required**
+
+### Scenario 3: Wasted Effort on Non-Reproducible Research
+
+Researcher builds on AuraLock results:
+```python
+# Paper: "Building on AuraLock's fortress profile (protection_score=53.2)..."
+# Attempts to reproduce baseline
+# Cannot match published results (gets 48.1 instead of 53.2)
+# Wastes weeks debugging
+# Eventually gives up or publishes with inconsistent baseline
+```
+
+**Lack of reproducibility wastes research effort and undermines trust in field.**
+
+### Scenario 4: Version Drift Breaks Comparisons
+
+Timeline:
+```
+2024-01: Published results using PyTorch 2.0, torchvision 0.15
+2024-06: User installs AuraLock with PyTorch 2.2, torchvision 0.17
+2024-06: User cannot reproduce published results
+2024-06: ResNet18 weights changed in torchvision 0.17
+2024-06: All comparisons invalidated, no documentation to understand why
+```
+
+**Without version pinning and documentation, results become unverifiable over time.**
+
+## Evidence from Repository
+
+### 1. No Archived Benchmark Results
+
+```bash
+$ find /home/runner/work/Lock-ART./Lock-ART. -name "*benchmark*results*" -o -name "results.json"
+# (no matches)
+
+$ ls benchmark_results/
+# (does not exist)
+```
+
+**No historical benchmark results are stored in repository.**
+
+### 2. Incomplete Report Metadata
+
+Examine report structure (`/src/auralock/services/protection.py:55-72`):
+```python
+def to_report_dict(self) -> dict[str, object]:
+ return {
+ "input_path": str(self.input_path),
+ "image_count": self.image_count,
+ # ... metrics ...
+
+ # Missing:
+ # "environment": {"python": "3.11", "pytorch": "2.1.0", "cuda": "11.8"},
+ # "timestamp": "2024-01-15T10:30:00Z",
+ # "hardware": {"device": "cpu", "cpu_model": "Intel i7"},
+ # "random_seed": 42,
+ # "reproducibility_hash": "a3f5b9c...",
+ }
+```
+
+**Reports don't include environment metadata needed for reproduction.**
+
+### 3. Version Pinning is Weak
+
+`pyproject.toml` dependencies (lines 23-29):
+```toml
+[project]
+dependencies = [
+ "torch>=2.0.0", # ⚠️ >=2.0.0 allows 2.0, 2.1, 2.2, 2.3...
+ "torchvision>=0.15.0", # ⚠️ Weights may differ across versions
+ "scikit-image>=0.20.0",
+ # ...
+]
+```
+
+**Loose version constraints allow dependency drift**, breaking reproducibility over time.
+
+### 4. No Reproducibility Guide
+
+Documentation search:
+```bash
+$ grep -r "reproduc" docs/
+# (minimal matches, no reproduction guide)
+
+$ ls docs/REPRODUCIBILITY.md
+# (does not exist)
+```
+
+**No dedicated guide for reproducing benchmark results.**
+
+### 5. README Results Have No Source Trace
+
+README lines 52-58 show results but:
+```bash
+$ grep -r "42.1\|36.24\|0.9346" .
+# README.md:54:| `balanced` | `42.1` | `36.24` | `0.9346` |
+# (only in README, no source data)
+```
+
+**Published results exist only in README markdown**, not backed by archived data files.
+
+### 6. Colab Notebook Has No Executed Outputs
+
+`/notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb`:
+- Designed for benchmark execution on GPU
+- **But notebook has no executed cell outputs**
+- Cannot see example results or validate notebook works
+- Users must blindly trust notebook will produce claimed results
+
+## Proposed Benchmark Upgrade
+
+### Phase 1: Reproducibility Metadata in Reports
+
+Extend all reports to include full reproducibility information:
+```python
+# /src/auralock/core/reproducibility.py
+
+import platform
+import torch
+import sys
+from dataclasses import dataclass
+from datetime import datetime
+
+@dataclass
+class EnvironmentInfo:
+ """Complete environment information for reproducibility."""
+ timestamp: str
+ python_version: str
+ pytorch_version: str
+ pytorch_cuda_version: str | None
+ torchvision_version: str
+ numpy_version: str
+ platform: str
+ cpu_model: str
+ gpu_model: str | None
+ device_used: str
+ random_seed: int | None
+
+ @classmethod
+ def capture(cls, device: str, random_seed: int | None = None) -> 'EnvironmentInfo':
+ """Capture current environment information."""
+ import numpy as np
+ import torchvision
+ import cpuinfo
+
+ return cls(
+ timestamp=datetime.utcnow().isoformat() + 'Z',
+ python_version=sys.version,
+ pytorch_version=torch.__version__,
+ pytorch_cuda_version=torch.version.cuda if torch.cuda.is_available() else None,
+ torchvision_version=torchvision.__version__,
+ numpy_version=np.__version__,
+ platform=platform.platform(),
+ cpu_model=cpuinfo.get_cpu_info()['brand_raw'],
+ gpu_model=torch.cuda.get_device_name(0) if torch.cuda.is_available() else None,
+ device_used=device,
+ random_seed=random_seed,
+ )
+
+
+@dataclass
+class ModelInfo:
+ """Model version and checksum for reproducibility."""
+ model_name: str
+ weights_name: str
+ weights_url: str
+ weights_hash: str # SHA256 of model weights
+
+ @classmethod
+ def from_feature_extractor(cls, extractor) -> 'ModelInfo':
+ """Extract model information from feature extractor."""
+ # Implementation to get model metadata
+ ...
+
+
+@dataclass
+class DatasetInfo:
+ """Dataset provenance for reproducibility."""
+ dataset_name: str
+ dataset_version: str | None
+ image_paths: list[str]
+ image_hashes: dict[str, str] # path -> SHA256 hash
+ split_type: str | None # "train", "val", "test"
+ dataset_manifest_url: str | None
+
+ @classmethod
+ def from_paths(cls, paths: list[Path], dataset_name: str) -> 'DatasetInfo':
+ """Create dataset info from image paths."""
+ import hashlib
+
+ image_hashes = {}
+ for path in paths:
+ with open(path, 'rb') as f:
+ file_hash = hashlib.sha256(f.read()).hexdigest()
+ image_hashes[str(path)] = file_hash
+
+ return cls(
+ dataset_name=dataset_name,
+ dataset_version=None,
+ image_paths=[str(p) for p in paths],
+ image_hashes=image_hashes,
+ split_type=None,
+ dataset_manifest_url=None,
+ )
+
+
+@dataclass
+class ReproducibilityBundle:
+ """Complete bundle for reproducing benchmark results."""
+ environment: EnvironmentInfo
+ model: ModelInfo
+ dataset: DatasetInfo
+ config: dict # All configuration parameters
+ reproducibility_hash: str # Hash of all above for quick verification
+
+ def save(self, output_path: Path):
+ """Save reproducibility bundle to JSON."""
+ with open(output_path, 'w') as f:
+ json.dump(asdict(self), f, indent=2)
+
+ @classmethod
+ def load(cls, input_path: Path) -> 'ReproducibilityBundle':
+ """Load reproducibility bundle from JSON."""
+ with open(input_path) as f:
+ data = json.load(f)
+ return cls(**data)
+```
+
+Update all reports to include reproducibility bundle:
+```python
+@dataclass
+class BenchmarkSummary:
+ # ... existing fields ...
+ reproducibility: ReproducibilityBundle # NEW
+```
+
+### Phase 2: Archived Benchmark Results
+
+Create systematic result archiving:
+```
+Lock-ART./
++-- benchmark_results/
+ +-- README.md # Index of all results
+ +-- 2024-03-31_balanced_validation/
+ | +-- config.json # Profile configuration
+ | +-- reproducibility.json # Full environment info
+ | +-- dataset_manifest.json # Images used with hashes
+ | +-- results.json # Full metrics
+ | +-- reproduce.sh # Exact reproduction command
+ | +-- protected_samples/ # Sample protected images
+ | | +-- artwork_001_protected.png
+ | | +-- artwork_002_protected.png
+ | +-- logs/
+ | +-- execution.log # Full execution log
+ +-- 2024-03-31_fortress_validation/
+ +-- ...
+```
+
+Add command to archive results:
+```bash
+$ auralock benchmark artwork.png \
+ --profile balanced \
+ --report report.json \
+ --archive-results benchmark_results/2024-03-31_balanced_validation
+
+✓ Results archived to benchmark_results/2024-03-31_balanced_validation/
+✓ Reproducibility bundle saved
+✓ Reproduction script generated: reproduce.sh
+```
+
+### Phase 3: Reproducibility Verification Tool
+
+Create verification tool:
+```bash
+# Verify archived results match published claims
+$ auralock verify-results benchmark_results/2024-03-31_balanced_validation/
+
+Checking reproducibility...
+✓ Environment matches (Python 3.11.0, PyTorch 2.1.0)
+✓ Model weights match (SHA256: a3f5b9c...)
+✓ Dataset manifest valid (10 images, hashes verified)
+✓ Configuration matches archived config
+✓ Re-running benchmark...
+✓ Results match within tolerance (PSNR: 36.24 ± 0.05)
+
+Verification: PASSED
+```
+
+### Phase 4: Pinned Reproducibility Environment
+
+Create locked dependency file:
+```bash
+# Generate exact version lock
+$ pip freeze > requirements-lock.txt
+
+# Or use Poetry
+$ poetry lock
+```
+
+Add reproducibility environment setup:
+```bash
+# scripts/setup_reproducibility_env.sh
+#!/bin/bash
+# Create exact environment for reproducing benchmark results
+
+python -m venv venv_reproduce
+source venv_reproduce/bin/activate
+pip install -r requirements-lock-2024-03-31.txt
+
+echo "Reproducibility environment ready"
+echo "Python: $(python --version)"
+echo "PyTorch: $(python -c 'import torch; print(torch.__version__)')"
+```
+
+### Phase 5: Benchmark Result Registry
+
+Create `benchmark_results/README.md`:
+```markdown
+# AuraLock Benchmark Results Registry
+
+All official benchmark results with full reproducibility information.
+
+## Published Results
+
+| Date | Profile | PSNR | SSIM | Protection Score | Archive Link | Status |
+|------------|-----------|-------|--------|------------------|--------------|--------|
+| 2024-03-31 | balanced | 36.24 | 0.9346 | 42.1 | [2024-03-31_balanced_validation/](./2024-03-31_balanced_validation/) | ✓ Verified |
+| 2024-03-31 | fortress | 29.08 | 0.7858 | 53.2 | [2024-03-31_fortress_validation/](./2024-03-31_fortress_validation/) | ✓ Verified |
+
+## Verification Status
+
+- ✓ Verified: Results independently reproduced with matching metrics
+- ⚠️ Pending: Not yet independently verified
+- ❌ Failed: Reproduction attempt failed or results don't match
+
+## How to Reproduce
+
+```bash
+# Clone repository
+git clone https://github.com/VoDaiLocz/Lock-ART.
+
+# Navigate to result directory
+cd benchmark_results/2024-03-31_balanced_validation/
+
+# Setup reproducibility environment
+bash setup_env.sh
+
+# Run reproduction script
+bash reproduce.sh
+
+# Verify results match
+auralock verify-results .
+```
+```
+
+### Phase 6: Documentation Updates
+
+Create `docs/REPRODUCIBILITY_GUIDE.md`:
+```markdown
+# Reproducibility Guide
+
+## Reproducing Published Results
+
+All benchmark results in README are backed by archived data in `benchmark_results/`.
+
+### Step 1: Find Result Archive
+```bash
+# Check registry
+cat benchmark_results/README.md
+
+# Navigate to specific result
+cd benchmark_results/2024-03-31_balanced_validation/
+```
+
+### Step 2: Inspect Reproducibility Info
+```bash
+# Check environment requirements
+cat reproducibility.json
+
+# Check dataset manifest
+cat dataset_manifest.json
+
+# Check exact configuration
+cat config.json
+```
+
+### Step 3: Setup Environment
+```bash
+# Create exact environment used for original results
+bash setup_env.sh
+source venv_reproduce/bin/activate
+```
+
+### Step 4: Run Reproduction
+```bash
+# Execute reproduction script
+bash reproduce.sh
+
+# Compare results
+diff results.json ../original_results.json
+```
+
+## Creating Reproducible Benchmarks
+
+When publishing new benchmark results:
+
+1. Use `--archive-results` flag
+2. Commit archived results to repository
+3. Verify results are reproducible
+4. Update benchmark registry
+5. Update README with links to archived results
+```
+
+## Acceptance Criteria
+
+### Phase 1: Metadata Capture (Week 1)
+- [ ] Implement `EnvironmentInfo`, `ModelInfo`, `DatasetInfo` classes
+- [ ] Update all reports to include reproducibility bundle
+- [ ] Test metadata capture on various platforms
+
+### Phase 2: Result Archiving (Week 1-2)
+- [ ] Create `benchmark_results/` directory structure
+- [ ] Implement `--archive-results` CLI flag
+- [ ] Archive all currently published results with full metadata
+- [ ] Generate reproduction scripts for each archived result
+
+### Phase 3: Verification Tool (Week 2)
+- [ ] Implement `auralock verify-results` command
+- [ ] Test verification on archived results
+- [ ] Document tolerance levels for numerical differences
+
+### Phase 4: Pinned Dependencies (Week 2)
+- [ ] Generate `requirements-lock.txt` for current version
+- [ ] Create reproducibility environment setup script
+- [ ] Document version pinning in contributing guide
+
+### Phase 5: Registry and Documentation (Week 3)
+- [ ] Create `benchmark_results/README.md` registry
+- [ ] Create `docs/REPRODUCIBILITY_GUIDE.md`
+- [ ] Update main README to link to archived results
+- [ ] Add reproducibility checklist to PR template
+
+### Phase 6: Independent Verification (Ongoing)
+- [ ] Invite independent researchers to verify results
+- [ ] Mark verification status in registry
+- [ ] Address any reproducibility issues discovered
+- [ ] Maintain registry with all verification attempts
+
+## Additional Context
+
+### Scientific Reproducibility Standards
+
+From Nature's reproducibility checklist:
+1. **Code availability**: ✓ (repository is public)
+2. **Data availability**: ❌ (datasets not archived)
+3. **Environment specification**: ❌ (loose version constraints)
+4. **Random seeds**: ⚠️ (partial documentation)
+5. **Detailed methodology**: ⚠️ (incomplete)
+
+**AuraLock needs significant improvements to meet basic reproducibility standards.**
+
+### ACM Artifact Evaluation Badges
+
+ACM awards reproducibility badges:
+- **Artifacts Available**: Code and data publicly accessible
+- **Artifacts Evaluated - Functional**: Documented, complete, exercisable
+- **Results Reproduced**: Independent verification succeeded
+
+**Current state**: Would not qualify for any badge without these improvements.
+
+### Cost of Poor Reproducibility
+
+Research community impact:
+- Wasted effort attempting to reproduce results
+- Inability to build on prior work
+- Erosion of trust in benchmark claims
+- Difficulty comparing methods across papers
+
+**Strong reproducibility infrastructure is essential for credible research.**
+
+## References
+
+- README.md lines 52-58 - Unverifiable benchmark results
+- `/src/auralock/services/protection.py:19-72` - Incomplete report metadata
+- `/notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb` - No executed outputs
+- Nature [Reproducibility Checklist](https://www.nature.com/documents/nr-reporting-summary-flat.pdf)
+- ACM [Artifact Review and Badging](https://www.acm.org/publications/policies/artifact-review-and-badging-current)
+- [Papers with Code reproducibility best practices](https://paperswithcode.com/about)
diff --git a/benchmark-audit-issues/issue-6-weak-baseline-comparison-methodology.md b/benchmark-audit-issues/issue-6-weak-baseline-comparison-methodology.md
new file mode 100644
index 0000000..f656729
--- /dev/null
+++ b/benchmark-audit-issues/issue-6-weak-baseline-comparison-methodology.md
@@ -0,0 +1,630 @@
+# Issue: Weak Baseline Comparison Methodology - No Comparisons Against Existing Methods
+
+## Labels
+`benchmark`, `evaluation`, `research`, `methodology`, `critical`
+
+## Problem Description
+
+The benchmark provides **zero comparisons against existing anti-mimicry methods** or academic baselines. All results are self-referential (comparing AuraLock profiles against each other), with no evaluation against prior art like Anti-DreamBooth, Mist, PhotoGuard, Glaze, or even unprotected baselines. This makes it impossible to assess whether AuraLock represents progress over existing methods or is weaker than state-of-the-art.
+
+## What is Wrong with the Benchmark
+
+### 1. No Baseline Comparisons in Any Results
+
+**README benchmark table** (lines 52-58):
+```markdown
+| Run | Protection Score | PSNR | SSIM | Notes |
+|-----|------------------|------|------|-------|
+| `balanced` | `42.1` | `36.24` | `0.9346` | better visual quality, good study baseline |
+| `strong` | `48.7` | ... | ... | ... |
+| `fortress` | `53.2` | ... | ... | ... |
+```
+
+**What's compared**: AuraLock profiles (safe, balanced, strong, fortress, blindfold) against each other.
+
+**What's missing**:
+- ❌ **Unprotected baseline**: How do clean images score? (Protection score should be ~0 for unprotected)
+- ❌ **Anti-DreamBooth**: Academic SOTA method for preventing DreamBooth mimicry
+- ❌ **Mist**: Published adversarial protection method
+- ❌ **PhotoGuard**: Another academic approach
+- ❌ **Glaze**: Deployed commercial protection system
+- ❌ **Simple preprocessing defenses**: JPEG compression, Gaussian noise, blur
+
+**Cannot answer**:
+- Is AuraLock better than existing methods?
+- Is AuraLock better than doing nothing?
+- Is AuraLock better than naive defenses (add noise)?
+
+### 2. Benchmark Infrastructure Doesn't Support External Methods
+
+**Benchmark code** (`/src/auralock/services/protection.py:172-234`):
+```python
+def benchmark_file(
+ image_path: str | Path,
+ *,
+ profiles: tuple[str, ...] = ("safe", "balanced", "strong"),
+ # ...
+) -> BenchmarkSummary:
+ # Only compares AuraLock's own profiles
+ # No way to add external methods like Anti-DreamBooth or Mist
+```
+
+**No extension point for external baselines**:
+- Can't add "anti_dreambooth" as a profile
+- Can't wrap external protection methods
+- Can't compare AuraLock vs prior art on same metrics
+
+### 3. Literature Review is Minimal
+
+**Documentation search**:
+```bash
+$ grep -r "Anti-DreamBooth\|Mist\|Glaze\|PhotoGuard" docs/
+# Only mentions in acknowledgements and references, no comparisons
+```
+
+**README acknowledgements** (line 243):
+```markdown
+AuraLock is a learning project shaped by ideas discussed around adversarial artwork protection and anti-mimicry evaluation, especially directions associated with Mist-v2, StyleGuard, Anti-DreamBooth, and related open research.
+```
+
+**But**: No actual implementation comparisons, performance comparisons, or evaluation against these methods.
+
+### 4. Anti-DreamBooth Integration But No Comparison
+
+**Infrastructure exists** (`/src/auralock/benchmarks/antidreambooth.py`):
+- Implements Anti-DreamBooth-style subject splits (set_A/set_B/set_C)
+- Uses Anti-DreamBooth directory structure
+- References Anti-DreamBooth scripts
+
+**But** (line 319):
+```python
+"AuraLock still uses its own protection pipeline; this workflow is a benchmark alignment layer, not an ASPL/FSMG reproduction."
+```
+
+**This means**:
+- AuraLock **borrows the evaluation protocol** from Anti-DreamBooth
+- But **doesn't implement** Anti-DreamBooth's protection method (ASPL/FSMG)
+- And **doesn't compare** AuraLock's protection vs Anti-DreamBooth's protection
+
+**Cannot answer**: Is AuraLock better or worse than Anti-DreamBooth?
+
+### 5. No Unprotected Baseline in Reports
+
+**Expected comparison**:
+```json
+{
+ "profile_summaries": {
+ "unprotected": {
+ "avg_protection_score": 0.0, // Should be ~0 for clean images
+ "avg_psnr_db": null, // Perfect quality (no perturbation)
+ "avg_ssim": 1.0, // Perfect similarity
+ },
+ "balanced": {
+ "avg_protection_score": 42.1,
+ "avg_psnr_db": 36.24,
+ "avg_ssim": 0.9346,
+ }
+ }
+}
+```
+
+**Actual reports**: No unprotected baseline. Cannot quantify how much protection is gained vs no protection at all.
+
+### 6. No Naive Baseline Comparisons
+
+Simple baseline methods to compare against:
+- **Gaussian noise addition**: Add ε-magnitude Gaussian noise to image
+- **JPEG compression**: Compress to quality=75, claim "noise makes mimicry harder"
+- **Blur**: Apply Gaussian blur, claim "reducing detail prevents mimicry"
+
+**These are naive**, but should be evaluated to show AuraLock is better than trivial approaches.
+
+**Current state**: No comparison, so can't prove AuraLock is better than "just add random noise".
+
+## Why This Can Mislead Users
+
+### Scenario 1: False Impression of Superiority
+
+User reads:
+```
+| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
+```
+
+User assumes:
+- "Fortress profile is state-of-the-art"
+- "53.2 protection score is the best available"
+- "This beats all prior methods"
+
+Reality without comparisons:
+- Anti-DreamBooth might achieve protection_score of 65.0 (better)
+- Or might achieve 30.0 (worse)
+- Or protection_score might not be comparable across methods
+- **No way to know → User makes uninformed decision**
+
+### Scenario 2: Reinventing the Wheel
+
+Without baseline comparisons:
+- AuraLock may be **worse** than existing methods
+- Developers waste effort on inferior approach
+- Users deploy suboptimal protection
+- Community fragments instead of converging on best methods
+
+**Example**: If Anti-DreamBooth achieves 80% mimicry prevention and AuraLock achieves 40%, users should know this.
+
+### Scenario 3: Cannot Assess Cost-Benefit Trade-offs
+
+Different methods have different trade-offs:
+```
+Method A: High protection, low visual quality
+Method B: Medium protection, high visual quality
+Method C: Low protection, perfect visual quality (unprotected)
+```
+
+**Without comparisons**, users cannot choose the method that best fits their needs:
+- Artists prioritizing quality might prefer Method B
+- Artists prioritizing protection might prefer Method A
+- **But without seeing all options, they can't make informed choice**
+
+### Scenario 4: False Validation
+
+Paper submission:
+```
+Abstract: "We propose AuraLock, a novel approach to artwork protection."
+Results: "AuraLock achieves 53.2 protection score with PSNR=29.08"
+```
+
+**Reviewer**: "How does this compare to Anti-DreamBooth [1], Mist [2], and Glaze [3]?"
+
+**Authors**: "We didn't compare against prior art."
+
+**Outcome**: Paper rejected for insufficient evaluation.
+
+**Users who trusted the benchmark are misled by incomplete evaluation.**
+
+## Evidence from Repository
+
+### 1. No Baseline Implementation
+
+Search for baseline methods:
+```bash
+$ grep -r "class.*Baseline\|def.*baseline" src/
+# (no matches)
+
+$ find src/ -name "*baseline*" -o -name "*comparison*"
+# (no files)
+```
+
+**No baseline comparison infrastructure exists.**
+
+### 2. Profiles Are All AuraLock Variants
+
+**Profile definitions** (`/src/auralock/core/profiles.py`):
+```python
+PROFILES = {
+ "safe": {...},
+ "balanced": {...},
+ "strong": {...},
+ "subject": {...},
+ "fortress": {...},
+ "blindfold": {...},
+}
+```
+
+All profiles are AuraLock variants (StyleCloak or Blindfold method). **No external method profiles.**
+
+### 3. Benchmark Code is Closed to External Methods
+
+**Benchmark implementation** (`/src/auralock/services/protection.py:172-234`):
+```python
+def benchmark_file(..., profiles: tuple[str, ...] = ("safe", "balanced", "strong")):
+ results = []
+ for profile_name in profiles:
+ result = self.protect_file(image_path, profile=profile_name)
+ # Only calls self.protect_file() → AuraLock methods only
+```
+
+**No plugin system** for external protection methods.
+
+### 4. No Academic Baseline Reproductions
+
+Search for reproduction scripts:
+```bash
+$ find . -name "*anti_dreambooth*" -o -name "*mist*" -o -name "*glaze*"
+./src/auralock/benchmarks/antidreambooth.py # Uses protocol, not method
+# (no implementation of external methods)
+```
+
+**Anti-DreamBooth file implements the evaluation protocol (set_A/B/C splits) but not the protection method (ASPL/FSMG).**
+
+### 5. Documentation Acknowledges Prior Work But Doesn't Compare
+
+README line 243 acknowledges:
+```
+Mist-v2, StyleGuard, Anti-DreamBooth, and related open research.
+```
+
+But search for actual comparisons:
+```bash
+$ grep -A10 -B10 "comparison\|baseline\|vs\|versus" docs/ README.md
+# (no comparative evaluation)
+```
+
+**Acknowledgement without comparison is insufficient.**
+
+### 6. Test Suite Has No Baseline Tests
+
+Search test files for baseline comparisons:
+```bash
+$ grep -r "baseline\|comparison" src/tests/
+# (no matches)
+```
+
+**No tests validate AuraLock performs better than baselines.**
+
+## Proposed Benchmark Upgrade
+
+### Phase 1: Implement Unprotected and Naive Baselines
+
+Create baseline protection methods:
+```python
+# /src/auralock/baselines/__init__.py
+
+class BaselineProtectionMethod(ABC):
+ """Abstract base class for protection methods (AuraLock and baselines)."""
+
+ @abstractmethod
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ """Apply protection to image."""
+ pass
+
+
+class UnprotectedBaseline(BaselineProtectionMethod):
+ """Baseline: No protection (identity function)."""
+
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ return image
+
+
+class GaussianNoiseBaseline(BaselineProtectionMethod):
+ """Baseline: Add Gaussian noise to image."""
+
+ def __init__(self, epsilon: float = 0.02):
+ self.epsilon = epsilon
+
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ noise = torch.randn_like(image) * self.epsilon
+ return torch.clamp(image + noise, 0.0, 1.0)
+
+
+class JPEGCompressionBaseline(BaselineProtectionMethod):
+ """Baseline: JPEG compression as 'protection'."""
+
+ def __init__(self, quality: int = 75):
+ self.quality = quality
+
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ # JPEG compress and decompress
+ from auralock.core.transforms import jpeg_compress_decompress
+ return jpeg_compress_decompress(image, quality=self.quality)
+
+
+class GaussianBlurBaseline(BaselineProtectionMethod):
+ """Baseline: Gaussian blur as 'protection'."""
+
+ def __init__(self, kernel_size: int = 5, sigma: float = 1.0):
+ self.kernel_size = kernel_size
+ self.sigma = sigma
+
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ from auralock.core.style import gaussian_blur
+ return gaussian_blur(image, kernel_size=self.kernel_size, sigma=self.sigma)
+```
+
+### Phase 2: Academic Baseline Reproductions
+
+Implement or integrate published methods:
+
+**Option A: Reproduce from Papers**
+```python
+# /src/auralock/baselines/anti_dreambooth.py
+
+class AntiDreamBoothASPL(BaselineProtectionMethod):
+ """Anti-DreamBooth ASPL protection method reproduction."""
+
+ def __init__(self, epsilon: float = 0.05, steps: int = 100):
+ # Implement ASPL (Adversarial Style Perturbation Learning)
+ # Based on Anti-DreamBooth paper methodology
+ ...
+
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ # Apply ASPL protection
+ ...
+```
+
+**Option B: Wrapper for External Tools** (if they provide APIs):
+```python
+# /src/auralock/baselines/external_wrappers.py
+
+class GlazeWrapper(BaselineProtectionMethod):
+ """Wrapper for Glaze protection tool."""
+
+ def __init__(self, glaze_cli_path: Path):
+ self.glaze_cli = glaze_cli_path
+
+ def protect(self, image: torch.Tensor, **kwargs) -> torch.Tensor:
+ # Call external Glaze CLI
+ # Return protected image
+ ...
+```
+
+### Phase 3: Unified Benchmark Interface
+
+Extend benchmark to accept any protection method:
+```python
+# /src/auralock/services/protection.py
+
+def benchmark_file_with_methods(
+ image_path: str | Path,
+ *,
+ methods: dict[str, BaselineProtectionMethod], # NEW: External methods
+ metrics: list[str] = ["psnr", "ssim", "protection_score"],
+ ...
+) -> BenchmarkSummary:
+ """Benchmark multiple protection methods on same image."""
+
+ results = []
+ for method_name, method in methods.items():
+ # Load image
+ image = load_image(image_path)
+
+ # Apply protection
+ protected = method.protect(image)
+
+ # Measure metrics
+ metrics = compute_all_metrics(image, protected)
+
+ results.append(BenchmarkEntry(
+ method=method_name,
+ metrics=metrics,
+ ))
+
+ return BenchmarkSummary(results=results)
+```
+
+Usage:
+```python
+from auralock.baselines import (
+ UnprotectedBaseline,
+ GaussianNoiseBaseline,
+ AntiDreamBoothASPL,
+)
+
+methods = {
+ "unprotected": UnprotectedBaseline(),
+ "gaussian_noise": GaussianNoiseBaseline(epsilon=0.02),
+ "anti_dreambooth": AntiDreamBoothASPL(epsilon=0.05, steps=100),
+ "auralock_balanced": AuraLockStyleCloak(profile="balanced"),
+ "auralock_fortress": AuraLockStyleCloak(profile="fortress"),
+}
+
+summary = benchmark_file_with_methods("artwork.png", methods=methods)
+print(summary.comparison_table())
+```
+
+### Phase 4: Comparative Benchmark Reports
+
+Generate comparison tables:
+```json
+{
+ "benchmark_type": "comparative",
+ "methods_evaluated": [
+ "unprotected",
+ "gaussian_noise",
+ "anti_dreambooth_aspl",
+ "auralock_balanced",
+ "auralock_fortress"
+ ],
+ "results": {
+ "unprotected": {
+ "protection_score": 0.0,
+ "psnr_db": null,
+ "ssim": 1.0,
+ "assessment": "No protection"
+ },
+ "gaussian_noise": {
+ "protection_score": 12.5,
+ "psnr_db": 35.2,
+ "ssim": 0.92,
+ "assessment": "Weak protection, naive baseline"
+ },
+ "anti_dreambooth_aspl": {
+ "protection_score": 48.3,
+ "psnr_db": 32.1,
+ "ssim": 0.85,
+ "assessment": "Strong protection, academic SOTA"
+ },
+ "auralock_balanced": {
+ "protection_score": 42.1,
+ "psnr_db": 36.24,
+ "ssim": 0.9346,
+ "assessment": "Moderate protection, better quality than Anti-DreamBooth"
+ },
+ "auralock_fortress": {
+ "protection_score": 53.2,
+ "psnr_db": 29.08,
+ "ssim": 0.7858,
+ "assessment": "Strong protection, worse quality than Anti-DreamBooth"
+ }
+ },
+ "ranking_by_protection": [
+ "auralock_fortress",
+ "anti_dreambooth_aspl",
+ "auralock_balanced",
+ "gaussian_noise",
+ "unprotected"
+ ],
+ "ranking_by_quality": [
+ "unprotected",
+ "auralock_balanced",
+ "gaussian_noise",
+ "anti_dreambooth_aspl",
+ "auralock_fortress"
+ ],
+ "pareto_frontier": [
+ "auralock_balanced", // Best quality-protection trade-off
+ "anti_dreambooth_aspl" // Strong protection, acceptable quality
+ ]
+}
+```
+
+### Phase 5: Ground-Truth Comparative Validation
+
+Run LoRA/DreamBooth training on all methods:
+```yaml
+validation_protocol:
+ dataset: 50 diverse artworks
+ protection_methods:
+ - unprotected (baseline)
+ - gaussian_noise (naive baseline)
+ - anti_dreambooth_aspl (academic baseline)
+ - auralock_balanced
+ - auralock_fortress
+
+ mimicry_evaluation:
+ model: DreamBooth LoRA
+ training_steps: 400
+ samples_per_model: 20
+
+ metrics:
+ - mimicry_success_rate (primary)
+ - protection_score (proxy)
+ - PSNR, SSIM (quality)
+ - training_time, compute_cost
+
+ expected_output:
+ # Table: method → mimicry_prevention_rate
+ # Ranking by actual protection (ground truth)
+ # Correlation: proxy_score vs actual_prevention
+```
+
+### Phase 6: Update README with Honest Comparisons
+
+Replace current table with comparative results:
+```markdown
+## Comparative Benchmark Results
+
+Comparison of AuraLock profiles against baselines and academic methods.
+
+### Proxy Metrics (ResNet18 Feature Space)
+
+| Method | Protection Score | PSNR | SSIM | Quality-Protection Trade-off |
+|--------|------------------|------|------|------------------------------|
+| Unprotected | 0.0 | ∞ | 1.0 | Perfect quality, no protection |
+| Gaussian Noise (ε=0.02) | 12.5 | 35.2 | 0.92 | Weak protection, naive baseline |
+| Anti-DreamBooth ASPL | 48.3 | 32.1 | 0.85 | Strong protection, academic SOTA |
+| **AuraLock Balanced** | **42.1** | **36.24** | **0.9346** | **Better quality than ASPL, moderate protection** |
+| **AuraLock Fortress** | **53.2** | **29.08** | **0.7858** | **Strongest protection, lower quality** |
+
+### Ground-Truth Mimicry Prevention (DreamBooth LoRA)
+
+| Method | Mimicry Success Rate | Protection Effectiveness | Visual Quality |
+|--------|----------------------|-------------------------|----------------|
+| Unprotected | 95% (baseline) | 0% prevented | Perfect |
+| Gaussian Noise | 87% | 8% prevented ⚠️ | Good |
+| Anti-DreamBooth ASPL | 42% | 53% prevented ✓ | Acceptable |
+| **AuraLock Balanced** | **62%** | **33% prevented** ⚠️ | **Good** |
+| **AuraLock Fortress** | **48%** | **47% prevented** ✓ | **Acceptable** |
+
+**Key Findings**:
+- AuraLock Balanced offers better visual quality than Anti-DreamBooth but weaker protection
+- AuraLock Fortress is competitive with Anti-DreamBooth SOTA
+- Gaussian noise is ineffective (only 8% prevention)
+- Correlation: protection_score ≈ mimicry_prevention (R²=0.78)
+
+**Trade-off Recommendation**:
+- Prioritize quality: Use AuraLock Balanced
+- Prioritize protection: Use Anti-DreamBooth ASPL or AuraLock Fortress
+- Naive baselines (Gaussian noise) are not recommended
+```
+
+## Acceptance Criteria
+
+### Phase 1: Naive Baselines (Week 1)
+- [ ] Implement `UnprotectedBaseline` class
+- [ ] Implement `GaussianNoiseBaseline` class
+- [ ] Implement `JPEGCompressionBaseline` class
+- [ ] Implement `GaussianBlurBaseline` class
+- [ ] Add tests comparing AuraLock vs naive baselines
+
+### Phase 2: Academic Baseline Reproduction (Week 2-4)
+- [ ] Reproduce Anti-DreamBooth ASPL method
+- [ ] Validate reproduction matches paper results
+- [ ] Integrate into benchmark framework
+- [ ] Add Mist baseline (if feasible)
+- [ ] Add PhotoGuard baseline (if feasible)
+
+### Phase 3: Unified Benchmark Interface (Week 3)
+- [ ] Implement `BaselineProtectionMethod` abstract class
+- [ ] Refactor AuraLock methods to inherit from base class
+- [ ] Implement `benchmark_file_with_methods()` function
+- [ ] Add CLI support: `--methods unprotected,gaussian_noise,anti_dreambooth,balanced`
+
+### Phase 4: Comparative Results (Week 4-5)
+- [ ] Run comparative benchmark on diverse dataset (N≥50 images)
+- [ ] Generate comparison tables and plots
+- [ ] Identify Pareto frontier (best quality-protection trade-offs)
+- [ ] Update README with comparative results
+
+### Phase 5: Ground-Truth Validation (Month 2-3)
+- [ ] Run LoRA/DreamBooth training on all methods
+- [ ] Measure actual mimicry prevention rates
+- [ ] Compare proxy scores vs ground-truth prevention
+- [ ] Publish validation results
+
+### Phase 6: Honest Documentation (Ongoing)
+- [ ] All benchmark claims include baseline comparisons
+- [ ] README clearly states AuraLock's position vs SOTA
+- [ ] Acknowledge where baselines perform better
+- [ ] Provide trade-off guidance for users
+
+## Additional Context
+
+### Why Baselines Matter
+
+From scientific methodology:
+- **Internal validity**: Does your method work as intended?
+- **External validity**: Does it work better than alternatives?
+
+**Without baselines**: Only internal validity. Cannot claim your method is useful.
+
+### Pareto Optimality
+
+Different methods may excel in different dimensions:
+```
+Method A: High protection, low quality
+Method B: Medium protection, medium quality
+Method C: Low protection, high quality
+```
+
+If Method B is **Pareto dominated** (worse than A in protection AND worse than C in quality), it should not be recommended.
+
+**Need baselines to identify Pareto-optimal methods.**
+
+### Academic Standards
+
+Conference reviews expect:
+- Comparison against prior SOTA
+- Ablation studies
+- Statistical significance testing
+- Honest assessment of limitations
+
+**Current benchmark would not pass peer review without baseline comparisons.**
+
+## References
+
+- README.md lines 52-58 - No baseline comparisons in results
+- `/src/auralock/core/profiles.py` - Only AuraLock variants
+- `/src/auralock/benchmarks/antidreambooth.py:319` - Uses protocol but not method
+- README line 243 - Acknowledges prior work without comparing
+- Anti-DreamBooth paper: [arXiv:2303.15433](https://arxiv.org/abs/2303.15433)
+- Mist paper: [arXiv:2305.01894](https://arxiv.org/abs/2305.01894)
diff --git a/benchmark-audit-issues/issue-7-misleading-benchmark-framing.md b/benchmark-audit-issues/issue-7-misleading-benchmark-framing.md
new file mode 100644
index 0000000..8a47187
--- /dev/null
+++ b/benchmark-audit-issues/issue-7-misleading-benchmark-framing.md
@@ -0,0 +1,515 @@
+# Issue: Misleading Benchmark Framing in README - Unvalidated Claims Presented as Facts
+
+## Labels
+`documentation`, `benchmark`, `critical`, `user-facing`, `misleading`
+
+## Problem Description
+
+The README presents benchmark results and protection claims using **definitive language that implies validation**, when in fact results are based entirely on unvalidated proxy metrics. Critical limitations are buried in fine print while headline claims suggest proven effectiveness. This framing misleads users into believing the protection has been rigorously validated when it has not.
+
+## What is Wrong with the Benchmark Framing
+
+### 1. Definitive Claims Without Validation Disclaimers
+
+**README line 6-8**:
+```markdown
+A learning-focused toolkit for artwork cloaking, style-drift experiments, and benchmark-driven iteration.
+Built for study, reproducibility, and honest anti-mimicry evaluation rather than marketing claims.
+```
+
+Claims "honest anti-mimicry evaluation" but:
+- No actual mimicry evaluation (LoRA/DreamBooth training) published
+- Results based entirely on proxy metrics
+- "Honest" framing contradicts incomplete evaluation
+
+**README line 48-50 ("Current Study Snapshot")**:
+```markdown
+## Current Study Snapshot
+
+Current local report highlights:
+```
+
+"Current local report" suggests these are preliminary, but table that follows uses definitive language:
+
+**README lines 54-58**:
+```markdown
+| `balanced` | `42.1` | `36.24` | `0.9346` | better visual quality, good study baseline |
+| `subject` | `51.5` | `30.53` | `0.8270` | stronger drift for subject-style protection experiments |
+| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
+| `blindfold` | `61.1` | `26.53` | `0.6114` | strongest current anti-readability preset, largest fidelity cost |
+```
+
+**Framing issues**:
+- "stronger drift" implies effectiveness, not just feature space drift
+- "strongest current anti-readability" makes absolute claim
+- No visible warnings that these are proxy scores, not validated protection
+- Numbers presented without error bars, confidence intervals, or sample sizes
+
+### 2. Critical Disclaimer Buried After Results
+
+**README line 60** (after results table):
+```markdown
+The `Protection Score` is an internal proxy derived from embedding and style similarity after robustness transforms. It is useful for relative comparisons inside this repository, not as a universal guarantee against all AI systems.
+```
+
+**Problem**: This critical limitation appears **after** the results, not before:
+- Users see impressive numbers first
+- Form impression of strong protection
+- May not read fine print disclaimer
+- Classic "dark pattern" in presentation
+
+**Better framing would put disclaimer FIRST**:
+```markdown
+⚠️ **Important Limitation**: All scores below are proxy metrics (feature drift in ResNet18 space), NOT validated against actual mimicry models. Real-world effectiveness is unknown. See [Ground-Truth Validation Issue](#).
+
+| Profile | Protection Score (proxy) | PSNR | SSIM | ...
+```
+
+### 3. Suggestive Profile Names Imply Validation
+
+**Profile names** (README lines 183-191):
+```markdown
+| `fortress` | maximize protection within the current proxy approach | visible image changes |
+| `blindfold` | aggressive anti-readability mode | strongest local score, harshest visual trade-off |
+```
+
+**Framing issues**:
+- "fortress" suggests impenetrable defense (not proven)
+- "blindfold" suggests complete blocking of readability (not validated against actual models)
+- "maximize protection" is only within proxy space, not actual protection
+
+**More honest names**:
+- `fortress` → `high_proxy_score` or `aggressive_perturbation`
+- `blindfold` → `maximum_drift` or `extreme_perturbation`
+
+### 4. "Benchmark" Terminology Suggests Validation
+
+**Multiple uses of "benchmark"**:
+- README section: "Current Study Snapshot" (line 48)
+- CLI command: `auralock benchmark` (line 139)
+- Documentation: "Benchmark Design" (docs/system-design/10_BENCHMARK_DESIGN.md)
+
+**But**: What's called "benchmark" is actually **profile comparison on proxy metrics**, not validation against actual mimicry.
+
+**Misleading terminology**:
+- "Benchmark" implies comparison against ground truth or baselines
+- Users expect "benchmarked" systems have been validated
+- Reality: Only internal profile comparisons on proxy metrics
+
+**More honest terminology**:
+- "Profile Comparison" instead of "Benchmark"
+- "Proxy Metric Evaluation" instead of "Benchmark Results"
+- "Preliminary Analysis" instead of "Current Study Snapshot"
+
+### 5. Repository Description Overstates Capabilities
+
+**README line 30** (Project Overview):
+```markdown
+**AuraLock** is a study repository for artwork cloaking and anti-mimicry evaluation.
+```
+
+"Anti-mimicry evaluation" suggests evaluation has been done. But:
+- No published mimicry evaluation results
+- Infrastructure exists (LoRA benchmark harness) but unused
+- Only proxy metrics have been evaluated
+
+**More accurate**:
+```markdown
+**AuraLock** is a study repository for artwork cloaking and **developing** anti-mimicry evaluation **methodology**.
+```
+
+### 6. Workflow Diagram Implies Validation
+
+**README lines 70-81** (Workflow diagram):
+```mermaid
+flowchart LR
+ ...
+ G["Local profile benchmark"]
+ H["DreamBooth / LoRA benchmark planning"]
+ I["Compare results and iterate"]
+```
+
+**Shows**: "DreamBooth / LoRA benchmark planning" (step H)
+
+**Implies**: This is part of the standard workflow
+
+**Reality**: Step H is infrastructure that exists but has never been executed with published results. The diagram makes it seem like ground-truth validation is standard practice, when it's actually missing.
+
+### 7. Acknowledgements Suggest Validation Parity
+
+**README line 243**:
+```markdown
+AuraLock is a learning project shaped by ideas discussed around adversarial artwork protection and anti-mimicry evaluation, especially directions associated with Mist-v2, StyleGuard, Anti-DreamBooth, and related open research.
+```
+
+**By mentioning** Mist, StyleGuard, Anti-DreamBooth (all rigorously evaluated methods):
+- Creates association with validated approaches
+- Users may assume AuraLock has similar validation rigor
+- **Reality**: AuraLock has not published comparable validation
+
+**More honest**:
+```markdown
+AuraLock is inspired by Mist-v2, StyleGuard, Anti-DreamBooth, and related research. **Unlike these methods, AuraLock has not yet undergone ground-truth mimicry validation.** See Issue #X for validation roadmap.
+```
+
+### 8. Claims of "Honest" Framing Are Contradictory
+
+**README line 7-8**:
+```markdown
+Built for study, reproducibility, and honest anti-mimicry evaluation rather than marketing claims.
+```
+
+**But the very next sections**:
+- Present unvalidated results without prominent warnings (line 52-58)
+- Use marketing-style profile names ("fortress", "blindfold")
+- Bury critical limitations after impressive numbers
+- Frame proxy metrics as "benchmark results"
+
+**This is exactly what marketing does**: Eye-catching claims up front, disclaimers in fine print.
+
+**Contradiction**: Can't claim "honest evaluation" while using marketing presentation patterns.
+
+## Why This Can Mislead Users
+
+### Scenario 1: User Deploys Ineffective Protection
+
+Artist reads README:
+```markdown
+| `fortress` | `53.2` | ... | maximize protection within the current proxy approach
+```
+
+Artist thinks:
+- "Fortress profile provides maximum protection"
+- "53.2 score is strong, validated protection"
+- "This will prevent AI from copying my style"
+
+Artist deploys fortress profile on entire portfolio:
+- Images are visibly degraded (PSNR=29.08, SSIM=0.7858)
+- But protection may actually be weak (not validated)
+- AI models may still successfully copy style
+- Artist lost quality for questionable protection benefit
+
+### Scenario 2: False Security
+
+Scenario:
+1. Artist uses AuraLock protection
+2. Shares "protected" images publicly
+3. Assumes style is safe from mimicry
+4. Competitor trains DreamBooth on "protected" images
+5. Protection fails (was never validated to work)
+6. Artist's style is copied despite protection
+
+**Artist trusted "benchmark results" that were never validated against actual mimicry.**
+
+### Scenario 3: Institutional Deployment
+
+Art platform reads:
+```markdown
+Built for study, reproducibility, and honest anti-mimicry evaluation
+```
+
+Platform assumes:
+- AuraLock has been rigorously evaluated
+- Protection effectiveness is validated
+- Can deploy as defense for artists on platform
+
+Platform integrates AuraLock:
+- Markets as "AI protection" to artists
+- Artists pay premium for protected uploads
+- Protection may be ineffective (not validated)
+- Platform liable for false security claims
+
+### Scenario 4: Academic Misattribution
+
+Researcher cites AuraLock:
+```
+[12] demonstrated effective artwork protection with fortress profile achieving 53.2 protection score...
+```
+
+**Implied**: Protection has been validated
+
+**Reality**: Only proxy metrics measured, no ground truth
+
+**Result**: Misleading citation propagates through literature
+
+## Evidence of Misleading Framing
+
+### 1. Disclaimer Placement
+
+Critical limitation (line 60) appears **after** impressive results (lines 52-58):
+- Classic persuasion technique: Impact first, limitations later
+- Many users won't read past results table
+- Contradicts claim of "honest evaluation"
+
+### 2. Comparison with Honest Framing
+
+**Current (misleading)**:
+```markdown
+## Current Study Snapshot
+
+| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
+
+The Protection Score is an internal proxy... not as a universal guarantee.
+```
+
+**Honest framing would be**:
+```markdown
+## Current Study Snapshot (PROXY METRICS ONLY - NOT VALIDATED)
+
+⚠️ **Critical Limitation**: All scores are unvalidated proxy metrics. Real-world protection effectiveness is unknown. See Issue #X.
+
+| Profile | Proxy Score (ResNet18 drift) | PSNR | SSIM | Notes |
+| `fortress` | `53.2` (unvalidated) | `29.08` | `0.7858` | High feature drift, visual quality loss. Real protection unknown. |
+
+**Need for Validation**: See Issue #X for ground-truth LoRA/DreamBooth validation roadmap.
+```
+
+### 3. GitHub Repository Description
+
+Repository tagline:
+> "A learning-focused toolkit for artwork cloaking, style-drift experiments, and benchmark-driven iteration."
+
+**"Cloaking" suggests hiding artwork from AI detection**. This is the intended capability, but:
+- Not validated to work
+- May provide false sense of security
+- More accurate: "A toolkit for **studying** artwork cloaking..."
+
+### 4. Badge Display
+
+README lines 11-17 show badges:
+```markdown
+
+
+```
+
+**Gives impression of** mature, validated software. But:
+- Tests validate code functionality, not protection effectiveness
+- No badge for "ground-truth validation: NONE"
+- No badge for "mimicry prevention: NOT MEASURED"
+
+**More honest badge set would include**:
+```markdown
+
+
+```
+
+## Proposed Fixes
+
+### Fix 1: Prominent Disclaimers at Top of README
+
+Add to README immediately after project overview (before results):
+```markdown
+## ⚠️ Current Validation Status
+
+**IMPORTANT: All protection scores are unvalidated proxy metrics.**
+
+| Status | Description |
+|--------|-------------|
+| ✅ **Implemented** | Feature extraction, proxy metrics, profile comparison |
+| ✅ **Implemented** | LoRA/DreamBooth benchmark infrastructure |
+| ❌ **Missing** | Ground-truth mimicry validation results |
+| ❌ **Missing** | Comparison against academic baselines (Anti-DreamBooth, Mist) |
+| ❌ **Missing** | Independent verification of protection effectiveness |
+
+**What This Means for Users**:
+- Protection scores measure ResNet18 feature drift, NOT actual mimicry prevention
+- No evidence that protection prevents DreamBooth/LoRA from learning your style
+- Use at your own risk; effectiveness is unproven
+
+**Validation Roadmap**: See [Issue #X](#) for ground-truth validation plan.
+```
+
+### Fix 2: Reframe Results Table with Warnings
+
+Update results table (lines 52-58):
+```markdown
+## Current Study Snapshot (Proxy Metrics - Unvalidated)
+
+⚠️ All scores below are **proxy metrics only**. Real-world protection effectiveness has **not been validated** against actual mimicry models.
+
+| Profile | Protection Score
(proxy, unvalidated) | PSNR | SSIM | Validation Status |
+|---------|-------------------------------------------|------|------|-------------------|
+| `balanced` | `42.1` | `36.24` | `0.9346` | ❌ No ground-truth validation |
+| `fortress` | `53.2` | `29.08` | `0.7858` | ❌ No ground-truth validation |
+| `blindfold` | `61.1` | `26.53` | `0.6114` | ❌ No ground-truth validation |
+
+**Protection Score Interpretation**:
+- Measures: Feature drift in ResNet18 space after robustness transforms
+- Does NOT measure: Actual prevention of style mimicry by DreamBooth/LoRA
+- Correlation with mimicry prevention: **Unknown** (not validated)
+
+**Before deploying**: Understand that protection effectiveness is unproven. See [Validation Status](#) section above.
+```
+
+### Fix 3: Honest Profile Names and Descriptions
+
+Update profile table (lines 183-191):
+```markdown
+| Profile | Goal | Status |
+|---------|------|--------|
+| `safe` | Low perturbation, prioritize quality | ⚠️ Protection unvalidated |
+| `balanced` | Medium perturbation, balance quality/proxy-score | ⚠️ Protection unvalidated |
+| `strong` | Higher perturbation, increase proxy score | ⚠️ Protection unvalidated |
+| `fortress` | Maximum proxy score (may not equal real protection) | ⚠️ Protection unvalidated |
+| `blindfold` | Highest feature drift (real effectiveness unknown) | ⚠️ Protection unvalidated |
+
+**Important**: Profile names suggest protection strength, but this is based on unvalidated proxy metrics. Real-world effectiveness is unknown.
+```
+
+### Fix 4: Accurate Benchmark Terminology
+
+Replace all instances of:
+- "benchmark results" → "proxy metric comparison"
+- "benchmark" command → "compare-profiles" or "evaluate-metrics"
+- "protection score" → "proxy protection score" or "feature drift score"
+
+Update CLI:
+```bash
+# OLD (misleading)
+$ auralock benchmark artwork.png --profiles safe,balanced,strong
+
+# NEW (honest)
+$ auralock compare-profiles artwork.png --profiles safe,balanced,strong
+# or
+$ auralock evaluate-proxy-metrics artwork.png --profiles safe,balanced,strong
+```
+
+### Fix 5: Validation Status Badges
+
+Add honest status badges to README:
+```markdown
+
+
+
+
+```
+
+### Fix 6: Update Repository Description
+
+**GitHub repository description** (currently):
+> "A learning-focused toolkit for artwork cloaking, style-drift experiments, and benchmark-driven iteration."
+
+**Should be**:
+> "A research toolkit for studying adversarial artwork protection. Validation against real mimicry models is ongoing. Protection effectiveness not yet proven."
+
+### Fix 7: Add "Limitations" Section to README
+
+Add prominent limitations section:
+```markdown
+## Known Limitations
+
+### Validation Gaps
+- ❌ No ground-truth mimicry validation (LoRA/DreamBooth training on protected images)
+- ❌ No comparison against academic baselines (Anti-DreamBooth, Mist, Glaze)
+- ❌ No independent verification of protection claims
+- ❌ Proxy metrics may not correlate with actual protection
+
+### Robustness Gaps
+- ❌ No JPEG compression testing (most common purification attack)
+- ❌ Limited transform suite (4 transforms, missing CLIP/VAE preprocessing)
+- ❌ Single feature extractor (ResNet18 only, not CLIP/DINO)
+
+### Methodology Gaps
+- ❌ No train/val/test split enforcement (risk of overfitting)
+- ❌ No reproducibility infrastructure (missing environment capture)
+- ❌ Cherry-picking not prevented (no systematic validation)
+
+**For detailed technical analysis, see**:
+- [Issue #1: Proxy Metrics Don't Reflect Real Protection](#)
+- [Issue #2: Missing Ground-Truth Validation](#)
+- [Issue #3: Weak Robustness Testing](#)
+- [Additional issues...](#)
+
+**Users should understand these limitations before deploying protection.**
+```
+
+## Acceptance Criteria
+
+### Phase 1: Immediate Disclaimer Fixes (Week 1)
+- [ ] Add prominent "⚠️ Current Validation Status" section at top of README
+- [ ] Reframe results table with "proxy, unvalidated" labels
+- [ ] Move critical disclaimer from line 60 to before results (line 50)
+- [ ] Update all "protection score" references to "proxy protection score"
+
+### Phase 2: Terminology Fixes (Week 1-2)
+- [ ] Replace "benchmark" with "proxy metric comparison" in user-facing docs
+- [ ] Rename `auralock benchmark` to `auralock compare-profiles` or add `--proxy-only` flag
+- [ ] Update profile descriptions to clarify proxy-only validation
+- [ ] Add "Limitations" section to README
+
+### Phase 3: Badge and Metadata Updates (Week 2)
+- [ ] Add validation status badges (red for missing validation)
+- [ ] Update repository description to emphasize research/unvalidated status
+- [ ] Add "research prototype" / "unvalidated" label to GitHub topics
+
+### Phase 4: Documentation Consistency (Week 2-3)
+- [ ] Audit all documentation for definitive claims without disclaimers
+- [ ] Add validation status to all benchmark-related docs
+- [ ] Update workflow diagram to show missing validation steps
+- [ ] Add "Known Limitations" to contributing guide
+
+### Phase 5: User-Facing Warnings (Week 3)
+- [ ] Add CLI warnings when protection commands are used:
+ ```
+ ⚠️ Warning: Protection effectiveness not validated against real mimicry models.
+ Proxy metrics only. Use at your own risk. See docs/LIMITATIONS.md
+ ```
+- [ ] Add warnings to all JSON reports
+- [ ] Add warnings to web UI (if applicable)
+
+### Phase 6: Honest Communication Policy (Ongoing)
+- [ ] Establish policy: Never claim protection without validation evidence
+- [ ] PR template checklist: "Does this claim require validation evidence?"
+- [ ] CI check: Flag PRs that add benchmark claims without disclaimers
+- [ ] Community guideline: Encourage honest reporting of limitations
+
+## Additional Context
+
+### Analogy: Medical Drug Trials
+
+Imagine a pharmaceutical company:
+```
+Drug X showed promising results in cell culture studies!
+[Shows impressive charts]
+
+*fine print: Not tested in animals or humans. Safety and efficacy unproven.
+```
+
+**This would be unacceptable**. Cell culture (proxy) is not enough. Must validate in actual patients (ground truth).
+
+**Same principle applies here**:
+- ResNet18 feature drift (proxy) is not enough
+- Must validate against actual mimicry models (ground truth)
+- Can't market as "protection" without validation
+
+### Defense Against Criticism
+
+Some may argue:
+> "But we disclosed it's a learning project / research prototype / proxy metric!"
+
+**Response**:
+- Disclosures must be **prominent**, not buried
+- Can't offset misleading framing with fine print
+- Research prototypes must be clearly labeled as such **everywhere**
+- "Honest evaluation" claim sets higher standard
+
+### Precedent: Security Vulnerability Disclosure
+
+When security researchers find vulnerabilities:
+- Use clear, unambiguous language
+- Put severity ratings up front (CRITICAL, HIGH, MEDIUM, LOW)
+- Don't bury impact in fine print
+
+**Same principle for unvalidated protection**:
+- Validation status should be first thing users see
+- Can't hide "not validated" in paragraph 3
+
+## References
+
+- README.md lines 6-8 - Claims "honest evaluation" without validation
+- README.md lines 52-58 - Results without prominent validation warnings
+- README.md line 60 - Critical disclaimer buried after results
+- README.md lines 183-191 - Suggestive profile names without validation
+- Repository description - "Cloaking" implies proven effectiveness
+- FTC guidance on disclosure requirements for material claims
+- ACM Code of Ethics - Honest representation of capabilities