Add validation warnings and metadata for Ground-Truth LoRA/DreamBooth benchmarking

README.md

@@ -47,18 +47,32 @@ The repository is organized around a practical study loop:
 
 ## Current Study Snapshot
 
-> **⚠️ Important Note on Protection Score:**
-> The `Protection Score` is a **proxy metric** that measures drift in ResNet18 feature space. It is **NOT validated** against real-world mimicry attacks like DreamBooth or LoRA. This score is useful for relative comparisons within this repository only, not as a universal guarantee of protection effectiveness. Real-world effectiveness has not been empirically validated.
+> **⚠️ CRITICAL: Metrics Not Validated Against Real Attacks**
+>
+> All protection metrics in AuraLock are **PROXY MEASUREMENTS** and have **NOT been validated** against real-world attacks like DreamBooth or LoRA fine-tuning.
+>
+> **The `Protection Score` measures:**
+> - Drift in ResNet18 feature space under transforms
+> - Useful ONLY for relative comparisons within this repository
+>
+> **The `Protection Score` does NOT measure:**
+> - ❌ Actual prevention of style mimicry
+> - ❌ Real DreamBooth/LoRA training outcomes
+> - ❌ Protection against production AI systems
+>
+> **Status:** Awaiting Phase 3 GPU validation (see [VALIDATION_RESULTS.md](docs/VALIDATION_RESULTS.md))
+>
+> Use these metrics for research and comparison purposes only. Do not interpret scores as guarantees of protection effectiveness.
 
 Current local report highlights:
 
-| Run | Protection Score | PSNR | SSIM | Notes |
-|-----|------------------|------|------|-------|
-| `balanced` | `42.1` | `36.24` | `0.9346` | better visual quality, good study baseline |
-| `subject` | `51.5` | `30.53` | `0.8270` | stronger drift for subject-style protection experiments |
-| `fortress` | `53.2` | `29.08` | `0.7858` | more aggressive, visibly harsher output |
-| `blindfold` | `61.1` | `26.53` | `0.6114` | strongest current anti-readability preset, largest fidelity cost |
-| `collective n000050 / set_B` | `22.8` avg | `37.78` avg | `0.9666` avg | correct benchmark direction, objective still needs tuning |
+| Run | Protection Score (⚠️ Proxy) | PSNR | SSIM | Validation | Notes |
+|-----|---------------------------|------|------|------------|-------|
+| `balanced` | `42.1` | `36.24` | `0.9346` | ❌ Not validated | better visual quality, good study baseline |
+| `subject` | `51.5` | `30.53` | `0.8270` | ❌ Not validated | stronger drift for subject-style protection experiments |
+| `fortress` | `53.2` | `29.08` | `0.7858` | ❌ Not validated | more aggressive, visibly harsher output |
+| `blindfold` | `61.1` | `26.53` | `0.6114` | ❌ Not validated | strongest current anti-readability preset, largest fidelity cost |
+| `collective n000050 / set_B` | `22.8` avg | `37.78` avg | `0.9666` avg | ❌ Not validated | correct benchmark direction, objective still needs tuning |
 
 ## Workflow
 
@@ -217,6 +231,7 @@ Lock-ART./
 
 ## Notes and Documentation
 
+- [Validation Results](docs/VALIDATION_RESULTS.md) - **Critical:** Current validation status and limitations
 - [Product Audit](docs/PRODUCT_AUDIT.md)
 - [Implementation Plan](docs/IMPLEMENTATION_PLAN.md)
 - [Research Roadmap](docs/RESEARCH_ROADMAP.md)

docs/RESEARCH_ROADMAP.md

@@ -46,16 +46,22 @@ Xây dựng và đánh giá phương pháp bảo vệ artwork theo hướng:
 - Tăng độ tin cậy CI cho các luồng benchmark khô (dry-run)
 
 ### Giai đoạn 3 (dài hạn: 2-3 tháng)
+- **⚠️ YÊU CẦU QUAN TRỌNG: Xác thực GPU là bắt buộc cho giai đoạn này**
 - Chạy benchmark thực tế trên GPU (LoRA/DreamBooth)
 - Tổng hợp kết quả có kiểm định cơ bản
 - Đề xuất hướng cải tiến objective dựa trên dữ liệu thực nghiệm
+- **Lưu ý:** Tất cả các số liệu proxy hiện tại CHƯA được xác thực với các cuộc tấn công thực tế
+- **Mục tiêu chính:** Thiết lập mối tương quan giữa protection score proxy và hiệu quả bảo vệ thực tế
+- **Xem chi tiết:** [VALIDATION_RESULTS.md](VALIDATION_RESULTS.md)
 
 ---
 
 ## 4) Năng lực hạ tầng khuyến nghị
 
 - **CPU workflow**: đủ cho protect/analyze/batch cơ bản và benchmark dry-run
 - **GPU workflow**: cần cho benchmark huấn luyện thực tế LoRA/DreamBooth
+  - **⚠️ Chưa có kết quả xác thực:** Hiện tại tất cả số liệu đều là proxy, chưa có validation thực tế trên GPU
+  - GPU là yêu cầu bắt buộc để xác thực hiệu quả bảo vệ thực tế (Giai đoạn 3)
 - **Container workflow**: ưu tiên Docker benchmark runtime để giảm sai lệch môi trường
 
 ---
@@ -75,4 +81,5 @@ Mỗi vòng lặp nên có:
 
 - `README.md`: hướng dẫn sử dụng và snapshot kết quả hiện tại
 - `docs/PRODUCT_AUDIT.md`: bối cảnh cải tiến sản phẩm và các gap còn lại
+- `docs/VALIDATION_RESULTS.md`: **Trạng thái xác thực và hạn chế hiện tại** (quan trọng)
 - `notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb`: notebook benchmark trên Colab/GPU

docs/VALIDATION_RESULTS.md

@@ -0,0 +1,184 @@
+# Validation Results
+
+## Status: Not Yet Validated Against Ground-Truth Attacks
+
+**Last Updated:** 2026-04-01
+
+---
+
+## Executive Summary
+
+AuraLock's protection metrics are **proxy measurements** that have **NOT** been validated against real-world mimicry attacks such as DreamBooth, LoRA fine-tuning, or production-grade style transfer systems.
+
+The `Protection Score` and related metrics measure drift in ResNet18 feature space under robustness transforms. While these proxy metrics are useful for:
+- Relative comparisons within this repository
+- Quick iteration during development
+- Understanding feature-space perturbations
+
+They **DO NOT** guarantee protection against actual AI style mimicry in production environments.
+
+---
+
+## What Has Been Tested
+
+✅ **Proxy Metrics (Feature Space Drift)**
+- Embedding similarity using ResNet18 features
+- Style similarity using Gram matrices
+- Robustness under transforms (blur, resize, JPEG compression)
+- Image quality metrics (PSNR, SSIM)
+
+✅ **Internal Consistency**
+- Profile comparisons show expected trade-offs
+- Quality vs protection curves behave predictably
+- Batch processing produces consistent results
+
+✅ **Technical Correctness**
+- All tests pass in CI/CD pipeline
+- Code follows best practices
+- Infrastructure is reproducible
+
+---
+
+## What Has NOT Been Validated
+
+❌ **Real-World Attack Prevention**
+- No validation against actual DreamBooth training
+- No validation against LoRA fine-tuning
+- No validation against commercial style-transfer APIs
+- No validation against adversarial purification techniques
+- No validation with production-scale training datasets
+
+❌ **Ground-Truth Effectiveness Metrics**
+- No A/B testing with protected vs unprotected training data
+- No style similarity measurements from trained models
+- No human evaluation of generated outputs
+- No comparison with other protection methods (Glaze, Mist, etc.)
+
+❌ **Long-Term Robustness**
+- No testing against evolving model architectures
+- No testing against adaptive attacks
+- No testing against ensemble methods
+
+---
+
+## Why This Matters
+
+Without ground-truth validation, we cannot make claims about:
+1. **Actual Protection Effectiveness**: The proxy score may not correlate with real mimicry prevention
+2. **Attack Resistance**: Unknown behavior against adaptive or sophisticated attacks
+3. **Comparative Performance**: Cannot reliably compare with other protection tools
+4. **Production Readiness**: Unclear suitability for protecting valuable artwork
+
+---
+
+## Planned Validation Roadmap
+
+### Phase 1: Local Benchmark Infrastructure (Completed)
+- ✅ Proxy metric pipeline
+- ✅ Profile system with quality/protection trade-offs
+- ✅ Batch processing and reporting
+- ✅ Benchmark harness for DreamBooth/LoRA
+
+### Phase 2: Dry-Run Testing (Current)
+- ✅ Manifest generation for benchmark jobs
+- ✅ Docker runtime setup
+- ✅ Colab notebook for free GPU access
+- 🔄 Preflight validation and job planning
+
+### Phase 3: GPU Ground-Truth Validation (Planned - Requires GPU Access)
+- ⏳ Run baseline DreamBooth/LoRA training on unprotected artwork
+- ⏳ Run protected training with AuraLock-processed images
+- ⏳ Generate outputs from both models with identical prompts
+- ⏳ Compare style preservation using:
+  - Human evaluation
+  - Automated style metrics (CLIP similarity, FID, etc.)
+  - Feature-space analysis
+- ⏳ Document correlation between proxy score and actual effectiveness
+
+### Phase 4: Comprehensive Evaluation (Future)
+- ⏳ Multi-dataset validation across art styles
+- ⏳ Comparison with other protection methods
+- ⏳ Adaptive attack testing
+- ⏳ Publication of peer-reviewed results
+
+---
+
+## How to Interpret Current Metrics
+
+### Protection Score (0-100)
+**What it measures:** Drift in ResNet18 feature space under robustness transforms
+
+**What it does NOT measure:**
+- Actual style mimicry prevention
+- Real DreamBooth/LoRA training outcomes
+- Human perceptual similarity of generated art
+
+**Interpretation:**
+- Higher scores = more feature drift (potentially better protection)
+- Use ONLY for relative comparisons within this repository
+- DO NOT interpret as percentage of protection effectiveness
+- DO NOT compare directly with scores from other tools
+
+### Quality Metrics (PSNR, SSIM)
+**What they measure:** Perceptual similarity between original and protected images
+
+**What they do NOT measure:**
+- Effectiveness of protection
+- Robustness against attacks
+
+**Interpretation:**
+- Higher PSNR/SSIM = less visible perturbations
+- Use to evaluate quality trade-offs
+- Balance with protection score based on use case
+
+---
+
+## Transparency Statement
+
+This repository prioritizes **honest evaluation** over marketing claims. We explicitly document:
+- ✅ What we've tested
+- ❌ What we haven't tested
+- 🔬 What we're planning to test
+
+We encourage users to:
+1. Understand the limitations of proxy metrics
+2. Wait for Phase 3 validation before production use
+3. Contribute GPU resources or validation results
+4. Report any real-world testing outcomes
+
+---
+
+## Contributing Validation Data
+
+If you have access to GPU resources and want to help validate AuraLock:
+
+1. **Use our Colab notebook:** `notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb`
+2. **Follow the benchmark harness:** See `src/auralock/benchmarks/`
+3. **Share results:** Open an issue with your findings
+4. **Compare methods:** Test against unprotected baseline and other tools
+
+We welcome community contributions to ground-truth validation efforts.
+
+---
+
+## References
+
+- [Research Roadmap](RESEARCH_ROADMAP.md) - Planned validation timeline
+- [Product Audit](PRODUCT_AUDIT.md) - Current system capabilities
+- [Colab Benchmark Notebook](../notebooks/AuraLock_LoRA_Benchmark_Colab.ipynb) - GPU validation workflow
+- [Benchmark Infrastructure](../src/auralock/benchmarks/) - Technical implementation
+
+---
+
+## Disclaimer
+
+AuraLock is a research and learning project. The protection mechanisms are experimental and have not been validated against real-world attacks. Users should:
+
+- **NOT** rely solely on AuraLock for protecting valuable or commercial artwork
+- **NOT** interpret proxy metrics as guarantees of protection
+- **NOT** assume protection will work against all attack types
+- **DO** understand this is an educational and research tool
+- **DO** wait for validated results before production use
+- **DO** use multiple protection layers and legal safeguards
+
+For production artwork protection, consult with legal professionals and consider using multiple protection methods in combination.

src/auralock/cli.py

@@ -82,7 +82,7 @@ def _render_quality_table(report: dict[str, object]) -> Table:
 
 
 def _render_readability_table(report: dict[str, object]) -> Table:
-    table = Table(title="Protection Readability")
+    table = Table(title="Protection Readability (⚠️ PROXY - NOT VALIDATED)")
     table.add_column("Metric", style="cyan")
     table.add_column("Value", style="green")
 
@@ -93,10 +93,11 @@ def _render_readability_table(report: dict[str, object]) -> Table:
         f"{report['robust_style_similarity']:.4f}",
     )
     table.add_row(
-        "Protection score",
+        "Protection score (proxy)",
         f"{report['protection_score']:.1f}/100",
     )
     table.add_row("Assessment", str(report["assessment"]))
+    table.add_row("Validation status", "❌ Not validated against real attacks")
 
     # Add warning row if present
     if "warning" in report:
@@ -176,12 +177,12 @@ def _render_batch_table(summary: BatchProtectionSummary) -> Table:
 def _render_profile_summary_table(
     profile_summaries: dict[str, dict[str, object]],
 ) -> Table:
-    table = Table(title="Profile Summary")
+    table = Table(title="Profile Summary (⚠️ Protection scores are proxy metrics)")
     table.add_column("Profile", style="cyan")
     table.add_column("Images", style="green")
     table.add_column("Avg PSNR", style="yellow")
     table.add_column("Avg SSIM", style="yellow")
-    table.add_column("Avg Protect", style="magenta")
+    table.add_column("Avg Protect (proxy)", style="magenta")
     table.add_column("Avg Runtime", style="green")
 
     for profile, summary in profile_summaries.items():

src/auralock/services/protection.py

@@ -91,6 +91,13 @@ def to_report_dict(
             "perturbation_linf": self.perturbation_linf,
             "device": self.device,
             "model_name": self.model_name,
+            "validation_metadata": {
+                "is_validated": False,
+                "validation_status": "not_validated",
+                "validation_method": None,
+                "validation_date": None,
+                "notes": "Protection metrics are proxy measurements not validated against real attacks like DreamBooth or LoRA.",
+            },
         }
         if output_path is not None:
             payload["output_path"] = output_path
@@ -136,6 +143,13 @@ def to_report_dict(self) -> dict[str, object]:
                 "failed_count": self.failed_count,
                 "outputs": self.outputs,
                 "failures": self.failures,
+                "validation_metadata": {
+                    "is_validated": False,
+                    "validation_status": "not_validated",
+                    "validation_method": None,
+                    "validation_date": None,
+                    "notes": "Protection metrics are proxy measurements not validated against real attacks like DreamBooth or LoRA.",
+                },
             }
         )
 
@@ -167,6 +181,13 @@ def to_report_dict(self) -> dict[str, object]:
                 "runtime_sec": self.runtime_sec,
                 "quality_report": self.quality_report,
                 "protection_report": self.protection_report,
+                "validation_metadata": {
+                    "is_validated": False,
+                    "validation_status": "not_validated",
+                    "validation_method": None,
+                    "validation_date": None,
+                    "notes": "Protection metrics are proxy measurements not validated against real attacks like DreamBooth or LoRA.",
+                },
             }
         )
 
@@ -188,6 +209,13 @@ def to_report_dict(self) -> dict[str, object]:
                 "image_count": self.image_count,
                 "entries": [entry.to_report_dict() for entry in self.entries],
                 "profile_summaries": self.profile_summaries,
+                "validation_metadata": {
+                    "is_validated": False,
+                    "validation_status": "not_validated",
+                    "validation_method": None,
+                    "validation_date": None,
+                    "notes": "Protection metrics are proxy measurements not validated against real attacks like DreamBooth or LoRA.",
+                },
             }
         )