Skip to content

Add prominent warnings for protection score proxy metric - #14

Merged
VoDaiLocz merged 2 commits into
mainfrom
claude/fix-protection-score-calculation
Mar 31, 2026
Merged

Add prominent warnings for protection score proxy metric#14
VoDaiLocz merged 2 commits into
mainfrom
claude/fix-protection-score-calculation

Conversation

@Claude

@Claude Claude AI commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

The protection score is a proxy metric derived from ResNet18 feature drift, not validated against real-world mimicry attacks like DreamBooth or LoRA. Users were not adequately warned that high scores don't guarantee actual protection.

Changes

Core Metrics (src/auralock/core/metrics.py)

  • Added warning constants: PROTECTION_SCORE_WARNING, PROTECTION_SCORE_DISCLAIMER, PROTECTION_SCORE_METRIC_TYPE
  • Modified get_protection_readability_report() to include warning metadata in all responses
  • Updated assessment labels to include "(proxy space)" suffix (e.g., "Strong (proxy space)" instead of "Strong")

CLI Output (src/auralock/cli.py)

  • Added warning row to protection readability table
  • Display prominent yellow warning after protection reports in protect, analyze, batch, and benchmark commands
  • Warnings appear both inline in tables and as standalone messages for visibility

Example output:

Protection score: 48.2/100
Assessment: Strong (proxy space)
⚠️  Warning: Protection Score is a proxy metric. Real-world effectiveness is NOT validated.

⚠️  Protection Score is a proxy metric. Real-world effectiveness is NOT validated.
This score measures drift in ResNet18 feature space, NOT actual mimicry prevention...

Documentation (README.md)

  • Moved warning from buried text to prominent blockquote before benchmark results table
  • Emphasizes lack of empirical validation against real-world attacks

Tests (src/tests/test_metrics.py)

  • Added 3 test cases verifying warning metadata presence
  • Verify assessment labels include "(proxy space)" suffix
  • All existing tests pass with new changes

@Claude Claude AI linked an issue Mar 31, 2026 that may be closed by this pull request
- Added warning constants in metrics.py (PROTECTION_SCORE_WARNING, PROTECTION_SCORE_DISCLAIMER, PROTECTION_SCORE_METRIC_TYPE)
- Updated get_protection_readability_report() to include warning metadata in all reports
- Modified assessment labels to include "(proxy space)" suffix
- Updated CLI to display warnings after all protection score outputs (protect, analyze, batch, benchmark commands)
- Updated README to move warning to prominent position before results table
- Added comprehensive tests for warning inclusion
- All existing tests continue to pass

Agent-Logs-Url: https://github.com/VoDaiLocz/Lock-ART./sessions/7df59577-3ab4-4d8b-b91e-dcb0d3b35ae7

Co-authored-by: VoDaiLocz <88762074+VoDaiLocz@users.noreply.github.com>
@Claude Claude AI changed the title [WIP] Fix protection score calculation for reliability Add prominent warnings for protection score proxy metric Mar 31, 2026
@Claude
Claude AI requested a review from VoDaiLocz March 31, 2026 13:16
@VoDaiLocz
VoDaiLocz marked this pull request as ready for review March 31, 2026 13:17
@VoDaiLocz
VoDaiLocz merged commit 59db4ac into main Mar 31, 2026
0 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

điểm bảo vệ k đủ tin cậy

2 participants