DeDup Pro

On-device AI duplicate file manager for Android — find, review, and safely delete duplicate photos and files using Gemma 4 E2B running entirely on your device.

Vibe coded. This codebase was built through iterative AI-assisted development (Claude). Architecture decisions, security constraints, and keeper logic were designed collaboratively and refined through council review sessions. Active human testing is ongoing — expect rough edges. If something breaks, the in-app "Share logs" button captures the crash report without needing ADB.

Sideload-only distribution (no Play Store). No cloud. No data leaves your phone.

---

Why this exists

Android devices accumulate duplicate files silently:

• Photos re-downloaded from WhatsApp, Telegram, and cloud backups
• Screenshots copied between folders
• Videos exported from editors alongside originals
• RAW+JPEG pairs where the JPEG is the only copy anyone needs

Existing tools either delete blindly (dangerous), require a PC (inconvenient), or upload your files to a cloud service (unacceptable for private photos).

DeDup Pro takes a different approach: an on-device Gemma 4 E2B model recommends which file to keep, but Kotlin code makes every final decision and you confirm every deletion. The AI advises; you approve; the engine acts. Nothing leaves the device.

---

Core intentions

Intention	How it's enforced
No accidental deletion	Write-before-delete audit log. Every deletion is recorded before it happens.
No blind deletion	Keeper rules surface a recommended file per group. User confirms before anything is removed.
No data leaving device	Gemma 4 E2B runs on-device via LiteRT-LM. No cloud API calls.
No AI running wild	ToolRegistry allowlist — model cannot call filesystem tools directly. ContentIsolationFilter — absolute paths never reach the model.
Transparent operation	Immutable audit CSV. In-app crash log sharing (no ADB needed).
Recoverable mistakes	Audit log records every deleted path. Integration with device Recycle Bin where available (Android 13+).

---

What it does

1. Scans internal storage for duplicate files using a two-pass hash strategy
   • Pass 1: group by file size (cheap)
   • Pass 2a: fullMd5 — identical byte-for-byte duplicates (EXACT group)
   • Pass 2b: contentMd5 — pixel stream hash with EXIF stripped (EXIF_VARIANT group: same photo, different metadata)
2. Recommends a keeper per group using a six-rule chain (KR-01…KR-06), with Gemma 4 E2B breaking ties on human-named filenames
3. Presents groups with keeper highlighted — user can override any decision
4. Confirms with an explicit deletion review before any file is removed
5. Deletes with write-before-delete audit trail
6. Reports bytes freed with an AI-narrated completion summary

---

Architecture overview

┌─────────────────────────────────────────────────────────┐
│  Phase 0 — Pure Kotlin engine (no AI)         ✅ DONE   │
│   FileScannerImpl → KeeperEngine → FileDeleter           │
│   FallbackKeeperAdvisor (rule-based keeper selection)   │
└─────────────────────────────────────────────────────────┘
          ↓ Phase 1 adds
┌─────────────────────────────────────────────────────────┐
│  Phase 1 — Edge Gallery Skill + Ktor MCP server         │
│   ScanWorker (WorkManager foreground service)           │
│   GemmaKeeperAdvisor → LiteRtInferenceEngine            │
│   dedupd MCP server (localhost:8765, auth token)        │
└─────────────────────────────────────────────────────────┘
          ↓ Phase 2 adds
┌─────────────────────────────────────────────────────────┐
│  Phase 2 — Standalone app with full UI        📋 PLANNED│
│   Compose UI (scan screen, groups list, delete confirm) │
│   Room database, Hilt DI, LiteRtInferenceEngineImpl     │
│   DedupCompanion foreground service                     │
└─────────────────────────────────────────────────────────┘

Full architecture detail: ARCHITECTURE.md

---

Project layout

dedup-pro/
├── engine/                     Pure-Kotlin scan/delete engine (no Android deps)
│   └── src/main/kotlin/com/deduppro/engine/
│       ├── scanner/            FileScannerImpl — walk, size-bucket, two-pass hash
│       ├── keeper/             KeeperEngine, FallbackKeeperAdvisor, KeeperAdvisor
│       ├── deleter/            FileDeleter — write-before-delete protocol
│       ├── logger/             AuditLogger
│       └── model/              Domain types (DuplicateGroup, ScannedFile, …)
│
├── app/                        Android application module
│   └── src/main/kotlin/com/deduppro/app/
│       ├── ai/                 GemmaKeeperAdvisor, LiteRtInferenceEngine interface
│       │   ├── guardrail/      GemmaGuardrail, ContentIsolationFilter, ToolRegistry
│       │   └── prompt/         PromptSanitizer, FileMetadata, FolderMetadata
│       ├── crash/              CrashLogger — filesDir crash reports + FileProvider share
│       ├── permission/         StoragePermissionHelper
│       ├── worker/             ScanWorker (WorkManager CoroutineWorker)
│       ├── DedupApp.kt         Application class — installs CrashLogger
│       └── MainActivity.kt     Temporary debug shell (Phase 2 replaces this)
│
└── mcp/                        Ktor MCP server (dedupd) — phased out in Phase 2
    └── SKILL.md                Edge Gallery Skill definition

---

Two-hash EXIF model

Hash	What is hashed	Group type
fullMd5	entire file bytes	EXACT — identical bit-for-bit
contentMd5	pixel stream, all APPn EXIF stripped	EXIF_VARIANT — same photo, different metadata

---

Keeper rules (KR-01 → KR-06)

Rules tried in chain order; first non-null result wins. KR-06 always returns.

Rule	Strategy
KR-01	Prefer file with earlier lastModified (original capture)
KR-02	Prefer larger file (higher quality / less compression)
KR-03	Prefer file in DCIM over Downloads
KR-04	Prefer file whose name matches human-named pattern (via Gemma)
KR-05	Prefer file in folder with more total files
KR-06	First file in group (deterministic fallback)

---

Security constraints

• MANAGE_EXTERNAL_STORAGE held only by dedupd / DedupCompanion
• AI cannot call filesystem-mutating tools directly (ToolRegistry allowlist)
• Absolute file paths never reach the Gemma model (ContentIsolationFilter)
• MCP server binds to 127.0.0.1 only — never 0.0.0.0
• Model URL must use HTTPS (checkUrlScheme assertion + CI gate)
• Release APK has debuggable=false (CI gate S-22)
• Session auth token: 256-bit SecureRandom, X-Dedup-Token header, 401 on mismatch
• Write-before-delete: audit log entry written and flushed before any file is removed

---

Requirements

• Android 10+ (API 29) — scoped storage permission model
• Android 12+ (API 31) — foreground service data-sync type
• ~500 MB free storage for model download (Phase 2)
• Java 17+ for building

---

Deploying / trying it now

See DEPLOYING.md — covers what works today, sideloading the APK, verifying crash logging, and starting the MCP server. Honest about what's stub vs real.

Building

See SETUP.md for full environment setup (JDK, Android SDK, emulator).

cd dedup-pro
.\gradlew.bat assembleDebug
adb install app\build\outputs\apk\debug\app-debug.apk

Running tests

.\gradlew.bat test                # all unit tests (JVM, no emulator needed)
.\gradlew.bat :engine:test        # engine module only
.\gradlew.bat :app:test           # app module only

---

Roadmap

See ROADMAP.md for full phased plan and issue tracking. GitHub issues are labeled by phase: phase-0, phase-1, phase-2.