Real-Time KYC Video Integrity & Deepfake Risk Detection System
ShadowKYC is a multi-layer video integrity validation system designed to identify indicators of deepfakes, replay attacks, presentation attacks, face manipulation, temporal inconsistencies, and environmental anomalies during Video KYC.
Instead of replacing existing KYC providers, ShadowKYC acts as an additional security and risk-analysis layer that can run alongside existing identity-verification systems.
Modern Video KYC systems can be exposed to attacks such as:
- AI-generated or face-swapped videos
- Replay attacks using previously recorded KYC footage
- Frozen-frame or screen-replay attacks
- Synthetic facial texture and frequency artifacts
- Abnormal facial geometry
- Identity drift during a session
- Lip/mouth movement inconsistencies
- Background and lighting manipulation
- Overlay and screen-capture artifacts
Traditional identity verification may establish who a person claims to be, but a separate integrity layer is useful for determining whether the presented video itself appears trustworthy.
ShadowKYC analyzes a KYC video at multiple levels and combines the resulting signals into a single explainable session-level risk score.
ββββββββββββββββββββββββββββ
β KYC Video Stream β
β Live Camera / Upload β
ββββββββββββββ¬ββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββ
β Frame Sampling & Face β
β Detection β
ββββββββββββββ¬ββββββββββββββ
β
ββββββββββββββββββββββββΌβββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β Layer 1 β β Layer 2 β β Layer 3 β
β Temporal β β Face Match β β Texture / β
β Liveness β β & Identity β β Frequency β
ββββββββ¬βββββββ ββββββββ¬βββββββ ββββββββ¬βββββββ
β β β
ββββββββββββββββ¬ββββββββ΄βββββββββββββββ¬ββββββββ
β β
βΌ βΌ
βββββββββββββββ βββββββββββββββ
β Layer 4 β β Layer 5 β
β Geometry / β β Lip-Sync β
β Pseudo-Depthβ β Integrity β
ββββββββ¬βββββββ ββββββββ¬βββββββ
β β
ββββββββββββ¬ββββββββββββ
β
ββββββββββββ΄βββββββββββ
β β
βΌ βΌ
βββββββββββββββ βββββββββββββββ
β Layer 6 β β Layer 7 β
β Environment ββββββββΆβ Meta Fusion β
β Integrity β β & Decision β
βββββββββββββββ ββββββββ¬βββββββ
β
βΌ
ββββββββββββββββββββββββββ
β Session Risk Assessmentβ
β LOW / MEDIUM / HIGH β
ββββββββββββββββββββββββββ
ShadowKYC currently uses seven complementary analysis layers.
Module: app/modules/layer_1_temporal_liveness.py
Analyzes temporal facial behavior to identify signs of replay or unnatural movement.
- MediaPipe FaceMesh landmarks
- Eye Aspect Ratio (EAR)
- Blink variation
- Head pose / yaw / pitch
- Temporal head movement
- Frozen-frame detection
- Perceptual hashing
- Face mesh availability
FROZEN_FRAME_REPLAY
L1_UNNATURAL_HEAD_STABILITY
L1_NO_BLINK_VARIATION
L1_EXTREME_HEAD_POSE
L1_NO_FACE_MESH
Module: app/modules/layer_2_face_match.py
Maintains a rolling facial baseline and checks whether the identity representation remains consistent throughout the session.
- HOG-based CPU-friendly face representation
- Cosine distance
- Rolling face baseline
- Embedding drift
- Sudden facial appearance changes
L2_FACE_SWAP_DETECTED
L2_IDENTITY_DRIFT
L2_SUDDEN_APPEARANCE_CHANGE
This layer is intended to identify identity continuity problems, rather than acting as a standalone identity-verification provider.
Module: app/modules/layer_3_texture_artifact.py
Analyzes facial texture and frequency characteristics that may become abnormal after synthetic generation, manipulation, compression, or presentation attacks.
- Laplacian variance
- FFT high-frequency energy ratio
- Simplified LBP variance
- Face ROI texture analysis
- Excessive smoothing
- Frequency-domain anomalies
BLUR_THRESHOLD = 80.0
FFT_LOW_THRESHOLD = 0.08
FFT_HIGH_THRESHOLD = 0.45
LBP_LOW_THRESHOLD = 20.0
L3_OVER_SMOOTH_FACE
L3_LOW_HF_ENERGY_AI_FACE
L3_GAN_GRID_ARTIFACT
L3_UNIFORM_SKIN_TEXTURE
The layer also records metrics such as:
l3_laplacian_var
l3_fft_hf_ratio
l3_lbp_var
Module: app/modules/layer_4_geometry.py
Uses facial landmarks and geometric relationships to identify abnormal facial proportions or temporal geometry changes.
- MediaPipe FaceMesh
- Eye-width ratio
- Eye-to-nose ratio
- Nose-to-chin ratio
- Facial symmetry
- Ratio drift across frames
- Eye-width distortion
SYMMETRY_THRESHOLD = 0.25
RATIO_DRIFT_THRESH = 0.15
HISTORY_LEN = 20
L4_FACE_ASYMMETRY
L4_LANDMARK_RATIO_DRIFT
L4_EYE_WIDTH_DISTORTION
L4_NO_LANDMARKS
Module: app/modules/layer_5_lipsync.py
Analyzes mouth movement over time to identify frozen or highly abnormal facial motion.
- MediaPipe FaceMesh
- Mouth Aspect Ratio (MAR)
- Mouth movement variance
- Optional audio RMS
- Optional audio/MAR correlation
MAR_FROZEN_THRESHOLD = 0.0005
MAR_ERRATIC_THRESHOLD = 0.08
HISTORY_LEN = 30
MIN_FRAMES_FOR_EVAL = 10
L5_MOUTH_FROZEN_NO_SPEECH
L5_ERRATIC_MOUTH_MOVEMENT
L5_AUDIO_LIP_MISMATCH
Note: The current implementation primarily uses visual mouth-motion analysis. Audio correlation is optional and should not be described as a complete audio deepfake detector.
Module: app/modules/layer_6_environment.py
Analyzes the environment surrounding the face to detect inconsistencies that can occur during replay, overlays, screen capture, or manipulated video.
- Color histogram drift
- Bhattacharyya distance
- Background motion
- Frame-difference analysis
- Edge shimmer / overlay halo
- Background flickering
HIST_DIFF_THRESHOLD = 0.35
BG_MOTION_THRESHOLD = 15.0
EDGE_SHIMMER_THRESHOLD = 0.12
HISTORY_LEN = 20
L6_SUDDEN_LIGHTING_SHIFT
L6_BACKGROUND_MOTION_DETECTED
L6_EDGE_SHIMMER_OVERLAY
L6_BACKGROUND_FLICKERING
Module: app/modules/layer_7_meta_fusion.py
The final layer converts the outputs from the six detection layers into a unified risk score.
Each individual module produces a goodness score, where:
1.0 = clean / authentic signal
0.0 = suspicious signal
Risk contribution is calculated as:
risk contribution = 1 - layer score
| Layer | Signal | Weight |
|---|---|---|
| L1 | Temporal Liveness | 0.25 |
| L2 | Face Match | 0.20 |
| L3 | Texture / Frequency | 0.15 |
| L4 | Geometry | 0.15 |
| L5 | Lip-Sync | 0.10 |
| L6 | Environment | 0.15 |
| Total | 1.00 |
ShadowKYC also applies additional penalties when multiple suspicious signals contradict each other in meaningful ways.
| Combination | Additional Penalty |
|---|---|
| Unnatural head stability + background motion | +0.10 |
| Face swap + frozen-frame replay | +0.15 |
| Over-smooth face + eye-width distortion | +0.10 |
| Frozen mouth + background motion | +0.12 |
This allows the final decision to consider cross-layer evidence, rather than treating every layer independently.
The final risk score is classified into three levels.
| Risk Score | Classification | Recommendation |
|---|---|---|
< 0.35 |
π’ LOW RISK | APPROVE β Session appears authentic |
0.35 β < 0.65 |
π‘ MEDIUM RISK | REVIEW β Manual verification recommended |
>= 0.65 |
π΄ HIGH RISK | REJECT β High probability of fraud |
Example output:
{
"risk_score": 0.72,
"risk_pct": 72,
"classification": "HIGH_RISK",
"recommendation": "REJECT β High probability of fraud. Do not proceed.",
"top_reasons": [
"Face swap indicators detected",
"Frozen frame replay detected",
"Cross-layer contradiction detected"
]
} ββββββββββββββββββββββββ
β Frontend β
β React + Vite β
β Tailwind CSS β
ββββββββββββ¬ββββββββββββ
β
HTTP / WebSocket
β
βΌ
ββββββββββββββββββββββββ
β FastAPI β
β Backend β
ββββββββββββ¬ββββββββββββ
β
βΌ
ββββββββββββββββββββββββ
β Orchestrator β
β app/core/orchestratorβ
ββββββββββββ¬ββββββββββββ
β
ββββββββββββββββββββββββΌβββββββββββββββββββββββ
β β β
βΌ βΌ βΌ
ββββββββββββ ββββββββββββ ββββββββββββ
β L1 β β L2 β β L3 β
β Liveness β βFace Matchβ β Texture β
ββββββ¬ββββββ ββββββ¬ββββββ ββββββ¬ββββββ
β β β
βββββββββββββββββββββββΌββββββββββββββββββββββ
β
ββββββββββ΄βββββββββ
β β
βΌ βΌ
ββββββββββββ ββββββββββββ
β L4 β β L5 β
β Geometry β β Lip-Sync β
ββββββ¬ββββββ ββββββ¬ββββββ
β β
ββββββββββ¬βββββββββ
β
βΌ
ββββββββββββ
β L6 β
βEnvironmentβ
ββββββ¬ββββββ
β
βΌ
βββββββββββββββ
β L7 β
β Meta Fusion β
ββββββββ¬βββββββ
β
βΌ
ββββββββββββββββββββ
β Risk + Evidence β
β + Recommendation β
ββββββββββββββββββββ
ShadowKYC2/
β
βββ app/
β βββ core/
β β βββ orchestrator.py
β β
β βββ modules/
β β βββ layer_1_temporal_liveness.py
β β βββ layer_2_face_match.py
β β βββ layer_3_texture_artifact.py
β β βββ layer_4_geometry.py
β β βββ layer_5_lipsync.py
β β βββ layer_6_environment.py
β β βββ layer_7_meta_fusion.py
β β
β βββ main.py
β
βββ frontend/
β βββ src/
β βββ pages/
β βββ DevDashboard.jsx
β
βββ evidence/
βββ reports/
βββ temp_uploads/
β
βββ tests/
β
βββ requirements.txt
βββ implementation_plan.md
βββ supabase_schema.sql
βββ db.json
βββ run_server.bat
βββ README.md
ShadowKYC supports two primary analysis workflows.
The live workflow is designed for real-time KYC sessions.
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β SHADOW KYC β LIVE β
ββββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββ€
β β β
β CAMERA FEED β RISK ANALYSIS β
β β β
β ββββββββββββββββββ β Risk Score: 42% β
β β β β MEDIUM RISK β
β β CAMERA β β β
β β FEED β β L1 ββββββββββ β
β β β β L2 ββββββββββ β
β ββββββββββββββββββ β L3 ββββββββββ β
β β L4 ββββββββββ β
β β L5 ββββββββββ β
β β L6 ββββββββββ β
β β β
β [ START ] [ STOP ] β β Evidence β
β β β Identity drift β
ββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββ€
β FLAGGED FRAMES / EVENTS β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
The live interface can display:
- Camera feed
- Rolling risk score
- Layer status
- Detection flags
- Flagged frames
- Session timeline
- Start / stop controls
- Session report
Recorded KYC videos can also be uploaded and analyzed after the session.
Upload Video
β
βΌ
Frame Extraction
β
βΌ
Face Detection
β
βββ L1 Temporal Liveness
βββ L2 Face Match
βββ L3 Texture
βββ L4 Geometry
βββ L5 Lip-Sync
βββ L6 Environment
β
βΌ
L7 Meta Fusion
β
βΌ
Final Risk Report
The report can expose:
- Overall risk score
- Risk classification
- Recommendation
- Layer scores
- Detection flags
- Top reasons
- Contradiction penalties
- Evidence frames
- Session-level metadata
The backend is implemented with FastAPI.
Important API workflows include:
/analyze-session
/upload-recording
The frontend communicates with the backend using HTTP requests and WebSocket-based live-session communication.
Recommended environment:
- Python 3.10
- Node.js 18+
- npm
- Git
- Webcam for live testing
- Linux / WSL / Windows supported depending on dependency configuration
git clone https://github.com/Nirmal0804/ShadowKYC2.git
cd ShadowKYC2python -m venv .venv
.venv\Scripts\activatepython3 -m venv .venv
source .venv/bin/activatepip install -r requirements.txtuvicorn app.main:app --host 0.0.0.0 --port 8000 --reloadThe backend will be available at:
http://localhost:8000
FastAPI documentation:
http://localhost:8000/docs
Open another terminal:
cd frontend
npm install
npm run devThe frontend will normally be available through the Vite development server.
Run the test suite from the project root:
python -m unittest discover -s tests -vThe project should be tested after changes to:
- Individual detection layers
- Orchestrator logic
- API routes
- Risk fusion
- Frontend/backend communication
Browser Camera
β
βΌ
WebSocket
β
βΌ
FastAPI
β
βΌ
Orchestrator.start_session()
β
βΌ
process_single_frame()
β
βββββββββββββββ
β β
βΌ βΌ
Face Detection Frame Metadata
β
βΌ
Seven-Layer Analysis
β
βΌ
Rolling Session State
β
βΌ
Live Risk Updates
β
βΌ
Browser Dashboard
β
βΌ
finalize_live_session()
β
βΌ
Session Report
ShadowKYC is designed around evidence-based risk scoring rather than a single black-box prediction.
Instead of returning only:
FAKE
the system can provide:
Risk Score: 72%
Classification:
HIGH_RISK
Evidence:
- Frozen frame replay indicator
- Identity drift
- Facial texture anomaly
- Cross-layer contradiction
Recommendation:
Manual investigation / reject according to KYC policy
This makes the system more useful for fraud analysts and compliance workflows.
ShadowKYC should be deployed as a security layer around a KYC workflow.
Customer
β
βΌ
Video KYC Flow
β
βΌ
βββββββββββββββββββββ
β Existing KYC β
β Provider β
βββββββββββ¬ββββββββββ
β
β
βββββββββββΌββββββββββ
β ShadowKYC β
β Integrity Layer β
βββββββββββ¬ββββββββββ
β
βΌ
Risk + Evidence
β
βββββββββββ΄ββββββββββ
βΌ βΌ
Approve Review /
Reject
ShadowKYC does not need to replace the customer's existing identity-verification provider.
For production deployment, separate application logic from persistent storage.
Recommended architecture:
βββββββββββββββββββ
β Frontend β
β React / Vite β
ββββββββββ¬βββββββββ
β
βΌ
βββββββββββββββββββ
β FastAPI Server β
β ShadowKYC β
βββββββββ¬ββββββββββ
β
βββββββββββββββΌββββββββββββββ
β β β
βΌ βΌ βΌ
MongoDB Atlas S3/Object Processing
Metadata Storage Worker
β β
β β
ββββββββ¬βββββββ
βΌ
Session Reports
Large video and image files should preferably be stored in object storage rather than directly inside a database.
Database records should primarily contain:
- Session ID
- User/tenant reference
- Analysis status
- Risk score
- Classification
- Layer results
- Evidence references
- Timestamps
- Storage object references
- Python
- FastAPI
- OpenCV
- MediaPipe
- NumPy
- SciPy
- PyZBar
- WebSockets
- MediaPipe FaceMesh
- OpenCV
- HOG-based facial representation
- FFT analysis
- Laplacian variance
- LBP-based texture analysis
- Facial geometry analysis
- Temporal signal analysis
- React
- Vite
- Tailwind CSS
- Chart.js
- Lucide icons
- MongoDB Atlas
- Amazon S3
- Cloud deployment
- Private object storage
- Presigned media URLs
MongoDB Atlas and S3 should only be considered production architecture components once they are actually integrated into the application.
Real-time video analysis can be computationally expensive.
Potential optimizations include:
- Process only selected frames per second
- Resize face crops before analysis
- Use conditional execution
- Batch independent operations
- Cache repeated calculations
- Avoid unnecessary full-resolution processing
- Use asynchronous processing where appropriate
- Use ONNX / optimized inference where applicable
- Use CPU-friendly algorithms for lightweight layers
A production system should balance:
Detection Accuracy
β
Processing Latency
β
Infrastructure Cost
A robust evaluation should test ShadowKYC against multiple attack categories.
- Normal webcam sessions
- Different lighting
- Different backgrounds
- Different camera qualities
- Different face angles
- Natural speaking and blinking
- Replay videos
- Frozen-frame attacks
- Screen recordings
- Face-swap videos
- Synthetic/deepfake videos
- Overlay attacks
- Manipulated lighting/background
- Low-quality compressed videos
Recommended evaluation metrics:
Precision
Recall
F1 Score
False Positive Rate
False Negative Rate
ROC-AUC
Detection Latency
Per-layer contribution
Session-level accuracy
ShadowKYC should be treated as a risk-analysis system, not an infallible deepfake detector.
Potential limitations include:
- Low-light environments
- Poor camera quality
- Strong video compression
- Occluded faces
- Extreme head poses
- Multiple faces
- Network-induced frame drops
- Legitimate unusual facial motion
- False positives from environmental changes
- New deepfake generation techniques
- Device-specific capture behavior
The risk score is an analytical signal and should be incorporated into an organization's broader fraud and compliance decision process.
Video KYC data can contain highly sensitive personal information.
A production implementation should consider:
- Encryption in transit
- Encryption at rest
- Private object storage
- Short media retention periods
- Access-controlled evidence
- Tenant isolation
- Audit logs
- Secure deletion policies
- Least-privilege IAM
- No unnecessary storage of raw biometric data
Do not expose uploaded KYC videos or evidence frames through public URLs.
Potential future improvements:
- Dedicated deepfake classifier
- Improved temporal transformer models
- Audio deepfake detection
- Stronger replay detection
- Device/camera fingerprinting
- Screen-reflection detection
- Advanced face embeddings
- Depth estimation
- Multi-face attack handling
- Background processing workers
- Redis-based session state
- MongoDB Atlas integration
- Amazon S3 integration
- Horizontal scaling
- GPU inference workers
- Queue-based processing
- Tenant dashboard
- Fraud analyst dashboard
- Session history
- Evidence explorer
- Risk analytics
- API keys
- Webhooks
- KYC provider integrations
- Configurable risk thresholds
- Organization-specific policies
ShadowKYC can be used as an additional integrity layer for:
- Fintech onboarding
- Banking KYC
- Insurance onboarding
- Digital lending
- Remote account opening
- Government digital services
- Telecom onboarding
- High-value transaction verification
- Remote employee verification
- Fraud investigation workflows
ShadowKYC follows four core principles:
No single visual signal is treated as sufficient evidence.
Video should be analyzed across time, not only as independent images.
Contradictory signals can provide stronger evidence than isolated anomalies.
The system should provide reasons and evidence alongside the final risk score.
| Term | Meaning |
|---|---|
| Liveness | Evidence that the presented subject behaves like a live person |
| Replay Attack | Presenting previously recorded KYC footage |
| Presentation Attack | Presenting a photo, screen, mask, or other artifact as the subject |
| Face Swap | Replacing one person's facial identity with another |
| Identity Drift | Significant change in facial representation during a session |
| Temporal Consistency | Whether facial behavior remains coherent over time |
| Texture Artifact | Abnormal visual texture caused by synthesis, manipulation, or processing |
| Geometry Drift | Abnormal change in facial landmark relationships |
| Lip-Sync | Consistency between mouth movement and speech activity |
| Meta Fusion | Combining multiple detection signals into one risk assessment |
ShadowKYC is a security and fraud-analysis prototype intended to provide risk indicators and supporting evidence.
It should not be treated as a guaranteed deepfake detector or as the sole basis for identity, compliance, or financial decisions.
Production deployments should be evaluated against representative datasets, attack scenarios, regulatory requirements, privacy requirements, and organization-specific risk policies.
This project is actively developed as a research and engineering project focused on:
Computer Vision
+
Video Forensics
+
Fraud Detection
+
Real-Time Systems
+
KYC Security
Add the project's chosen license here before public production distribution.
Make Video KYC harder to fool by validating not only who is being presented, but whether the video itself appears trustworthy.