Facial image preprocessing begins at the frontend capture layer and continues through standardization before feature extraction. The complete preprocessing pipeline ensures consistency, robustness, and optimal input for the deep neural network.
The facial image acquisition occurs via the FaceCapture React component, which interfaces with the device's webcam through the WebRTC API.
Capture Parameters:
- Video Source: Device front-facing camera (facingMode: 'user')
- Requested Resolution: Ideal 640×480 pixels (browsers negotiate actual resolution based on device capabilities)
- Capture Method: Canvas-based snapshot from video stream
- Output Format: PNG (lossless compression)
- Mirror Transformation: Horizontal flip applied to match expected camera perspective (users expect to see themselves as in a mirror)
Capture Process:
// Frontend capture pipeline
1. Request camera: navigator.mediaDevices.getUserMedia({
video: { width: { ideal: 640 }, height: { ideal: 480 }, facingMode: 'user' },
audio: false
})
2. Render live video stream to hidden canvas element
3. On user action (snapshot button):
- Mirror canvas context: ctx.scale(-1, 1); ctx.translate(-w, 0)
- Draw video frame to canvas
- Convert canvas to PNG blob: canvas.toBlob(..., 'image/png')
4. Transmit PNG file to backend via multipart/form-data HTTP POSTInput Specification for Backend:
- Format: PNG or JPEG
- Expected dimensions: Minimum 320×240, typical 640×480 or larger
- Color space: RGB (3 channels)
- No normalization applied at this stage (performed during model preprocessing)
Upon receiving the image file at the /enroll or /verify endpoint, the backend performs initial validation and format conversion.
Image Processing Steps:
# backend/main.py - Image handling
face_img = Image.open(io.BytesIO(await face.read())).convert('RGB')- Format Validation: Accepts JPEG, PNG, BMP, GIF (via PIL.Image)
- Color Space Conversion:
.convert('RGB')ensures:- Grayscale images (L mode) → RGB (replicates single channel across 3 channels)
- RGBA images → RGB (discards alpha channel)
- Other formats → RGB (consistent 3-channel representation)
- No Resizing: Image is passed to MTCNN at native resolution; face detection handles scale-invariance
MTCNN (Multi-Task Cascaded Convolutional Networks) performs face detection, alignment, and region extraction. This stage is critical as it standardizes facial regions regardless of input image size or camera angle.
MTCNN Configuration:
# src/models/face_model.py
mtcnn = MTCNN(image_size=160, margin=20, device=device)MTCNN Processing Parameters:
| Parameter | Value | Purpose |
|---|---|---|
| image_size | 160 | Output resolution: 160×160 pixels |
| margin | 20 | Pixel padding around detected face |
| min_face_size | 20 (default) | Minimum detectable face size in pixels |
| device | 'cpu' or 'cuda' | Computational device |
MTCNN Processing Pipeline:
-
Stage 1 - Proposal Generation:
- Image processed at multiple scales (pyramid)
- Coarse 12×12 CNN generates region proposals
- Output: bounding boxes of potential face regions
-
Stage 2 - Refinement:
- Refine proposals with 24×24 CNN
- Removes false positives
- Improves bounding box accuracy
- Output: refined bounding boxes
-
Stage 3 - Output:
- Final verification with 48×48 CNN
- Predicts 5 facial landmarks: (left_eye, right_eye, nose, left_mouth, right_mouth)
- Performs geometric alignment to normalize face pose
Face Extraction with Margin:
- Detected bounding box: (x_min, y_min, x_max, y_max)
- Expanded region with margin=20: (x_min-20, y_min-20, x_max+20, y_max+20)
- Prevents overly tight crops that exclude important contextual features
- Handles slight pose variations and alignment errors
Output Specification:
- Format: PyTorch tensor
- Dimensions: 1×3×160×160 (batch_size=1, channels=3, height=160, width=160)
- Pixel range: [0, 255] (still integer values, not normalized)
- Returns
Noneif no face detected with sufficient confidence
Error Handling:
face = self.mtcnn(img)
if face is None:
return None # Face detection failedAfter MTCNN produces the 160×160 face region, the InceptionResnetV1 feature extractor applies implicit internal normalization.
InceptionResnetV1 Preprocessing (Internal to Model): The PyTorch implementation of InceptionResnetV1 expects input in the following format:
- Input Range: [0, 255] integers OR [0, 1] floats (handled by model)
- Internal Normalization:
- The model was trained with standard ImageNet-style normalization
- Per-channel normalization applied internally using learned parameters
- No explicit preprocessing required at inference time
Forward Pass:
with torch.no_grad():
emb = self.face_model(face) # face: 1×3×160×160
# Output: 1×512 tensor (batch_size=1, embedding_dim=512)Following feature extraction, the 512-dimensional embedding is L2-normalized to unit length.
L2 Normalization Formula:
emb_normalized = emb / ||emb||₂
where ||emb||₂ = sqrt(sum(emb_i²))
Purpose of L2 Normalization:
- Metric Space Property: Constrains embeddings to unit hypersphere
- Cosine Similarity Equivalence: L2-normalized dot product equals cosine similarity
- Numerical Stability: Bounded range prevents numerical overflow in subsequent operations
- Interpretability: Enables direct comparison of angles between embeddings
Implementation:
emb = emb / emb.norm() # PyTorch norm() defaults to L2 norm
# Result: 1×512 tensor with ||emb||₂ = 1.0Reference Embedding Storage:
- Stored in SQLite database as BLOB (binary serialized PyTorch tensor)
- Remains L2-normalized throughout storage and all comparisons
- Format: Float32 precision (4 bytes per scalar × 512 = 2048 bytes per embedding)
Signature preprocessing is more complex than facial recognition due to variable input sources (canvas drawing vs. hand-tracked gesture) and the need for strict spatial standardization.
Signatures are captured through one of two methods:
Method A: Canvas-Based Drawing (SignatureInput.jsx)
- User draws directly on HTML5 canvas element
- Free-form pen input via mouse or touch
- Strokes rendered as vector paths, then rasterized
Method B: Air Signature with Hand Tracking (AirSignatureCapture.jsx)
- Uses MediaPipe Hands for hand landmark detection
- Detects pinch gesture (thumb + index finger) to initiate/pause drawing
- Hand position tracked in 3D space, projected to 2D canvas
- Strokes rendered as the user "writes" in air
Common Capture Parameters:
| Parameter | Value | Purpose |
|---|---|---|
| Canvas Width | 480 | Pixel width of capture canvas |
| Canvas Height | 360 | Pixel height of capture canvas |
| Output Format | PNG | Lossless compression |
| Background | White (for canvas) | Signature on white background |
| Stroke Color | Dark blue/indigo | High contrast for preprocessing |
| Stroke Width | 3-4 pixels | Represents natural pen width |
Signature Capture Pipeline:
// Frontend: Convert canvas to PNG blob
canvas.toBlob((blob) => {
const file = new File([blob], 'signature.png', { type: 'image/png' });
// Transmit file via multipart/form-data HTTP POST
}, 'image/png');Upon receiving the signature image file, the backend converts it to grayscale for processing.
Image Processing Steps:
# backend/main.py - Signature handling
sig_img = Image.open(io.BytesIO(await signature.read())).convert('L')- Format Validation: Accepts PNG, JPEG, BMP (via PIL.Image)
- Grayscale Conversion:
.convert('L')- RGB → Single-channel luminance (L mode)
- Conversion formula: L = 0.299×R + 0.587×G + 0.114×B (standard ITU-R BT.601)
- Reduces dimensionality from 3 channels to 1 channel
- Signature content is intensity-based, not color-dependent
Rationale for Grayscale:
- Signatures are single-color (pen on paper); color information is redundant
- Reduces model input size and computational cost
- Focuses network capacity on shape and motion patterns, not color variations
- Improves generalization across writing instruments (ballpoint, marker, etc.)
The grayscale signature image is resized to a standardized 150×220 pixel format, the input expected by the Siamese network.
Resize Parameters:
# src/models/signature_model.py
INPUT_SIZE = (150, 220) # (height, width) in PIL convention
transform = transforms.Resize(INPUT_SIZE)Resize Semantics:
- Input: Arbitrary dimensions (e.g., 480×360 from canvas, or native file resolution)
- Output: Exactly 150×220 pixels
- Algorithm: Bilinear interpolation (PIL/torchvision default)
- For upsampling: interpolates pixel values from neighbors
- For downsampling: averages neighboring pixels
- Aspect Ratio: Not forced; resizing may distort aspect ratio to fill 150×220
Why 150×220?
- Chosen based on typical signature bounding box dimensions
- Width > Height (signatures typically wider than tall)
- Provides sufficient spatial resolution for Siamese network
- Balances model complexity (FC layer input: 64×18×27 after pooling) with discriminative capacity
Computational Impact:
- After 3 MaxPool2d(2) stages: 150/8 = 18.75 ≈ 18, 220/8 = 27.5 ≈ 27
- FC layer input: 64 channels × 18 pixels × 27 pixels = 30,528 features
- FC1 transforms to 256 dimensions, then FC2 to 128-dimensional embedding
After resizing, the image is converted to a PyTorch tensor and normalized to zero-mean, unit-variance format.
Tensor Conversion:
transforms.ToTensor()- Converts PIL Image (integer pixel values [0, 255]) to PyTorch tensor
- Automatically divides by 255: tensor_value = image_pixel / 255.0
- Output range: [0, 1.0] (floating point)
- Dimensions: 1×150×220 (1 channel for grayscale)
Normalization - Min-Max Scaling:
transforms.Normalize((0.5,), (0.5,))- Applied per channel to zero-mean, unit-variance format
- Formula: normalized = (tensor - mean) / std
- For single channel: mean=0.5, std=0.5
- Effective transformation:
- Input range [0, 1.0] → intermediate [0, 1.0]
- Subtract 0.5: [-0.5, 0.5]
- Divide by 0.5: [-1.0, 1.0]
- Output range: [-1.0, 1.0] (zero-centered, symmetric)
Why This Normalization?
- Matches training data distribution (Siamese network trained with [-1, 1] inputs)
- Zero-centering improves gradient flow during backprop (if finetuning)
- Symmetric range facilitates network learning
- Standard practice in deep learning (similar to ImageNet normalization)
The preprocessed tensor is wrapped in a batch dimension for inference.
Batch Creation:
tensor = transforms.ToTensor()(...) # Output: 1×150×220
tensor = tensor.unsqueeze(0) # Output: 1×1×150×220 (batch_size=1)
tensor = tensor.to(device) # Move to CPU or GPU as neededBatch Dimension Purpose:
- PyTorch models expect batch inputs (even for single samples)
- Dimension 0 represents batch size (1 for inference)
- Enables future batching if processing multiple signatures simultaneously
Reference Signature Storage:
- Reference signature preprocessed identically to test signatures
- Stored as normalized 150×220×1 tensor in database
- Ensures consistent preprocessing applied to both comparison inputs
| Stage | Face | Signature |
|---|---|---|
| 1. Capture | Webcam RGB (640×480) | Canvas/Hand-tracked (480×360) |
| 2. Format | PNG file | PNG file |
| 3. Load | .convert('RGB') |
.convert('L') |
| 4. Detect/Extract | MTCNN face detection → 160×160 | - (skip for signature) |
| 5. Resize | MTCNN handles it (to 160×160) | Resize((150, 220)) |
| 6. Tensor | ToTensor() [0, 1] |
ToTensor() [0, 1] |
| 7. Normalize | Internal to model | Normalize((0.5), (0.5)) → [-1, 1] |
| 8. Batch | unsqueeze(0) → 1×3×160×160 |
unsqueeze(0) → 1×1×150×220 |
| 9. Device | .to(device) |
.to(device) |
| 10. Model | InceptionResnetV1 | Siamese Network |
| 11. Output | 512-dim embedding | 128-dim embedding |
| 12. Normalize Output | L2 norm → unit length | Distance metric → similarity |
# src/models/face_model.py
from facenet_pytorch import MTCNN, InceptionResnetV1
from PIL import Image
import torch
class FaceRecognitionModel:
def __init__(self, device='cpu'):
self.mtcnn = MTCNN(image_size=160, margin=20, device=device)
self.face_model = InceptionResnetV1(pretrained='vggface2').eval().to(device)
def get_embedding(self, image):
"""Complete preprocessing and feature extraction pipeline"""
# Step 1: Load image
if isinstance(image, str):
img = Image.open(image)
else:
img = image
# Step 2: Face detection and alignment (MTCNN)
face = self.mtcnn(img)
if face is None:
return None
# Step 3: Add batch dimension
face = face.unsqueeze(0).to(self.device)
# Step 4: Feature extraction (InceptionResnetV1 with internal normalization)
with torch.no_grad():
emb = self.face_model(face)
# Step 5: L2 normalization
return emb / emb.norm()# src/models/signature_model.py
from PIL import Image
import torchvision.transforms as transforms
import torch
class SignatureModel:
INPUT_SIZE = (150, 220)
def __init__(self, checkpoint_path, device='cpu'):
self.device = device
self.model = SiameseNetwork().to(device)
self.model.load_state_dict(torch.load(checkpoint_path, map_location=device))
self.model.eval()
# Complete preprocessing pipeline
self.transform = transforms.Compose([
transforms.Grayscale(), # RGB → L
transforms.Resize(self.INPUT_SIZE), # Any size → 150×220
transforms.ToTensor(), # PIL Image → tensor [0, 1]
transforms.Normalize((0.5,), (0.5,)), # [0, 1] → [-1, 1]
])
def _load_and_transform(self, image):
"""Complete preprocessing pipeline"""
if isinstance(image, str):
img = Image.open(image).convert('L')
else:
img = image.convert('L') if image.mode != 'L' else image
# Apply transforms: Grayscale → Resize → ToTensor → Normalize
tensor = self.transform(img)
# Add batch dimension
return tensor.unsqueeze(0).to(self.device)// Frontend: Face capture preprocessing
const capturePhoto = () => {
const video = videoRef.current;
const canvas = canvasRef.current;
const w = video.videoWidth || 640;
const h = video.videoHeight || 480;
canvas.width = w;
canvas.height = h;
const ctx = canvas.getContext('2d');
ctx.save();
ctx.scale(-1, 1); // Mirror horizontally
ctx.translate(-w, 0);
ctx.drawImage(video, 0, 0, w, h);
ctx.restore();
canvas.toBlob((blob) => {
const file = new File([blob], 'face_capture.png', { type: 'image/png' });
onChange(file);
}, 'image/png');
};
// Frontend: Signature capture preprocessing
canvas.toBlob((blob) => {
const file = new File([blob], 'signature.png', { type: 'image/png' });
onChange(file);
}, 'image/png');After L2-normalized embedding comparison via cosine similarity (range: [-1, 1]), the score is normalized to [0, 1]:
similarity = torch.cosine_similarity(emb1, emb2).item() # [-1, 1]
face_score = (similarity + 1) / 2 # [0, 1]After Euclidean distance comparison, the score is transformed via exponential function:
distance = F.pairwise_distance(emb1, emb2).item() # [0, ∞)
sig_score = math.exp(-distance) # (0, 1]Both scores are then fused: final_score = 0.6 * face_score + 0.4 * sig_score for the final decision.