🔍 PromptSeg: Language-Guided Segmentation for Industrial QA

Fine-tuned CLIPSeg for text-conditioned drywall defect segmentation. One model, two tasks, zero retraining — just swap the prompt. Achieves 0.628 Dice (+358% over zero-shot) at 83 FPS. Trained in 34 min on Kaggle 2× T4.

"segment crack"        →  binary crack mask
"segment taping area"  →  drywall joint / seam mask
wrong-domain prompt    →  empty mask  (88% accuracy)

Built and trained end-to-end in a single Kaggle notebook on 2× T4 GPUs. Total training time: ~34 minutes.

---

🌟 Why PromptSeg?

✅ Single Model, Multiple Tasks — Prompt-switched at inference time, no retraining needed
✅ Vision-Language Foundation — Leverages CLIP's pretrained alignment for domain adaptation
✅ Robotics-Ready — 83 FPS inference, ONNX/TensorRT compatible, ROS2 integrable
✅ Structured Negative Training — Model learns when to output nothing — critical for real-world deployment
✅ 358% Improvement over Zero-Shot — Fine-tuning is essential; this project quantifies exactly why

---

🚀 Core Capabilities

🧠 Text-Conditioned Segmentation

• Crack Detection — Segments surface fractures from natural language prompts. Handles thin hairline cracks, branching cracks, and varied surface textures (concrete, brick, drywall).
• Taping Area Detection — Identifies drywall joint/seam regions. Robust to noisy bbox-derived ground truth via domain-specific threshold (τ=0.30).
• Prompt Robustness — 8 natural language variants per domain sampled during training. Achieves 0.907 pairwise IoU across prompt variants — the model understands intent, not just keywords.
• Negative Awareness — 20% of each training batch uses cross-domain structured negatives (wrong prompt → zero mask). Teaches the model when not to fire.

⚙️ Two-Stage Training Strategy

Stage 1 — Decoder Only (5 epochs)

• Freeze CLIP encoders; train 1.1M-parameter decoder only
• Learning rate: 1×10⁻⁴
• Result: Dice 0.137 → 0.623

Stage 2 — Full Fine-Tune (5 epochs)

• Unfreeze all 150.7M parameters with differential learning rates
• Encoder LR: 1×10⁻⁶ | Decoder LR: 1×10⁻⁵
• Result: Dice 0.623 → 0.628 (stable refinement, no overfitting)

📐 Loss Function

L = 0.7 × Dice Loss + 0.3 × BCE Loss + 0.05 × Area Regularization

Component	Weight	Purpose
Dice Loss	0.70	Handles class imbalance — cracks cover < 5% of image area
BCE Loss	0.30	Stable per-pixel gradient signal
Area Regularization	0.05	Suppresses over-prediction on noisy bbox-derived labels

---

📊 Results

Model predictions on validation samples — crack (τ=0.5) and taping (τ=0.3)

Metric	Value	Target	Status
Overall Dice	0.6284	> 0.50	✅
Crack Dice	0.6319	> 0.50	✅
Taping Dice	0.6189	> 0.50	✅
Negative Accuracy	88.0%	> 90%	⚠️
Prompt Consistency (IoU)	0.907	> 0.85	✅

On Negative Accuracy: 88% is 2 points below target, but median false-positive area = 0.0% — erroneous activations are sparse and negligible in practice.

Zero-shot baseline: 0.137 Dice → Fine-tuned: 0.628 Dice (+358%, 4.6×)

Loss curves and Dice scores across 10 epochs — Stage 1 (decoder only) → Stage 2 (full fine-tune)

Full ablation, threshold sensitivity, per-prompt robustness → RESULTS.md

---

🛠️ Technology Stack

Component	Technology	Details
Base Model	CLIPSeg clipseg-rd64-refined	CIDAS / HuggingFace
Vision Encoder	CLIP ViT-B/32	Frozen in Stage 1
Text Encoder	CLIP Transformer	Frozen in Stage 1
Decoder	Transposed convolutions	1.1M trainable params
Framework	PyTorch 2.x + Transformers 4.x	HuggingFace ecosystem
Hardware	Kaggle 2× T4 GPU (16 GB each)	~34 min total training
Input Resolution	352 × 352 px	CLIP native size

Why CLIPSeg over SAM / Grounded-SAM?

Approach	Limitation
SAM	Strong zero-shot, but requires point/box prompts — not native text conditioning
Grounded-SAM	Two-stage pipeline (detection → segmentation) — higher latency, complex deployment
CLIPSeg (ours)	Single forward pass, native text conditioning, fine-tunable end-to-end ✅

---

🗂️ Dataset

Split	Cracks	Taping	Total
Train	12,884	820	13,704
Validation	537	202	739
Test	537	—	537
Total	13,958	1,022	14,980

Sources: Roboflow cracks-3ii36 (polygon annotations) + drywall-join-detect (bounding box annotations)

Ground truth generation:

• Cracks → cv2.fillPoly() on COCO polygon format → pixel-perfect binary masks
• Taping → filled bbox rectangles + Gaussian blur (5×5) + re-threshold → approximate masks

  
Left: crack samples with pixel-perfect polygon masks  |  Right: taping samples with bbox-derived masks

Training batch — mixed domains with prompt augmentation (neg=False = positive sample)

---

⚡ Performance

Metric	Value
Training Time	~34 min (Kaggle 2× T4)
Inference Time	~12 ms / image
Throughput	~83 FPS
GPU Memory (inference)	~2 GB
Model Parameters	150.7 million
Checkpoint Size	575 MB

Edge deployment: ONNX/TensorRT compatible → NVIDIA Jetson ready for embedded robotic inspection pipelines.

---

📦 Installation

Prerequisites

• Python 3.8+
• PyTorch 2.x + CUDA 12.x
• HuggingFace Transformers 4.x

git clone https://github.com/N-SriKrishna/PromptSeg.git
cd PromptSeg
pip install torch torchvision transformers pillow opencv-python numpy

best_model.pt (575 MB) — download from Kaggle Models

---

▶️ Quick Inference

from transformers import CLIPSegProcessor, CLIPSegForImageSegmentation
import torch
from PIL import Image

# Load model
processor = CLIPSegProcessor.from_pretrained("CIDAS/clipseg-rd64-refined")
model = CLIPSegForImageSegmentation.from_pretrained("CIDAS/clipseg-rd64-refined")
checkpoint = torch.load("best_model.pt", map_location="cuda")
model.load_state_dict(checkpoint["model_state_dict"])
model.eval().cuda()

# Inference
image = Image.open("your_image.jpg")
prompt = "segment crack"  # or "segment taping area"

inputs = processor(text=[prompt], images=[image], return_tensors="pt", padding=True)
inputs = {k: v.cuda() for k, v in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)

# Domain-aware threshold
threshold = 0.5 if "crack" in prompt else 0.3
mask = (torch.sigmoid(outputs.logits) > threshold).squeeze().cpu().numpy()
# mask: boolean array — True = defect pixel

Supported prompts (crack):
"segment crack" · "segment wall crack" · "find cracks" · "highlight surface cracks" · "detect fractures in wall" · "crack" · "segment the crack" · "mark damaged regions"

Supported prompts (taping):
"segment taping area" · "segment drywall seam" · "segment joint tape" · "highlight taped area" · "find joint compound" · "detect plastered joints" · "mark seam region" · "drywall joint"

---

🏗️ Repository Structure

PromptSeg/
├── prompted-segmentation.ipynb     # Full training + evaluation (Kaggle notebook)
├── best_model.pt                   # Trained checkpoint (575 MB — see Releases)
├── metrics.json                    # Complete evaluation metrics
├── predictions/                    # Binary output masks {0, 255} PNG
│   └── {image_id}__{prompt_slug}.png
├── visuals/
│   ├── training_history.png        # Loss curves + Dice progression
│   ├── predictions_grid.png        # Validation sample predictions
│   ├── cracks_train_samples.png    # Crack dataset samples
│   └── taping_train_samples.png    # Taping dataset samples
├── README.md
└── RESULTS.md                      # Full ablation, threshold sensitivity, failure analysis

---

🎯 Future Work

• SAM2 Integration — Refine noisy bbox taping annotations into precise instance masks for better supervision
• Multi-Task Expansion — Extend prompt vocabulary: "segment moisture damage", "segment nail pops" — zero architectural changes needed
• Higher Resolution — Train at 512×512 for improved hairline crack detection
• Edge Optimization — INT8 quantization + pruning for NVIDIA Jetson / embedded deployment
• Active Learning — Deploy → collect failure cases → iteratively fine-tune

---

💡 Impact

PromptSeg demonstrates that vision-language pretraining can be efficiently fine-tuned for domain-specific industrial inspection — enabling flexible, prompt-switchable defect detection without task-specific architectures. The single-model design simplifies deployment in autonomous robotic inspection pipelines where task specifications may originate from natural language planners.

---

📚 References

• CLIPSeg Paper (ArXiv) — Image Segmentation Using Text and Image Prompts
• CIDAS/clipseg-rd64-refined — HuggingFace Model Hub
• PP-LiteSeg Paper (ArXiv)
• Roboflow Universe

---

🧑‍💻 Author

Sri Krishna Nurandu
Repository: github.com/N-SriKrishna/PromptSeg
License: MIT

---

Part of an ongoing exploration into edge AI, computer vision, and robotics-inspired systems.