The perceive model of the Eximius stack — the counterpart to Fusion Embedding (the search model). Fusion Embedding answers "find the moment where X happened" (unified multimodal memory); Fusion Perception answers "what is in this frame, and where am I" — dense scene understanding and geometric place recognition. The two run side by side.
It is a frozen DINO backbone plus light task heads (the backbone is never fine-tuned). This open release uses DINOv2 ViT-B/14 (Apache-2.0); the hosted version uses DINOv3.
This repository is the code. Model weights (the trained heads + projector) live on the Hugging Face model repo: EximiusLabs/fusion-perception-1-preview. A separate repo from
fusion-embedding, by design — each family is its own codebase.
| Capability | Metric | Fusion Perception | Baseline |
|---|---|---|---|
| Place recognition (R-Oxford medium, zero-training) | mAP | 0.804 | 0.596 (semantic base) |
| Segmentation — general (ADE20K) | mIoU | 0.392 | — |
| Segmentation — driving (Cityscapes) | mIoU | 0.447 | — |
| Shared-space embedding (projected into Fusion Embedding 2) | R-Oxford mAP / text→img R@10 | 0.764 / 0.749 | 0.596 base image path |
The shared-space projector is the differentiated piece: it maps the geometric DINO descriptor into the frozen Fusion Embedding 2 space so it is language-searchable and cross-modal, while keeping a +17-point place-recognition edge over the base model's own image path.
pip install -r requirements.txtfrom PIL import Image
from fusion_perception import FusionPerception
fp = FusionPerception.from_pretrained("EximiusLabs/fusion-perception-1-preview")
fp.embed_place(Image.open("frame.jpg")) # [768] place / re-localization descriptor
fp.embed_shared(Image.open("frame.jpg")) # [2048] language-searchable, in the Fusion Embedding space
fp.segment(Image.open("frame.jpg"), domain="driving") # per-pixel class idsSee examples/quickstart.py.
The train/ scripts reproduce the shipped heads (they run on Modal; the
backbone stays frozen and only the small heads / projector train):
train/seg_heads.py— linear segmentation heads on frozen DINOv2 dense features (ADE20K, Cityscapes).train/place_recognition.py— the frozen-descriptor place-rec recipe + R-Oxford eval (and the PCA-whitening experiment, deferred to v0.2).train/dino_fe2_projector.py— the projector into the Fusion Embedding 2 shared space (image-caption InfoNCE on Flickr30k), with the R-Oxford / text→image evaluation.
Code: Apache-2.0 (see LICENSE). Model weights (on Hugging Face): CC-BY-NC-4.0 — the
DINOv2 backbone is Apache-2.0 and downloads separately; the driving segmentation head is trained on
Cityscapes (non-commercial), so the weights are NC overall, with the NC component kept separable.
- Fusion Embedding — the search/memory model (text, image, video, audio) and its sensor packs (Ember: thermal; Tremor: motion).
- Fusion Perception is a separate family; its projector bridges into the Fusion Embedding space without changing any existing vector.