$\text{Differentiable Hierarchical Visual Tokenization}$

Marius Aasan$^1$, Martine Hjelkrem-Tan$^1$, Nico Catalano$^2$, Chankyu Choi$^3$, Adín Ramírez Rivera$^1$

${}^1\underset{\text{Department of Informatics}}{\text{University of Oslo}}$ $\hspace{1em}$ ${}^2\underset{\text{Artificial Intelligence and Robotics Lab}}{\text{Polytechnic University of Milan}}$ $\hspace{1em}$ ${}^3\underset{\text{Department of Physics and Technology}}{\text{UiT The Arctic University of Norway}}$

$\text{Abstract}$

Vision Transformers rely on fixed patch tokens that ignore the spatial and semantic structure of images. In this work, we introduce an end-to-end differentiable tokenizer that adapts to image content with pixel-level granularity while remaining backward-compatible with existing architectures for retrofitting pretrained models. Our method uses hierarchical model selection with information criteria to provide competitive performance in both image-level classification and dense-prediction tasks, and even supports out-of-the-box raster-to-vector conversion.

$\partial\text{HT}$: Differentiable Hierarchical Visual Tokenization

This repo contains code for Differentiable Hierarchical Visual Tokenization, accepted as a spotlight paper for NeurIPS 2025.

For an introduction to our work, visit the project webpage.

Installation

The repo can currently be installed as a package via:

# HTTPS
pip install git+https://github.com/dsb-ifi/dHT.git

# SSH
pip install git+ssh://git@github.com/dsb-ifi/dHT.git

Loading models

You can load the Superpixel Transformer models easily via torch.hub:

# Example with raster-to-vector model
model = torch.hub.load(
    'dsb-ifi/dht', 
    'dht_ras2vec',
    pretrained=True,
    source='github',
)

This will load the model and downloaded the pretrained weights, stored in your local torch.hub directory.

Citation

If you find our work useful, please consider citing our paper.

@inproceedings{aasan2025dht,
  title={Differentiable Hierarchical Visual Tokenization},
  author={Aasan, Marius and Hjelkrem-Tan, Martine and Catalano, Nico and Choi, Changkyu and Ram\'irez Rivera, Ad\'in},
  booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
  year={2025},
  url={https://openreview.net/forum?id=y8VWYf5cVI}
}

---

🚧 NOTE: The hubconf.py is still under construction, and will be updated with classification models soon.