Demo2Tutorial-Unofficial

Unofficial PyTorch Reproduction of

Demo2Tutorial: From Human Experience to Multimodal Software Tutorials

[CVPR 2026 / arXiv 2026]

Paper · Project / Reference · Issues

This is an unofficial implementation maintained by @StaryMoon. If this repository helps your reading, reproduction, or course project, please consider giving it a star and following my GitHub profile.

News

• 2026-06-10: Initial public release with model interfaces, configuration, smoke test, and reproduction roadmap.

Overview

This repository organizes a PyTorch implementation for Demo2Tutorial: From Human Experience to Multimodal Software Tutorials, focusing on multimodal software tutorial generation from screen recordings and action traces. The codebase is structured like a standard research repository so that each paper component can be replaced, tested, and extended independently.

Main goals:

• provide a clean PyTorch module layout for the paper;
• keep training, inference, evaluation, and configuration entry points explicit;
• track paper-reported metrics separately from local reproduction logs;
• make it easy for contributors to fill in missing paper-specific details.

Repository Structure

Demo2Tutorial-Unofficial/
├── configs/
│   └── default.yaml
├── scripts/
│   └── smoke_test.py
├── src/demo2tutorial_unofficial/
│   ├── __init__.py
│   └── model.py
├── README.md
├── requirements.txt
└── pyproject.toml

Installation

git clone https://github.com/StaryMoon/Demo2Tutorial-Unofficial.git
cd Demo2Tutorial-Unofficial
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For CUDA-enabled experiments, install the PyTorch build matching your CUDA version from the official PyTorch website before installing the rest of the dependencies.

Quick Check

Run the minimal forward-pass check:

python scripts/smoke_test.py

Expected output:

output: (...)
loss: ...

This confirms that the package import path, model interface, and tensor flow are working.

Data Preparation

Create a local data directory:

mkdir -p data checkpoints outputs

Recommended layout:

data/
├── train/
├── val/
└── test/

Dataset-specific converters will be added under scripts/ as the reproduction becomes more complete. Please do not commit private datasets, downloaded checkpoints, or generated outputs.

Training

Current minimal module usage:

import torch
from demo2tutorial_unofficial import StarterConfig, UnofficialModel, reconstruction_loss

config = StarterConfig(hidden_dim=128, num_layers=2, num_heads=4)
model = UnofficialModel(config)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)

x = torch.randn(2, 3, 64, 64)
token_ids = torch.randint(0, config.vocab_size, (2, 8))
target = torch.zeros(2, config.output_dim)

pred = model(x, token_ids=token_ids)
loss = reconstruction_loss(pred, target)
loss.backward()
optimizer.step()

Full training scripts will be added as paper-specific datasets and loss terms are implemented.

Inference

import torch
from demo2tutorial_unofficial import UnofficialModel

model = UnofficialModel().eval()
with torch.no_grad():
    x = torch.randn(1, 3, 64, 64)
    y = model(x)
print(y.shape)

Evaluation

Planned evaluation entry points:

python scripts/smoke_test.py
# future:
# python scripts/evaluate.py --config configs/default.yaml --ckpt checkpoints/model.pt

Metrics and protocols will follow the original paper as closely as possible. Paper-reported values and local reproduction values should be kept in separate columns.

Paper Results

For copyright and license clarity, this repository links to the original paper figures and tables instead of redistributing screenshots copied from the PDF. The table below tracks the paper-reported result locations so readers can quickly compare against future local logs.

Result Type	Paper Location	Source
Main quantitative comparison	Main paper tables	Original paper / project page
Ablation study	Ablation section	Original paper / project page
Qualitative examples	Main paper figures and supplement	Original paper / project page

Reproduction Log

Date	Config	Dataset Split	Metric	Value	Notes
2026-06-10	configs/default.yaml	smoke check	forward pass	ok	package interface validation

Implementation Status

• Package layout and install metadata
• Core PyTorch module interfaces
• Config file and smoke test
• Dataset-specific preprocessing
• Paper-specific losses and heads
• Training script
• Evaluation script
• Model zoo / checkpoints
• Reproduction logs

Model Zoo

Model	Checkpoint	Config	Notes
default	TBA	configs/default.yaml	compact implementation interface

Citation

If you find this repository useful, please cite the original paper:

@article{demo2tutorialunofficial,
  title = {Demo2Tutorial: From Human Experience to Multimodal Software Tutorials},
  author = {Zechen Bai, Zhiheng Chen, Yiqi Lin, Kevin Qinghong Lin, Difei Gao, Xiangwu Guo, Xin Wang, Mike Zheng Shou},
  year = {2026},
  note = {CVPR 2026 / arXiv 2026}
}

Acknowledgements

• Thanks to the authors of Demo2Tutorial: From Human Experience to Multimodal Software Tutorials for the original research.
• This repository is inspired by standard open-source PyTorch research codebases.
• The implementation is unofficial and all paper names, datasets, and trademarks belong to their respective owners.

License

This repository is released under the MIT License. The original paper, datasets, official code, and project assets remain governed by their own licenses.

Keywords

gui-agent, multimodal-tutorial, screen-recording, software-agent, pytorch, cvpr-2026, unofficial-pytorch, reproduction