Setup • Pipeline • Interactive Demo • Results • Citation

---

How to Take a Memorable Picture?
Empowering Users with Actionable Feedback

Francesco Laiti, Davide Talon, Jacopo Staiano, Elisa Ricci

Updates

• [Jun 17, 2026] Demos are live!
• [Mar 02, 2026] MemCoach is live!
• [Feb 28, 2026] MemBench is live!
• [Feb 26, 2026] Arxiv paper is live!

Setup

1. Install dependencies

# clone project
git clone https://github.com/laitifranz/MemCoach
cd MemCoach

# (recommended) use uv to set up the python version
# and to install the required dependencies
curl -LsSf https://astral.sh/uv/install.sh | sh
# --frozen ensures uv.lock is respected strictly and not modified,
# guaranteeing you get the exact same environment as intended
uv sync --frozen

# (alternative) use pip to install the required dependencies
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e .

Note

For those who are not familiar with uv, uv run is a command that activate at runtime the virtual environment while running the command.

CUDA 13 / newer GPUs

The default lockfile targets the standard PyTorch wheels for maximum compatibility. If your machine requires CUDA 13 (e.g. newer NVIDIA driver/GPU stacks), you can install the cu130 builds of PyTorch and a prebuilt flash-attn wheel instead, without modifying the project files:

uv pip install -U torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu130
uv pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.0/flash_attn-2.8.3+cu130torch2.9-cp312-cp312-linux_x86_64.whl

Check out this awesome page mjunya.com/flash-attention-prebuild-wheels for pre-built flash attn wheels.

Note that after running these commands, uv sync or uv run would restore the locked versions; activate instead the virtual environment and run python normally to use the new versions, or use bash scripts/schedule_python.sh.

2. (Optional) Setup environment variables

You can skip this step if you are not using OpenRouter API or if you are using the default paths.

# copy .env.example to .env
cp .env.example .env

# edit .env file
vim .env

3. Download the dataset MemBench

MemBench is a benchmark dataset, hosted on HuggingFace, introduced alongside MemCoach.

uv run hf download --repo-type dataset laitifranz/MemBench --local-dir dataset/
unzip dataset/images.zip -d dataset/ && mv dataset/images dataset/ppr10k # rename the folder to ppr10k

Pipeline

General information

Important

Launch the scripts from the project root directory.

Tip

The repository keeps a portable Python wrapper to activate the virtual environment and set the PYTHONPATH to the project root:

bash scripts/schedule_python.sh <PATH_TO_PYTHON_SCRIPT> <ARGUMENTS_FOR_THE_SCRIPT>

You can also use the following command with uv to run the script:

PYTHONPATH=$(pwd) uv run <PATH_TO_PYTHON_SCRIPT> <ARGUMENTS_FOR_THE_SCRIPT>

Tip

The scripts directory contains various utility scripts for job scheduling on SLURM clusters. Use bash scripts/schedule_sbatch.sh -h to get more information. Take a look at the examples in the scripts/slurm_configs directory to see how to setup a SLURM environment.

---

Preliminary Stage - Memorability scores generation

Generate the memorability scores with our target predictor model.

bash scripts/schedule_python.sh src/pipelines/membench_gen/generate_target_scores.py --mlp_checkpoint_path "ckpt/target_predictor/memorability/ours/model_weights.pth"

---

Stage A - Contrastive generation

Tip

If you run the scripts on a SLURM environment cluster, you can parallelize the generation by using SLURM array argument.

Note

The model settings provided in the config files are for the InternVL3.5-8B model. You can change the model settings by editing the config files.

A1. Teacher generation (MemBench-style generation)

bash scripts/schedule_python.sh src/pipelines/membench_gen/constr_data_gen/runner.py --config_path config/data_generation/teacher/internvl3_5_8B.yaml

A2. Student generation (Zero-shot generation)

bash scripts/schedule_python.sh src/pipelines/zero_shot/runner.py --config_path config/data_generation/student/internvl3_5_8B.yaml

---

Stage B - Steering vector extraction

B1. Positive activation extraction

bash scripts/schedule_python.sh src/pipelines/method/training.py --config_path config/method_steering/training/internvl3_5_8B_positive.yaml

B2. Negative activation extraction

bash scripts/schedule_python.sh src/pipelines/method/training.py --config_path config/method_steering/training/internvl3_5_8B_negative.yaml

Note

We provide the pre-built steering vectors for the InternVL3.5-8B model used for the paper experiments in the ckpt/memcoach directory. You can use them to skip the activation extraction stage.

---

Stage C - MemCoach inference

Run inference using the exact activation files produced in Stage B:

bash scripts/schedule_python.sh src/pipelines/method/inference.py # add --config-name config_paper to use our steering pre-built vectors

Tip

By default, Hydra will use the config/method_steering/inference/config.yaml file pointing to config/method_steering/inference/internvl3_5_8B.yaml. You can override the key dicts in the config file by passing them as arguments to the script, e.g.:

bash scripts/schedule_python.sh src/pipelines/method/inference.py activation_settings.coeff=55 activation_settings.target_layer=12 runtime.include_datetime=true

---

Run the editing evaluation pipeline

Based on the editing evaluation you want to perform, you can choose the appropriate config file. We provide config files for the flux baseline, teacher oracle, zero-shot, and MemCoach editing evaluation.

• For the MemCoach evaluation:

bash scripts/schedule_python.sh src/pipelines/evaluation/editing/runner.py --config_path config/evaluation/editing/memcoach.yaml

---

Run the analysis pipeline

bash scripts/schedule_python.sh src/analysis/editing_metrics.py --root experiments/evaluation/editing --run-scope latest

Interactive Demo

We provide two interactive demos to test out MemCoach in real time:

Demo	What it does	Entry point
Demo 1 — Live Camera Coach	Capture a photo from your mobile device and receive a memorability score plus actionable feedback in real time	src/api/camera_app.py
Demo 2 — Before/After Studio	Upload a photo; MemCoach scores it, generates feedback, automatically applies the edit with FLUX.2-klein, re-scores the result, and ranks your improvement on a live leaderboard	src/api/studio_app.py

Demo 1 — Live Camera Coach

Capture images directly from your mobile device through a FastAPI backend exposed via ngrok. You take a picture from your phone and MemCoach returns a memorability score and actionable feedback to improve your shot.

Note

• Since the camera access on mobile devices requires a secure context, we need to use a proxy to forward the requests to the FastAPI server via HTTPS. We use ngrok for this purpose
• API requests are logged in the outputs/api_requests directory. Check web/camera/README.md for more information
• The default steering settings loaded in the MemCoach API are configured in the config/method_steering/inference/internvl3_5_8B_paper.yaml file.

1. Open two terminals:

1.1 Get an ngrok authtoken and install ngrok:

NGROK_AUTHTOKEN=<your_token_here> uvx ngrok http 8000

Save the ngrok forward URL for later use.

1.2 Start the FastAPI server:

bash scripts/schedule_python.sh -m uvicorn src.api.camera_app:app --host 0.0.0.0 --port 8000

2. Connect to the ngrok tunnel from your mobile device via browser:

https://<your_ngrok_subdomain>.ngrok-free.dev/camera/?api=https://<your_ngrok_subdomain>.ngrok-free.dev

3. Enjoy! 🎉

Demo 2 — Before/After Studio

Upload or capture a photo and let MemCoach do the full makeover: the server scores the memorability of your image, generates actionable feedback, applies the suggested edit automatically with FLUX.2-klein, and re-scores the edited result. You get the before/after images side by side, both memorability scores, and a live leaderboard ranking the best improvements across all participants.

1. Start the FastAPI server:

bash scripts/schedule_python.sh -m uvicorn src.api.studio_app:app --host 0.0.0.0 --port 8020

2. Open the web UI in a browser:

https://<your_ngrok_subdomain>.ngrok-free.dev/studio/

3. Enjoy! 🎉

Note

• Demo settings (FLUX model, memorability checkpoint, steering config) are configured in the config/api/studio_server.yaml file
• All three models (memorability predictor, feedback VLM, FLUX.2-klein) are loaded at startup, so the first launch may take a while (model weights are downloaded on first use) and requires a GPU with significant memory
• Each request is logged in the outputs/studio_requests directory (source image, edited image, and metadata), and leaderboard entries are stored in outputs/studio_leaderboard.db

Reproducing Paper Results

For transparency and reproducibility, we provide our evaluation artifacts for MemCoach on InternVL3.5-8B model. We report the IR and RM metrics.

Option A — Compact download (recommended): A single zip archive containing all 4 experiment folders is available to avoid hitting the Hugging Face rate limit:

uv run hf download --repo-type dataset laitifranz/MemBench-InternVL3.5-Eval MemBench-InternVL3.5-Eval-Artifacts.zip --local-dir artifacts/
unzip artifacts/MemBench-InternVL3.5-Eval-Artifacts.zip -d artifacts/hf_internvl3_5_8B_eval

Option B — Full dataset download (for individual downloads):

HF_XET_HIGH_PERFORMANCE=1 uv run hf download --repo-type dataset laitifranz/MemBench-InternVL3.5-Eval --include "teacher_oracle/*" --local-dir artifacts/hf_internvl3_5_8B_eval

Note

You may need to resume the download if you hit the Hugging Face rate limit.

Then run the analysis pipeline:

bash scripts/schedule_python.sh src/analysis/editing_metrics.py --root artifacts/hf_internvl3_5_8B_eval --run-scope latest

Maintenance

make setup-pre-commit # one-time setup: install pre-commit globally with uv + install git hooks + autoupdate
make check # run pre-commit hooks (format + lint via ruff + security checks via gitleaks)
make clean-logs # clean logs
make run-tests # run tests via pytest

Citation

If you find this work useful to your research, please consider citing as:

@inproceedings{laiti2026memcoach,
  title={How to Take a Memorable Picture? Empowering Users with Actionable Feedback},
  author={Laiti, Francesco and Talon, Davide and Staiano, Jacopo and Ricci, Elisa},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2026}
}

Acknowledgments

Thanks to these great repositories: dottxt-ai/outlines to make structured output easy with LLMs, google/python-fire to make CLI management easy, PPR10K for the PPR10M dataset, and web-haptics for the experimental haptic feedback solution in the web demo.

The needs of this project led to a contribution back to the community: PR #1728 was merged into dottxt-ai/outlines, improving the handling of multimodal chat inputs for Transformers models.

Disclaimers

Since this project relies a lot on automatic generation of data, the generated feedback can be different due to the stochastic nature of models. We tried to make the pipeline as reproducible as possible, but there might be some variations in the generated feedback using your machine and virtual env setup. See Reproducing Paper Results for more information.