TM MCP Pack Visual

A visual inspector for Trackmania: point the game camera at a world position from a set of offsets, take clean screenshots, and get back enough geometry that a host script can crop and annotate them. Registers as a tm-control-mcp tool pack, so every tool shows up over the MCP socket as Visual.<Name>.

Built so an agent can look at things in the game without clicking around: "show me this block from the front, the left and above" is one call.

License

Dual-licensed: UNLICENSE (public domain) or CC0 1.0 — choose either.

Depends

• tm-control-mcp (hard). Every engine poke — camera writes, screenshots, world→screen projection, the Openplanet overlay — is delegated to a tm-control-mcp tool through TmMcp::CallTool. The pack itself is frame maths, orchestration and JSON, so a game update that moves a camera offset is fixed in one place, not two.

Build

./build.sh dev          # stage + openplanet-lsp the staged bytes + hot-load
just build              # the same

Quick smoke

CALL=../tm-control-mcp/tools/call.py
python3 $CALL ListToolPacks          # Visual should be listed
python3 $CALL Visual.Ping
python3 $CALL Visual.GetContext --pretty
just test                            # 119 live conformance checks

The idea in one call

python3 ../tm-control-mcp/tools/call.py --timeout 120 Visual.ShotSeries '{
  "subject": {"kind": "car"},
  "relativeTo": "subject",
  "targetDistance": 9,
  "presets": ["front", "left", "top", "iso"],
  "lookAtOffset": {"x": 0, "y": 1, "z": 0}
}'

Four clean screenshots of the car from its own front / left / above / three- quarter, each with the camera pose, the fov, and the pixel the target landed on; the camera is put back where it was afterwards.

Tools

Tool	What it does
Visual.Ping	Liveness + pack version.
Visual.ListCoverage	The pack's tools with one-line summaries, and the preset names.
Visual.GetContext	Which capture context is live (editor orbital camera / playground free cam / none), whether capture is possible and why not, screen size, the current camera pose, HUD + Openplanet-overlay state, saved camera slots, whether a series is running.
Visual.ShotSeries	The main one. N shots around a target from explicit offsets, named presets and/or an orbit; per-shot pose + projected pixels; restores the camera on every exit path.
Visual.Shot	One shot from a single offset; ShotSeries with a flatter answer.
Visual.LookAt	Aim the camera and stop there. No capture, no restore — for host-side window grabs, or for a human to see what the agent sees.
Visual.Capture	One capture of the current view. No camera movement.
Visual.ProjectPoints	World → screen pixels for up to 256 points in the current view. The crop maths.
Visual.ControlHud	Get/hide/show the game HUD (CGamePlaygroundUIConfig.OverlayHideAll) and optionally the Openplanet overlay.
Visual.SaveCameraState / Visual.RestoreCameraState	Named camera bookmarks. Slot auto holds the pose from before the last series.
Visual.Recover	Escape hatch: clear the busy guard, unhide the HUD and overlay, restore the camera.
Visual.SelfTest (DEV)	Run the frame-maths self checks, return JSON.

Visual.ShotSeries

Where the shots go — pick any combination; they append in this order.

Argument	Meaning
cameraOffsets: [{x,y,z}]	Explicit offsets from the target.
presets: ["front", …]	Named unit offsets: front back left right top bottom iso threeQuarter frontHigh. Groups: sixAxis, quad, hero.
orbit: {count, elevationDeg, startDeg, sweepDeg}	Turntable. sweepDeg (default 360) is divided by count, so a full turn does not repeat its first frame.

What they look at

Argument	Meaning
target: {x,y,z}	The world point the shots are about.
subject: {kind}	Resolve the target — and a heading — from something live. car (the viewed vehicle) or camera.
targetDistance	Put every camera this far from the target along its offset direction; the offset's length is then ignored, only its direction matters.
lookAtOffset: {x,y,z}	Added to the target to get the aim point. {"y":1} frames a car nicely: the camera aims a metre above the ground.
relativeTo: "world"|"subject"	subject rotates the offsets and the look-at offset by the subject's heading (yaw only), so front/left/top mean the same thing whichever way the car is facing.
fov	Field of view for the shots (playground free cam only); restored afterwards.

How they are taken

Argument	Default	Meaning
capture	native	native = the game's own clean screenshot. none = frame the shot, take no picture (for a host-side window grab).
format	webp	webp jpg tga dds.
settleFrames	4	Frames to wait after moving before capturing (the editor runs at ~30 fps).
hideHud	true	Hide the game HUD for the shots (OverlayHideAll), restored after.
hideOverlay	false	Viewport.DisableOverlayRender. hideHud already removes the HUD, and this flag is implicated in captures that never complete.
hideOpenplanetOverlay	true	Hide Openplanet's own windows, restored after.
forceRes / width / height	off	Render the capture at a forced resolution.
waitMs / retries	4000 / 1	Per-capture wait and re-queue count.
budgetMs	60000	Wall-clock ceiling for the whole series. Openplanet kills a coroutine that runs too long — stopping ourselves keeps the cleanup reachable.
namePrefix	(off)	Name the shots yourself: frame i becomes Documents/Trackmania/Agent/<namePrefix>-NN-<label>.<ext>. See Naming your captures.
fileName	(off)	Same, for a one-frame series (what Visual.Shot sends).
labels	generated	Per-frame labels.
dryRun	false	Compute and return the frames without moving anything.
restore	true	Put the camera back afterwards.

Response. Per frame: index, label, offset, cameraPos, lookAt, distance, yawDeg, pitchDeg, file, detected, sizeBytes, and targetPixel / lookAtPixel each with screenX, screenY, u, v, behind, onScreen. Top level: context, screen, frameCount, capturedCount, capture, restore, savedSlot.

Host-side helper: tools/visual.py

python3 tools/visual.py context
python3 tools/visual.py series --subject car --distance 9 \
        --presets front,left,top,iso --look-at-offset 0,1,0 --relative-to subject \
        --out ./shots --crop 700 --annotate --max-width 640 --contact-sheet
python3 tools/visual.py series --target 1005,8,731 --orbit 8,25 --distance 14 \
        --backend window --out ./turntable --contact-sheet
python3 tools/visual.py grab -o /tmp/now.png
python3 tools/visual.py diff before.png after.png -o /tmp/diff.png

It maps the game's C:\…\ScreenShotNN.webp into the Proton prefix, confirms freshness by mtime (see below), crops a square around the frame's projected target, draws a crosshair + a thirds grid + the frame label, downsizes for agent consumption, and can montage the series into a contact sheet. diff reports the changed-pixel bounding box and writes a heatmap — for before/after comparisons.

--backend window grabs the game's X11 (XWayland) window instead, driving the series frame by frame from the host. It takes a throwaway grab first: XWayland returns the previous frame roughly one grab in three, and without a baseline the opening shot of a series has nothing to compare against (observed — the first frame of a turntable came back showing the pre-series view). The image then contains whatever Openplanet is drawing, but this path does not depend on the engine's capture queue. Pillow is needed only for the image options.

Naming your captures

The engine names captures itself — ScreenShotNN.<ext> in the user game folder root, lowest free index, and it silently overwrites when all 99 are taken (the .jpg slots on this install are full). Agent shots then also mix into the user's own screenshots with no way to tell them apart.

So Visual.Capture and Visual.Shot take fileName, and Visual.ShotSeries takes namePrefix:

just tool Capture '{"fileName":"corner-42"}'
# -> Documents/Trackmania/Agent/corner-42.webp

just tool ShotSeries '{"subject":{"kind":"car"},"presets":["front","left"],
                       "targetDistance":10,"namePrefix":"hero"}'
# -> Documents/Trackmania/Agent/hero-00-front.webp, hero-01-left.webp

• Opt-in and inert without it: no name means the engine's own name, unchanged.
• Names are sanitized to [A-Za-z0-9._-] (96 chars max, one known extension trimmed), so a name is always a single file inside Agent/ — .. and path separators cannot escape it.
• The capture is moved out of its ScreenShotNN slot, which frees the slot again: named captures consume no slots and can never overwrite a screenshot of the user's. With no free slot for the format, a named capture refuses (NO_FREE_SLOT) instead of letting the engine overwrite one.
• The move is also the freshness check — a file in a slot that was empty before the capture is proof, which ScreenShotFullName never was.

How this works, and why it is a move rather than a memory patch: docs/CAPTURE-FILENAMES.md.

Capture: read this before trusting a screenshot

Measured on this install, 2026-09-11:

• Viewport.ScreenShotFullName is written when a capture is QUEUED, not when the file lands. So "the name changed" is not proof a file was written, and a reported path can point at an old file from a previous session. ShotSeries flags repeatedFile when two frames report the same path; tools/visual.py does the authoritative check by diffing the folder's mtimes.
• The engine intermittently stops completing queued captures — per format. An hour of 34 ms captures, then every capture timing out while the game kept rendering at a steady 30 fps. Narrowed on 2026-09-11: with every webp capture timing out, tga captures in the same minutes completed in 35-110 ms, so the queue and the file writer are alive and the webp encoder is what stops delivering. Window focus is not the trigger (six focused webp captures in a row failed). Workaround while it is in that state: capture tga and convert host-side. When the first frame's capture does not land, ShotSeries aborts the whole series with NATIVE_CAPTURE_UNAVAILABLE in a few seconds rather than burning waitMs on every remaining frame. The fallback is capture:"none" + tools/visual.py --backend window.
• jpg screenshot slots 01–99 are full, so an unnamed jpg capture overwrites one of the user's existing screenshots. That is why the default format is webp — and why a named capture refuses outright when its format has no free slot.
• The plain import command rings the X bell (a notification sound on this machine — clankercode/tm-control-mcp#18); tools/visual.py always passes -silent.

Contexts

Context	Camera driven via	Note
Map editor (MapEditor)	SetEditorCamera (orbital: target + h/v + distance)	No fov control.
Editor test run / race (playground)	ControlPlayCamera set_position (free cam)	fov supported. Beats the editor: while a test run is up the orbital camera is not the camera on screen.
Menu / loading	—	NO_CAPTURE_CONTEXT.

Frame maths

A   = target + lookAtOffset                     (the aim point)
C_i = target + offset_i                         (no targetDistance)
C_i = target + targetDistance * normalize(offset_i)
yaw   = atan2(d.x, d.z)      pitch = -asin(d.y / |d|)      d = A - C_i

which is the orbital convention the game itself uses (lookDir(h,v) = (cos v·sin h, −sin v, cos v·cos h), see tm-control-mcp's CameraMaths.md). Verified in-game: the target lands within 0.5 px of screen centre at 1920×1080. Zero-length offsets and aim points that coincide with the camera are refused by index rather than normalised into a random heading.

Full write-up, including every degenerate case and the ideas that were deliberately left out: docs/PLAN.md.

Testing

• just test — tests/test_visual_frames.py, 119 live checks against the running game. Every one either uses dryRun or restores what it moved, so it is safe to run against a session someone else is using. --allow-capture additionally writes real screenshot files.
• just lsp — static check.
• The frame maths also has in-game [Test] bodies (src/Frames_Test.as) that run on every DEV load and via Visual.SelfTest; tools/run-plugin-tests.sh turns those log markers into an exit code.

Authoring reference: ../tm-control-mcp/docs/tool-packs.md.