[Feature] Add Macro and Command Terminal

[Keybored02]

Description

This PR introduces a complete macro scripting system for Bambuddy, enabling users to automate printer actions — filament drying, print control, notifications, spool assignments, and more — through Jinja2 template scripts stored as plain .cfg files on disk.

Related Issue

Fixes #1139

Documentation

Companion docs PRs (delete lines that don't apply):

• Wiki: Add Macros and Gcode docs bambuddy-wiki#23

Pick one:

• Docs PR(s) linked above
• No docs update required — reason: ___

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update
• Code refactoring
• Performance improvement
• Test addition or update

Changes Made

File-first storage model

Macros live in Klipper-style .cfg files (e.g. data/macros/my_macros.cfg) rather than as DB text blobs. A single .cfg file can hold multiple [macro name] blocks. The DB stores only metadata (name, trigger type, cron expression, linked printer) and a foreign key to the MacroCfgFile record — the actual script body is always read from disk at render time.

This means macros can be version-controlled, edited externally with any text editor, and shared between Bambuddy instances by copying files.

Three-layer execution pipeline

1. Jinja2 render — The script body is rendered inside a SandboxedEnvironment (no arbitrary Python execution). Context variables (printer, ams, queue, vars, assignments) are injected from live MQTT state and DB queries.
2. Line dispatch — Each rendered line is classified as a G-code command, a system command, or a blank/comment and routed accordingly.
3. G-code whitelist enforcement — Before forwarding any G-code to the printer via MQTT, is_whitelisted() checks the first token against a curated allowlist. Unknown tokens are rejected with a log warning, never silently dropped.

Registry-based system commands

System commands (AMS_DRYING, NOTIFY, WAIT, SET_VAR, etc.) are registered via a @macro_function decorator on plain async coroutines in macro_integrations/*.py. The registry auto-discovers all files in that package at startup — adding a new command requires only a new decorated function, with zero changes to any other file.

Each function receives a typed FunctionContext (flags dict, printer_id, run_id, log callable) and returns a FunctionResult(ok, message, value). The value field is used by context_var functions that inject data into the Jinja2 render namespace.

---

New files

Backend

File	Role
backend/app/models/macro.py	MacroCfgFile, Macro, MacroRun SQLAlchemy 2.0 models
backend/app/models/macro_var.py	MacroVar — persistent key/value store for macro scripts
backend/app/schemas/macro.py	Pydantic request/response schemas with field-level validation
backend/app/api/routes/macros.py	REST API: CRUD for cfg files and macros, run/cancel, exec terminal, function catalogue
backend/app/services/macro_runner.py	Jinja2 render + line dispatch + cron scheduler + exec_line terminal
backend/app/services/macro_functions.py	@macro_function decorator, registry, FunctionContext, FunctionResult
backend/app/services/macro_files.py	Disk I/O for .cfg files with cross-platform path-traversal guard
backend/app/services/macro_cfg_parser.py	Parses [macro name] blocks from .cfg text; extracts config headers
backend/app/services/macro_cfg_watcher.py	Syncs parsed macros to DB on file save/delete
backend/app/services/gcode_whitelist.py	Allowlist of ~60 G-code prefixes + is_whitelisted() predicate
backend/app/services/macro_integrations/printer.py	PRINTER_PAUSE/RESUME/STOP, AMS_DRYING, WAIT_FOR_TEMP
backend/app/services/macro_integrations/notify.py	NOTIFY, WAIT
backend/app/services/macro_integrations/printer_extended.py	CLEAR_HMS_ERRORS, PRINT_QUEUE_ADD
backend/app/services/macro_integrations/assignments.py	ASSIGN_SPOOL, UNASSIGN_SPOOL, assignments context var
backend/app/services/macro_integrations/vars.py	SET_VAR, DELETE_VAR, vars context var

Frontend

File	Role
frontend/src/pages/MacrosPage.tsx	Full macros UI: cfg file list, macro list, editor, run history, terminal
frontend/src/components/CfgFileEditor.tsx	Editor for .cfg files with syntax hints panel

---

Detailed feature breakdown

.cfg file format (Klipper-style)

[macro preheat_bed]
description: Heat bed to 60C and wait
trigger: schedule
cron: 0 8 * * *
printer: My X1C

M140 S60
WAIT_FOR_TEMP --target=60 --tolerance=2
NOTIFY --message="Bed ready"

[macro log_filament]
trigger: manual
{% if vars.last_material != printer.ams[0].tray[0].material %}
SET_VAR --key=last_material --value="{{ printer.ams[0].tray[0].material }}"
NOTIFY --message="Filament changed to {{ vars.last_material }}"
{% endif %}

Config headers (description:, trigger:, cron:, printer:) are parsed before the script body. Everything after the first non-header line is the Jinja2 body. Duplicate block names are caught at parse time and surfaced in MacroCfgFile.parse_error.

Jinja2 context variables

Variable	Type	Source
printer	dict	Live MQTT state from printer_manager
ams	list[dict]	AMS tray data from MQTT state
queue	int	Count of pending PrintQueueItem rows
vars	dict	All non-expired MacroVar rows (global + macro-scoped)
assignments	list[dict]	Current SpoolAssignment rows for the target printer

Context variables are populated by context_var functions in the registry — the same @macro_function mechanism, with context_var="vars" instead of a command name. They run eagerly before each Jinja2 render.

System commands

All commands return structured FunctionResult(ok, message). On failure, ok=False and the log line is prefixed with error: so the UI can highlight it in red.

Printer control (allowed_in_embed=False — cannot run from G-code embeds):

• PRINTER_PAUSE / RESUME / STOP — wraps the printer client pause/resume/stop methods; distinguishes "not connected" from "command rejected by firmware"
• AMS_DRYING --ams=N --temp=T --duration=H — validates ams 0-3, temp 20-90C, duration 1-12h; calls send_drying_command()
• WAIT_FOR_TEMP --target=T [--tolerance=D] [--max_wait=S] — polls nozzle temp every 2s; validates target 0-350C

Notifications / timing (allowed_in_embed=True):

• NOTIFY --message="..." — dispatches through the configured notification providers; distinguishes "no providers configured" (ok, no-op) from "dispatch error"
• WAIT --seconds=N — async sleep; validates N > 0

Extended printer (allowed_in_embed=False):

• CLEAR_HMS_ERRORS — sends HMS clear command; reports "no errors to clear" vs "cleared N errors"
• PRINT_QUEUE_ADD --file_id=N [--plate=N] — validates file_id > 0, plate >= 1; adds item to print queue

Spool assignments (ASSIGN/UNASSIGN allowed_in_embed=False):

• ASSIGN_SPOOL --spool_id=N --ams=A --tray=T — upserts SpoolAssignment, captures fingerprint color/type from live MQTT state, broadcasts spool_assignment_changed via WebSocket
• UNASSIGN_SPOOL --ams=A --tray=T — deletes assignment, broadcasts same event
• assignments context var — injects current assignments as a list of dicts

Persistent variables (allowed_in_embed=True):

• SET_VAR --key=K --value=V [--ttl=S] [--scope=global|macro] — upserts MacroVar; value is JSON-encoded (numbers, bools, lists all work); optional TTL for auto-expiry
• DELETE_VAR --key=K [--scope=...] — removes the var
• vars context var — merges global and macro-scoped vars; scoped vars shadow globals with the same key; expired rows are excluded at read time

Execution engine details

Log buffering (_LogBuffer): Instead of one DB write per log line (N+1 problem), log writes are batched in memory and flushed to MacroRun.log every 10 lines (or on explicit flush at end of run). This keeps SQLite happy during long macro runs.

G-code safety (_preflight()):

• Rejects any command not in GCODE_WHITELIST
• Rejects G0/G1 with XY coordinates — XY movement via gcode_line is unsafe on Bambu firmware because G91 is ignored for XY, making coordinates always absolute and risking toolhead crashes. Z-only moves are allowed.
• Rejects homing/heating commands (G28, M104, M140, etc.) while printer is in RUNNING state

Concurrency: run_macro() uses asyncio.create_task() and registers the task in self._running_tasks[run_id]. cancel_run(run_id) calls task.cancel(). All DB I/O uses fresh async_session() per call — no shared sessions across async tasks.

Cron scheduler: start_scheduler() launches a background asyncio task that wakes every 60s. It uses croniter.match() to check each schedule macro against the current UTC time. A last_fired: dict[int, datetime] guard prevents double-fires within the same minute window.

Sub-macro calls: The Jinja2 context includes a run_macro("name") callable. It resolves the name to a Macro DB record, reads and renders the .cfg body, and executes it inline within the parent run log. A call_stack: set[str] is threaded through recursive calls; a repeated name raises an error to prevent infinite recursion.

Terminal exec (POST /macros/exec): Single-line execution for the in-app terminal. Invokes a full macro by name using the run: macro_name prefix — this avoids the ambiguity of a macro named e.g. G28 shadowing the actual G-code token. Plain G-code lines go directly through _dispatch_gcode().

Path-traversal protection

macro_files._safe_path() uses Path.relative_to(macros_dir.resolve()) rather than a string prefix check. The string-prefix approach broke on Windows because Path.resolve() returns backslash-separated paths while the old code appended a forward-slash sentinel, causing all valid filenames to be incorrectly rejected with a 500 error.

Permissions

Five new permissions added to the Permission enum and wired into DEFAULT_GROUPS:

Permission	Administrators	Operators	Viewers
macros:read	yes	yes	no
macros:create	yes	no	no
macros:update	yes	no	no
macros:delete	yes	no	no
macros:run	yes	yes	no

Operators can read the macro list and trigger runs but cannot create or modify macros.

Terminal session inheritance fix

Opening the G-code terminal from the Printers page previously created an isolated browsing context (window.open with noopener) that had no auth token, forcing the user to log in again. Fixed by appending ?token=<current_token> to the popup URL — AuthContext already reads and applies this query parameter on load.

---

Test plan

• Start backend — verify macro_cfg_files, macros, macro_runs, macro_vars tables created
• GET /api/v1/macros/gcode-whitelist returns a sorted list of strings
• GET /api/v1/macros/functions returns all registered system commands with args
• Create a .cfg file via UI; verify file appears on disk at data/macros/*.cfg
• Edit and save the file; verify DB macros synced to match the new block list
• Run a manual macro with G28; verify MacroRun.status = success and log contains the dispatched line
• Run a macro with an unlisted G-code (e.g. G999); verify it is blocked with an error: log line
• Run AMS_DRYING --ams=0 --temp=65 --duration=4; verify send_drying_command() called in logs
• Run WAIT --seconds=2; verify ~2s delay visible in run log
• Run SET_VAR --key=x --value=42 then a macro using {{ vars.x }}; verify value injected
• Run ASSIGN_SPOOL --spool_id=1 --ams=0 --tray=0; verify SpoolAssignment row and WebSocket event
• Cancel an active run via POST /macros/runs/{id}/cancel; verify task cancelled and status updated
• Create a schedule macro with cron * * * * *; wait 60s; verify auto-run created
• Terminal: type run: macro_name to invoke a macro; type G28 directly to dispatch G-code
• Navigate to /macros in frontend; create, edit, run a macro end-to-end; watch run history poll update
• Open terminal from Printers page; verify session is inherited (no login prompt in popup)
• Attempt path traversal: PUT cfg-file with relative_path ../../../etc/passwd; verify 400 rejected

Screenshots

Testing

• I have tested this on my local machine
• I have tested with my printer model: H2C

Checklist

• My code follows the project's coding style
• I have commented my code where necessary
• My changes generate no new warnings
• I have tested my changes thoroughly

Additional Notes

[Keybored02]

What needs to be done:

• Testing: there's a ton of stuff that can break herr
• Per-printer specs: axis length, max temperatures, end stops, nozzle definitions are currently not defined
• System commands: the ones that I could set as auto-discoverable are included. The rest needs to be added by hand (like {{ assignments }})
• Gcode commands: the list is mostly untested. Most printer-specific commands are not really helpful, but they still need testing across all machone types.

[maziggy]

Open tasks should be done before merging.

[Keybored02]

Beside testing, the other tasks are way outside of the scope of this PR and would need broad community support. They're not requisites for this, but rather ways in which the feature can move forward.

[Keybored02]

Critical fixes — macro execution engine

• Webhook double-run / orphan row (C1): The webhook endpoint was creating a MacroRun row itself and then calling run_macro() without passing run_id=, causing the runner to silently create a second row. The first row was left permanently in pending. Fixed by passing run_id to the runner.

• Webhook trigger-type bypass (C2): Any API key with macro permissions could trigger manual or schedule macros via the webhook URL — the trigger_type field was purely cosmetic. The endpoint now returns HTTP 400 if the target macro is not configured as trigger_type = "webhook".

• Broken {{ run_macro() }} Jinja2 callable (C3): The Jinja2 template context exposed a run_macro("name") function that internally called asyncio.ensure_future() inside a synchronous render — meaning the sub-macro ran as a background task with no ordering guarantee, no error propagation, and no allow_printer_commands enforcement. This is replaced by a proper imperative system command: MACRO --name=, which executes synchronously in the dispatch loop and propagates errors like any other command. The old callable has been removed from the context.

• Embed-mode bypass via sub-macro (C4): _run_sub_macro didn't accept or honour allow_printer_commands, so a gcode_embed-triggered macro that called a sub-macro could issue G-code and printer system commands that the embed restriction was supposed to block. allow_printer_commands is now threaded through the full call chain.

• Silent undefined variables (C5): SandboxedEnvironment was created without undefined=StrictUndefined, so typos in template variables (e.g. {{ priinter.nozzle_temp }}) silently rendered as empty strings. This made debugging macro scripts painful. StrictUndefined now raises a UndefinedError at render time, which the runner catches and logs with status error.

• Migration note for users: If you have existing macros that use {{ run_macro("name") }} syntax, replace those calls with MACRO --name= on its own line in the script body.

[Keybored02]

Medium/low fixes — macro system reliability and data integrity

• Log append race (M2/H5): Both _LogBuffer.flush() and the cancel API path used an ORM read-modify-write pattern (run.log = (run.log or "") + blob). If the runner task and the cancel endpoint wrote concurrently, one write silently overwrote the other. Both paths now use a single UPDATE … SET log = COALESCE(log,'') || :blob statement, which is atomic at the DB level.

• Duplicate G-code log lines (M3): The main dispatch loop logged [GCODE] {line} before adding each line to the batch, and then _send_gcode logged each line again after the MQTT publish succeeded. The result was every G-code command appearing twice in run logs. The pre-batch log write has been removed.

• Silent queue count failure (M5): _get_queue_count caught all exceptions and returned 0. A real DB error was indistinguishable from an empty queue, causing macros that branch on {{ queue }} to silently take the wrong path. It now logs a warning.

• Silent command disappearance on startup (M7): discover() was catching all exceptions from integration module imports and logging them, then continuing. A typo or missing dependency in any integration file would silently disable all its commands — users would only notice when a macro failed with "Unknown system command". Import errors now propagate and crash startup, which is the correct behaviour for a configuration error.

• Orphaned .cfg files on non-IntegrityError DB failure (M8): create_cfg_file only cleaned up the on-disk file when sync_file raised IntegrityError. Any other DB error (e.g. connection failure, constraint on a different column) left the file on disk with no DB record. The cleanup now catches all exceptions.

• Double scheduler task (L2): start_scheduler would create a second scheduler task if called twice (e.g. due to a lifespan restart). Both tasks would fire cron jobs independently. It now guards against an already-running task. stop_scheduler also nulls the task reference.

• Concurrent SET_VAR produces duplicate rows (L3): MacroVar had no uniqueness constraint on (key, macro_id). Two concurrent SET_VAR calls for the same key and scope could both INSERT, leaving duplicate rows — subsequent reads would pick one unpredictably. A UNIQUE(key, macro_id) constraint is added to both the model and via an idempotent migration.

• Leading semicolon in parse_error (L7): When a name-conflict was the only error on a cfg file, parse_error was set to "; Name conflict: ..." (leading semicolon). Fixed to produce "Name conflict: ..." when there is no prior error, or "{prior}; Name conflict: ..." when there is.

[Keybored02]

Low-severity fixes — flag parsing, cron validation, session hygiene

• _parse_flags drops bare flags and mishandles --value args (L1): The parser guarded elif not tokens[i+1].startswith("--") before consuming the next token as a flag value. This had two problems: (1) a legitimate quoted argument whose text starts with -- (e.g. NOTIFY --message="--help", which shlex unwraps to ["NOTIFY", "--message", "--help"]) would be silently dropped as if it were a second flag; (2) a bare --flag with no following value (e.g. WAIT --quiet) was not stored at all, making it impossible for a function to detect flag presence vs. absence. Bare flags are now stored as "" so callers can distinguish flag in ctx.flags from ctx.flags.get("flag", "default").

• Cron expression only validated at scheduler tick (L5): A malformed cron: value in a .cfg file was stored in the DB as-is and only caught 60 seconds later when the scheduler tick tried croniter.match() and logged a warning. The error wasn't surfaced to the user and parse_error on the file was never updated. macro_cfg_parser now calls croniter.is_valid() at parse time; an invalid expression adds to parse_error immediately on file save so the UI shows it inline.

• Redundant SELECT after commit in run_macro route (L8): After committing the new MacroRun row, the route did run = await db.get(MacroRun, run_id) — re-fetching the same row it had just flushed. The ORM object is still in the session; db.refresh(run) is both correct and avoids the unnecessary round-trip.

[Keybored02]

Test Coverage Report — feature/macro

91 tests across 5 files. 71% overall line coverage on new macro code.

Module	Coverage	Notes
macro_files.py	100%	All file I/O paths exercised
macro_cfg_parser.py	99%	1 unreachable line (post-ImportError guard)
macro_functions.py	85%	Uncovered: error paths in the decorator registry and unused arg-spec helpers
macro_runner.py	74%	See below
macros.py (routes)	79%	See below
macro_cfg_watcher.py	36%	See below
webhook.py	42%	Macro webhook is covered; the 58% miss is pre-existing non-macro routes (queue, printer control)

Notable gaps

macro_runner.py (74%) — uncovered lines are:

• The cron scheduler loop (lines 370–406) — tested via direct tick invocation; the full asyncio.sleep loop body is not exercised
• _run_sub_macro edge cases: allow_printer_commands=False propagation into sub-macro G-code batching
• G-code batch flush and HMS error detection after send (lines 507–526)
• _build_context Jinja2 context construction paths (printer state enrichment)

macros.py routes (79%) — uncovered lines are:

• GET /macros/{id}/vars and PUT /macros/{id}/vars (macro variable endpoints, lines 102–148)
• exec terminal run: error path for already-running macros
• Some cancel-run edge paths

macro_cfg_watcher.py (36%) — the watcher's filesystem watch loop (start_watching / stop_watching, lines 37–55) and cross-file collision handling are not covered. These are integration-heavy: they require a real filesystem event or the full startup lifecycle. The sync path that route tests exercise is covered; the background watcher daemon is not.

What is covered well

• All cfg-file CRUD operations end-to-end (create, read, update, delete, duplicate rejection)
• Full macro execution pipeline: template rendering, G-code dispatch, system commands, embed-mode blocking, cycle detection, cancel
• All API status codes: 200, 404, 409, 400, 401/403
• Parser edge cases: comments, duplicate names, cron validation, case-insensitive config, serialize roundtrip
• Webhook auth enforcement and orphan-run prevention (run_id propagation)

[Keybored02]

Need anything else for this before a review?