From c33c6a03a1a5680899d1073acadb0ba539f4a535 Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 23 Jul 2026 13:41:33 +0200 Subject: [PATCH 01/23] Document v0.1.0 readiness and roadmap Replace the stale feature overview with a source-install quick start and an evidence-backed account of the compiler capabilities already implemented. Set expectations for the preview release by naming the unstable interfaces, recipe limitations, verification gaps, and the priorities on the active roadmap. --- README.md | 269 ++++++++++++++++++++++++------------------------------ 1 file changed, 119 insertions(+), 150 deletions(-) diff --git a/README.md b/README.md index b639ec278..96c531caa 100644 --- a/README.md +++ b/README.md @@ -3,202 +3,171 @@ [![Ask DeepWiki](https://deepwiki.com/badge.svg)]( https://deepwiki.com/leynos/netsuke) -A modern, declarative build system compiler. YAML + Jinja in, Ninja out. -Nothing more. Nothing less. +*A friendly build-system compiler: YAML and Jinja in, Ninja out.* -## What is Netsuke? +Netsuke turns a readable `Netsukefile` into a validated, static Ninja build +graph. It keeps the dynamic work in a higher-level manifest and leaves fast, +incremental execution to [Ninja](https://ninja-build.org/). -**Netsuke** is a friendly build system that compiles structured manifests into -a Ninja build graph. It’s not a shell-script runner, a meta-task framework, or -a domain-specific CI layer. It’s `make`, if `make` hadn’t been invented in 1977. +______________________________________________________________________ -### Key properties +## Why Netsuke? -- **Declarative**: Targets, rules, and dependencies described explicitly. -- **Dynamic when needed**: Jinja templating for loops, macros, conditionals, - file globbing. -- **Static where required**: Always compiles to a reproducible, fully static - dependency graph. -- **Unopinionated**: No magic for C, Rust, Python, JavaScript, or any other - blessed language. -- **Safe**: All variable interpolation is securely shell-escaped by default. -- **Fast**: Builds executed by [Ninja](https://ninja-build.org/), the fastest - graph executor we know of. +- **Readable manifests**: Describe rules, targets, dependencies, and defaults + in YAML instead of a tab-sensitive language. +- **Dynamic planning**: Use Jinja variables, macros, `foreach`, `when`, and + globbing before Netsuke creates the build graph. +- **Static execution**: Inspect the generated Ninja file or render the graph + before running any build command. +- **Useful diagnostics**: Get source-aware errors, localized output, progress + reporting, and an opt-in JSON diagnostic format. +- **No blessed toolchain**: Use the same manifest model for Rust, C, Python, + web projects, or anything else a command can build. -## Quick Example +______________________________________________________________________ -```yaml -netsuke_version: "1.0" - -vars: - cc: clang - cflags: -Wall -Werror - -rules: - - name: compile - command: "{{ cc }} {{ cflags }} -c {{ ins }} -o {{ outs }}" +## Quick start - - name: link - command: "{{ cc }} {{ cflags }} {{ ins }} -o {{ outs }}" - -targets: - - foreach: glob('src/*.c') - name: "build/{{ item | basename | with_suffix('.o') }}" - rule: compile - sources: "{{ item }}" - - - name: app - rule: link - sources: "{{ glob('src/*.c') | map('basename') | map('with_suffix', '.o') }}" -``` +### Prerequisites -Yes, it’s just YAML. Yes, that’s a Jinja `foreach`. No, you don’t need to define -`.PHONY` or remember what `$@` means. This is the present day. You deserve -better. +Netsuke currently requires: -## Key Concepts +- [Ninja](https://ninja-build.org/) on `PATH`; +- Rust 1.89 or later when installing from source. -### 🔨 Rules +### Installation -Rules are reusable command templates. Each one has exactly one of: +Until the v0.1.0 release is published, install the current source checkout with +Cargo: -- `command:` - a single shell string -- `script:` - a multi-line block -- (or) can be declared inline on a target - -```yaml -rules: - - name: rasterise - script: | - inkscape --export-png={{ ins }} {{ outs }} +```sh +git clone https://github.com/leynos/netsuke.git +cd netsuke +cargo install --path . ``` -### 🎯 Targets +### Your first build -Targets are things you want to build. +Create a new directory and add a file named `Netsukefile`: ```yaml -- name: build/logo.png - rule: rasterise - sources: assets/logo.svg -``` - -Targets can also define: - -- `deps`: implicit dependencies — trigger rebuilds but are not passed to - `$in`/`{{ ins }}`; map to Ninja `|` -- `order_only_deps`: e.g. `mkdir -p build` -- `vars`: per-target variables +netsuke_version: "1.0.0" -You may also use `command:` or `script:` instead of referencing a `rule`. +targets: + - name: hello.txt + command: "echo 'Hello from Netsuke!' > hello.txt" -## 🧪 Phony Targets and Actions +defaults: + - hello.txt +``` -Phony targets behave like Make’s `.PHONY`: +Run Netsuke, then inspect the result: -```yaml -- name: clean - phony: true - always: true - command: rm -rf build +```sh +netsuke +cat hello.txt ``` -For cleaner structure, you may also define phony targets under an `actions:` -block: +The second command prints `Hello from Netsuke!`. See the +[quick-start guide](docs/quickstart.md) for variables, templates, and `foreach`. -```yaml -actions: - - name: test - command: pytest -``` +______________________________________________________________________ -All `actions` are treated as `{ phony: true, always: false }` by default. +## What works today -## 🧠 Templating +The core build-system compiler is implemented: -Netsuke uses [MiniJinja](https://docs.rs/minijinja) to render your manifest -before parsing. +- YAML 1.2 manifest parsing with duplicate-key and schema validation; +- Jinja variables, macros, `foreach`, `when`, globbing, environment helpers, + executable discovery, and opt-in network helpers; +- reusable rules, targets, actions, defaults, and explicit, implicit, and + order-only dependencies; +- a deterministic intermediate build graph with duplicate-output, missing-rule, + and cycle checks; +- Ninja generation and execution, plus `clean` and standalone manifest + generation; +- reproducible dependency graphs as Graphviz DOT or self-contained, + accessible HTML; +- layered configuration, localized output, accessibility preferences, + progress reporting, stage timings, and JSON diagnostics; +- unit, behavioural, integration, property, snapshot, and initial Kani + verification coverage. -You can: +Release automation is configured to build packages for Linux, macOS, and +Windows, including platform help artefacts. v0.1.0 will be the first public +release of this work. -- Glob files: `{{ glob('src/**/*.c') }}` -- Read environment vars: `{{ env('CC') }}` -- Use filters: `{{ path | basename | with_suffix('.o') }}` -- Define reusable macros: +______________________________________________________________________ - ```yaml - macros: - - signature: "shout(msg)" - body: | - echo "{{ msg | upper }}" - ``` +## v0.1.0 status -Templating happens **before** parsing, so any valid output must be valid YAML. +v0.1.0 is a useful preview for early adopters, not a declaration that Netsuke +is finished or that every interface is stable. The compiler pipeline and +ordinary local-build workflow are substantial; the command-line interface, +configuration vocabulary, and advanced recipe model are still evolving. -## 🔐 Safety +Pin the Netsuke version in automation and expect some command names, flags, +diagnostic schemas, and manifest details to change before 1.0. -Shell commands are automatically escaped. Interpolation into `command:` or -`script:` will never yield a command injection vulnerability unless you -explicitly ask for `| raw`. +Known limitations include: -```yaml -command: "echo {{ dangerous_value }}" # Safe -command: "echo {{ dangerous_value | raw }}" # Unsafe (your problem now) -``` +- recipes are shell strings; structured executable arguments and recipe + environment mappings are not implemented yet; +- literal shell dollar expressions still need Ninja-aware escaping in + manifests; +- compiler-generated dependency imports such as GCC depfiles are planned but + not yet part of the manifest model; +- the current JSON mode covers diagnostics rather than a stable structured + result for every command; +- accessibility, terminal rendering, configuration precedence, and + cross-platform compiler invariants need broader verification. -## 🔧 CLI +A `Netsukefile` can execute commands and use impure template helpers. Treat it +with the same care as a `Makefile`: review untrusted manifests before running +them. Netsuke quotes supported path substitutions, but it is not a sandbox. -```shell -netsuke [build] [target1 target2 ...] -netsuke clean -netsuke graph - netsuke manifest FILE -``` +______________________________________________________________________ -- `netsuke` alone builds the `defaults:` targets from your manifest -- `netsuke graph` emits a Graphviz `.dot` of the build DAG -- `netsuke clean` runs `ninja -t clean` -- `netsuke manifest FILE` writes the Ninja manifest to `FILE` without invoking - Ninja +## The road ahead -You can also pass: +Work after the first release is organized around three priorities: -- `--file` to use an alternate manifest -- `--directory` to run in a different working dir -- `-j N` to control parallelism (passed through to Ninja) -- `-v`, `--verbose` to enable verbose logging +1. **Stabilize the command-line contract**: adopt canonical command and flag + names, non-interactive safeguards, stable exit codes, bounded output, and + one structured JSON result per command. +2. **Make recipes safer and clearer**: add structured executable arguments, + environment mappings, compiler dependency imports, backend dollar escaping, + and better conditional-action feedback. +3. **Strengthen confidence**: expand Kani and property-test coverage, verify + accessibility with assistive technology, and add regression coverage for + configuration precedence and terminal rendering. -Release builds include a `netsuke.1` manual page generated by `cargo-orthohelp` -from the CLI documentation metadata, providing the same flags and subcommands -documented via `--help`. Windows release artefacts also include PowerShell -external help for `Get-Help Netsuke -Full`. Manual page generation honours -`SOURCE_DATE_EPOCH` for reproducible dates. If the value is invalid, a warning -is emitted and the date falls back to `1970-01-01`. If `SOURCE_DATE_EPOCH` is -unset, the date deterministically falls back to `1970-01-01` without a warning. -The published crate does not include these files; packagers can source them -from release artefacts under `target/orthohelp//release/`. +Longer-term work explores machine-readable context, profiles, run history, +artefact delivery, and local-first feedback for human and agent workflows. The +[roadmap](docs/roadmap.md) tracks the detailed sequence and current progress. -## 🚧 Status +______________________________________________________________________ -Netsuke is **under active development**. It’s not finished, but it’s buildable, -usable, and increasingly delightful. +## Learn more -Coming soon: +- [Quick-start guide](docs/quickstart.md) — build something in five minutes. +- [Users' guide](docs/users-guide.md) — manifest and command reference. +- [Design document](docs/netsuke-design.md) — architecture and design + rationale. +- [Developers' guide](docs/developers-guide.md) — development workflow and + quality gates. +- [Roadmap](docs/roadmap.md) — completed foundations and planned work. -- `graph --html` for interactive DAGs -- Extensible plugin system for filters/functions -- Toolchain presets (`cargo`, `node`, etc.) +______________________________________________________________________ -## Why “Netsuke”? +## Licence -A **netsuke** is a small carved object used to fasten things securely to a -belt. It’s not the sword. It’s not the pouch. It’s the thing that connects them. +ISC — see [LICENSE](LICENSE) for details. -That’s what this is: a tidy connector between your intent and the tool that -gets it done. +______________________________________________________________________ -## Licence +## Contributing -Netsuke is distributed under the -[ISC licence](https://opensource.org/licenses/ISC). You don't need a legal -thesis to use a build tool. +Contributions are welcome. Start with the +[developers' guide](docs/developers-guide.md); automated contributors should +also follow [AGENTS.md](AGENTS.md). From e4e655a47a0af519b066ee19d3688387aace052b Mon Sep 17 00:00:00 2001 From: leynos Date: Thu, 23 Jul 2026 14:07:15 +0200 Subject: [PATCH 02/23] Align user guide with v0.1.0 behaviour Replace the sprawling draft guide with a task-oriented account of the implemented manifest, CLI, configuration, accessibility, diagnostic, and safety surfaces. Correct stale repository examples so their recipes compile and their defaults resolve to deterministic build edges. Bind every fenced README and user's-guide example to executable integration coverage, and reuse the novice first-run behavioural flow for both introductory manifests. --- README.md | 6 + docs/developers-guide.md | 17 + docs/users-guide.md | 1708 +++++------------ examples/basic_c.yml | 10 +- examples/photo_edit.yml | 21 +- examples/visual_design.yml | 19 +- examples/website.yml | 25 +- examples/writing.yml | 17 +- tests/bdd/steps/documentation_examples.rs | 19 + tests/bdd/steps/mod.rs | 1 + tests/bdd_tests.rs | 1 + tests/documentation_examples/mod.rs | 163 ++ tests/documentation_examples_tests.rs | 347 ++++ tests/features/documentation_examples.feature | 17 + 14 files changed, 1103 insertions(+), 1268 deletions(-) create mode 100644 tests/bdd/steps/documentation_examples.rs create mode 100644 tests/documentation_examples/mod.rs create mode 100644 tests/documentation_examples_tests.rs create mode 100644 tests/features/documentation_examples.feature diff --git a/README.md b/README.md index 96c531caa..381c6dbfa 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,8 @@ Netsuke currently requires: Until the v0.1.0 release is published, install the current source checkout with Cargo: + + ```sh git clone https://github.com/leynos/netsuke.git cd netsuke @@ -50,6 +52,8 @@ cargo install --path . Create a new directory and add a file named `Netsukefile`: + + ```yaml netsuke_version: "1.0.0" @@ -63,6 +67,8 @@ defaults: Run Netsuke, then inspect the result: + + ```sh netsuke cat hello.txt diff --git a/docs/developers-guide.md b/docs/developers-guide.md index f794841d2..b0620aa79 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -468,6 +468,23 @@ there. The tests require workflow YAML files under `.github/workflows` and ensure local composite action manifests under `.github/actions` are covered by the configured Dependabot directory patterns. + +### User-facing documentation examples + +Every fenced example in `README.md` and `docs/users-guide.md` has a stable +`tested-example` marker immediately before its opening fence. The shared +`tests/documentation_examples/mod.rs` loader owns this marker format and may be +called only by documentation-focused integration or behavioural tests. It +rejects unmarked fences, duplicate identifiers and unterminated examples. + +`tests/documentation_examples_tests.rs` compiles every manifest fence, +exercises the command and output examples against the current binary, and +compiles each complete manifest linked from the user's guide. The first-run +README and user's guide examples also run through +`tests/features/documentation_examples.feature`, reusing the same fake-Ninja +behavioural flow as the novice smoke tests. Tests must load the fenced text +through the shared helper instead of maintaining copied fixtures. + ### Property-based testing with proptest `proptest` generates randomized inputs to verify invariants that must hold for diff --git a/docs/users-guide.md b/docs/users-guide.md index a406cdccc..cc143a518 100644 --- a/docs/users-guide.md +++ b/docs/users-guide.md @@ -1,879 +1,595 @@ -# Netsuke User Guide +# Netsuke user's guide -## 1\. Introduction: What is Netsuke? +This guide is for people evaluating or using Netsuke v0.1.0. It covers the +first build, the manifest format, templating, command-line usage, +configuration, diagnostics, accessibility and the current safety boundary. -Netsuke is a modern, declarative build system designed to be intuitive, fast, -and safe. Think of it not just as a `make` replacement, but as a **build system -compiler**. You describe your build process in a human-readable YAML manifest -(`Netsukefile`), leveraging the power of Jinja templating for dynamic logic. -Netsuke then compiles this high-level description into an optimized build plan -executed by the high-performance [Ninja](https://ninja-build.org/ "null") build -system. +Netsuke v0.1.0 is an early-adopter release. The compiler pipeline is useful, +but command names, flags, diagnostic schemas and some manifest details may +change before 1.0. Pin the Netsuke version in automated workflows. -**Core Philosophy:** +## Install Netsuke -- **Declarative:** Define *what* you want to build, not *how* step-by-step. +Netsuke requires [Ninja](https://ninja-build.org/) on `PATH`. A source build +also requires Rust 1.89 or later. -- **Dynamic where needed:** Use Jinja for variables, loops (`foreach`), - conditionals (`when`), file globbing (`glob`), and more. +Until v0.1.0 is published, the current checkout can be installed with Cargo: -- **Static Execution Plan:** All dynamic logic is resolved *before* - execution, resulting in a static Ninja file for fast, reproducible builds. - -- **Safety First:** Automatic shell escaping prevents command injection - vulnerabilities. - -- **Fast Execution:** Leverages Ninja for efficient dependency tracking and - parallel execution. - -## 2\. Getting Started - -### Installation - -Netsuke is typically built from source using Cargo: + ```sh -cargo build --release -# The executable will be in target/release/netsuke - +git clone https://github.com/leynos/netsuke.git +cd netsuke +cargo install --path . ``` -Refer to the project's `README.md` or release pages for pre-compiled binaries -if available. Ensure the `ninja` executable is also installed and available in -your system's `PATH`. +Release archives contain platform-specific packages and help artefacts. Unix +archives include a `netsuke.1` manual page. Windows archives include PowerShell +external help: -### Release help artefacts - -Release archives include platform help files generated from the same command -metadata that powers `netsuke --help`. Unix-like release artefacts include the -`netsuke.1` manual page. Windows release artefacts include PowerShell external -help in a `Netsuke` module layout, so installed or unpacked artefacts can be -inspected with: + ```powershell Get-Help Netsuke -Full ``` -This change affects release packaging only. The Netsuke command-line flags, -subcommands, output, and exit statuses are unchanged. - -### Basic Usage - -The primary way to use Netsuke is through its command-line interface (CLI). The -default command is `build`. - -1. **Create a `Netsukefile`:** Define your build rules and targets in a YAML - file named `Netsukefile` in your project root. - -2. **Run Netsuke:** Execute the `netsuke` command. - -```sh -netsuke # Builds default targets defined in Netsukefile -netsuke build target_name another_target # Builds specific targets - -``` - -If no `Netsukefile` is found, Netsuke will provide a helpful error message: - -```text -Error: Manifest 'Netsukefile' not found in the current directory. -help: Ensure the manifest exists or pass `--file` with the correct path. -``` - -A different manifest path can be specified using the `-f` or `--file` option: - -```sh -netsuke -f path/to/manifest.yml -``` - -For a step-by-step introduction, see the [Quick Start guide](quickstart.md). - -## 3\. The Netsukefile Manifest +## Run the first build -The `Netsukefile` is a YAML file describing the build process. +Create an empty project directory and add a file named `Netsukefile` with the +following complete manifest: -Netsuke targets YAML 1.2 and forbids duplicate keys in manifests. If the same -mapping key appears more than once (even if a YAML parser would normally accept -it with “last key wins” behaviour), Netsuke treats this as an error. - -### Top-Level Structure + ```yaml -# Mandatory: Specifies the manifest format version netsuke_version: "1.0.0" -# Optional: Global variables accessible throughout the manifest -vars: - cc: gcc - src_dir: src - -# Optional: Reusable Jinja macros -macros: - - signature: "compile_cmd(input, output)" - body: | - {{ cc }} -c {{ input }} -o {{ output }} - -# Optional: Reusable command templates -rules: - - name: link - command: "{{ cc }} {{ ins }} -o {{ outs }}" - -# Optional: Phony targets often used for setup or meta-tasks -# Implicitly phony: true, always: false -actions: - - name: clean - command: "rm -rf build *.o" - -# Required: Defines the build targets (artefacts to produce) targets: - - name: build/main.o - # A target can define its command inline... - command: "{{ compile_cmd('src/main.c', 'build/main.o') }}" - sources: src/main.c - - - name: my_app - # ...or reference a rule - rule: link - sources: build/main.o + - name: hello.txt + command: "echo 'Hello from Netsuke!' > hello.txt" -# Optional: Targets to build if none are specified on the command line defaults: - - my_app - + - hello.txt ``` -### Key Sections Explained +Run Netsuke without a subcommand to build the manifest's `defaults`, then read +the generated file: -- `netsuke_version` (**Required**): Semantic version string (e.g., `"1.0.0"`) - indicating the manifest schema version. Parsed using `semver`. + -- `vars` (Optional): A mapping (dictionary) of global variables. Values can - be strings, numbers, booleans, or lists, accessible within Jinja expressions. +```sh +netsuke +cat hello.txt +``` -- `macros` (Optional): A list of Jinja macro definitions. Each item has a - `signature` (e.g., `"my_macro(arg1, arg2='default')"` ) and a multi-line - `body`. +The second command prints `Hello from Netsuke!`. -- `rules` (Optional): A list of named, reusable command templates. +If Netsuke cannot find `Netsukefile`, it reports the missing file and suggests +`--file`. A different path can be selected with +`netsuke --file path/to/manifest.yml build`. -- `targets` (**Required**): A list defining the primary build outputs (files - or logical targets). +The [quick-start guide](quickstart.md) provides a longer walkthrough. -- `actions` (Optional): Similar to `targets`, but entries here are implicitly - treated as `phony: true` (meaning they don't necessarily correspond to a file - and should run when requested). Useful for tasks like `clean` or `test`. +## Understand the build model -- `defaults` (Optional): A list of target names (from `targets` or `actions`) - to build when `netsuke` is run without specific targets. +Netsuke is a build-system compiler. A build moves through six stages: -## 4\. Defining Rules +1. Read the manifest. +2. Parse YAML 1.2. +3. Expand manifest-time `foreach` and `when` expressions. +4. Deserialize the typed manifest and render string fields. +5. Build and validate a static intermediate representation (IR). +6. Generate Ninja and, for `build` or `clean`, run Ninja. -Rules encapsulate reusable build commands or scripts. +All manifest-time decisions finish before Ninja starts. The generated graph is +therefore static and inspectable. -```yaml -rules: - - name: compile # Unique identifier for the rule - # Recipe: Exactly one of 'command', 'script', or 'rule' - command: "{{ cc }} {{ cflags }} -c {{ ins }} -o {{ outs }}" - # Optional: Displayed during the build - description: "Compiling {{ outs }}" +A `Netsukefile` is executable build configuration, not passive data. Commands, +scripts and impure template helpers can access the host. Review an untrusted +manifest with the same care as an untrusted `Makefile`. -``` +## Author a manifest -- `name`: Unique string identifier. +`Netsukefile` is a YAML mapping. Unknown fields and duplicate mapping keys are +errors. `netsuke_version` and `targets` are required; all other top-level +collections are optional. -- `recipe`: The action to perform. Defined by one of: +The following complete example shows every top-level section: - - `command`: A single shell command string. May contain `{{ ins }}` - (space-separated inputs) and `{{ outs }}` (space-separated outputs). These - specific placeholders are substituted *after* Jinja rendering but *before* - hashing the action. All other Jinja interpolations happen first. The final - command must be parseable by `shlex` (POSIX mode). + - - `script`: A multi-line script (using YAML `|`). If it starts with `#!`, - it's executed directly. Otherwise, it's run via `/bin/sh -e` (or PowerShell - on Windows) by default. Interpolated variables are automatically - shell-escaped unless `| raw` is used. +```yaml +netsuke_version: "1.0.0" - - `rule`: References another rule by name (less common within a rule - definition). +vars: + greeting: Hello -- `description` (Optional): A user-friendly message printed by Ninja when - this rule runs. Can contain `{{ ins }}` / `{{ outs }}`. +macros: + - signature: "message(name)" + body: | + {{ greeting }}, {{ name }}! -## 5\. Defining Targets and Actions +rules: + - name: write_message + command: "echo '{{ message('Netsuke') }}' > {{ outs }}" + description: "Writing greeting" -Targets define *what* to build or *what action* to perform. +actions: + - name: greet + command: "echo '{{ message('builder') }}'" -```yaml targets: - # Example 1: Building an object file using a rule - - name: build/utils.o # Output file(s). Can be a string or list. - rule: compile # Rule to use (mutually exclusive with command/script) - sources: src/utils.c # Input file(s). String or list. - deps: # Implicit dependencies: trigger rebuilds but - # not passed to $in/{{ ins }} - - build/utils.h - vars: # Target-local variables, override globals - cflags: "-O0 -g" - - # Example 2: Linking an executable using an inline command - - name: my_app - command: "{{ cc }} build/main.o build/utils.o -o my_app" - sources: # Explicit recipe inputs: passed to - # $in/{{ ins }} and trigger rebuilds - - build/main.o - - build/utils.o - order_only_deps: # Dependencies built before, but changes - # do not trigger rebuild - - build_directory # e.g., Ensure 'build/' exists - - # Example 3: A phony action (can also be in top-level 'actions:') - - name: package - phony: true # Doesn't represent a file, always considered out-of-date - command: "tar czf package.tar.gz my_app" - deps: my_app # Depends on the 'my_app' target - - # Example 4: A target that always runs - - name: timestamp - always: true # Runs every time, regardless of inputs/outputs - command: "date > build/timestamp.txt" + - name: greeting.txt + rule: write_message +defaults: + - greeting.txt ``` -- `name`: Output file path(s). Can be a string or a list (`StringOrList`). - -- `recipe`: How to build the target. Defined by one of `rule`, `command`, or - `script` (mutually exclusive). - -- `sources`: Input file path(s) (`StringOrList`). Sources are explicit recipe - inputs: they are passed to `$in` and `{{ ins }}` and trigger rebuilds when - changed. - -- `deps` (Optional): Implicit target dependencies (`StringOrList`). Changes - trigger rebuilds, but these paths are not passed to `$in` or `{{ ins }}`. - Maps to Ninja `|`. +The top-level fields are: -- `order_only_deps` (Optional): Dependencies that must run first but whose - changes don't trigger rebuilds (`StringOrList`). Maps to Ninja `||`. +- `netsuke_version`: required semantic version for the manifest schema. +- `vars`: global strings, numbers, booleans or lists available to Jinja. +- `macros`: named Jinja macro definitions registered before other fields + render. +- `rules`: reusable recipes referenced by targets or actions. +- `actions`: implicitly phony operations, such as `test` or `lint`. +- `targets`: required list of file-producing or logical build nodes. An empty + list is valid for an action-only manifest. +- `defaults`: target or action names used when `build` receives no explicit + targets. -Cycle detection traverses `sources` and `deps`, because both classes affect the -build graph and rebuild freshness. `order_only_deps` only enforce build -ordering and do not participate in Netsuke's cycle detection. +`defaults` entries are literal names in v0.1.0; Jinja expressions are not +rendered in this field. -- `vars` (Optional): Target-specific variables that override global `vars`. +### Rules and recipes -- `phony` (Optional, default `false`): Treat target as logical, not a file. - Always out-of-date if requested. +A rule or target must provide exactly one recipe: -- `always` (Optional, default `false`): Re-run the command every time - `netsuke` is invoked, regardless of dependency changes. +- `command`: one shell command. +- `script`: a multi-line POSIX shell script. +- `rule`: the name of another rule to use. -`StringOrList`: Fields like `name`, `sources`, `deps`, and `order_only_deps` -accept either a single string or a YAML list of strings for convenience. +Rules may also provide: -## 6\. Jinja Templating in Netsuke +- `description`: text used for Ninja's progress display. +- `deps`: dependencies inherited by targets that use the rule. -Netsuke uses the [MiniJinja](https://docs.rs/minijinja "null") engine to add -dynamic capabilities to your manifest. +The v0.1.0 `script` implementation invokes `/bin/sh -e`; it is not currently a +portable PowerShell abstraction. Prefer `command` or platform-selected actions +when a manifest must work on Windows. -### Basic Syntax +### Targets, inputs and dependencies -- Variables: `{{ my_variable }}` +A target supports these fields: -- Expressions: `{{ 1 + 1 }}`, `{{ sources | map('basename') }}` +- `name`: one output path or a list of output paths. +- `rule`, `command` or `script`: exactly one recipe. +- `sources`: explicit inputs. They affect freshness and become `{{ ins }}`. +- `deps`: implicit dependencies. They affect freshness but do not become + recipe arguments. +- `order_only_deps`: ordering dependencies. Their changes do not rebuild the + dependant target. +- `vars`: values that override global variables for this target. +- `phony`: marks a logical target that does not represent a file. +- `always`: forces the recipe to run whenever the target is requested. -- Control Structures (within specific keys like `foreach`, `when`, or inside - `macros`): `{% if enable %}…{% endif %}`, - `{% for item in list %}…{% endfor %}` +`name`, `sources`, `deps` and `order_only_deps` accept either one string or a +list of strings. -**Important:** Structural Jinja (`{% %}`) is generally **not** allowed directly -within the YAML structure outside of `macros`. Logic should primarily be within -string values or dedicated keys like `foreach` and `when`. +Netsuke quotes paths inserted through `{{ ins }}` and `{{ outs }}`. Other Jinja +values render as ordinary command text and are not automatically shell-quoted. +The `shell_escape` filter described in older drafts is not implemented in +v0.1.0. -### Processing Order +Cycle detection follows `sources` and `deps`. Order-only dependencies enforce +ordering but do not participate in cycle detection. -Netsuke processes the manifest in stages: +## Use Jinja safely -1. Initial YAML Parse: Load the raw file into an intermediate structure (like - `serde_json::Value`). +Jinja expressions are allowed in renderable string fields, including variables, +target fields and rule recipes. Structural Jinja blocks cannot reshape the YAML +document. Use the dedicated `foreach` and `when` keys for manifest-time +expansion. -2. Template Expansion (`foreach`, `when`): Evaluate `foreach` expressions to - generate multiple target or action definitions. The `item` (and optional - `index`) become available in the context. Evaluate `when` expressions to - conditionally include/exclude entries (`when` can exclude both targets and - actions). This is a manifest-time selection step; skipped entries are - removed before Netsuke builds the typed manifest AST, creates its IR, emits - `build.ninja`, or runs Ninja. +### Generate targets with `foreach` and `when` -3. Deserialization to AST: Convert the expanded intermediate structure into - Netsuke's typed Rust structs (`NetsukeManifest`, `Target`, etc.). +The next complete manifest creates two targets and excludes the disabled item: -4. Final Rendering: Render Jinja expressions **only within string fields** - (like `command`, `description`, `name`, `sources` etc.) using the combined - context (globals + target vars + iteration vars). - -### `foreach` and `when` - -These keys enable generating multiple similar targets or actions -programmatically. + ```yaml -targets: - # Generate a target for each .c file in src/ - - foreach: glob('src/*.c') # Jinja expression returning an iterable - # Only include if the filename isn't 'skip.c' - when: item | basename != 'skip.c' - # 'item' and 'index' (0-based) are available in the context - name: "build/{{ item | basename | with_suffix('.o') }}" - rule: compile - sources: "{{ item }}" - vars: - compile_flags: "-O{{ index + 1 }}" # Example using index +netsuke_version: "1.0.0" -``` +vars: + reports: + - daily + - weekly + - disabled -```yaml -actions: - # Generate named test actions, skipping disabled suites. - - foreach: - - unit - - integration - - disabled +targets: + - foreach: reports when: item != 'disabled' - name: "test-{{ item }}" - command: "cargo test --test {{ item }}" + name: "{{ item }}.txt" + command: "echo {{ index }} > {{ outs }}" + +defaults: + - daily.txt + - weekly.txt ``` -- `foreach`: A Jinja expression evaluating to a list (or any iterable). A - target or action definition will be generated for each item. +Each expansion receives `item` and a zero-based `index`. Target-local variables +take precedence over global variables, and iteration values take precedence +over both. -- `when` (Optional): A Jinja expression evaluating to a boolean. If false, - the target or action generated for the current `item` is skipped. +`when` is evaluated while Netsuke loads the manifest. It does not create a +runtime branch. Runtime decisions belong in a command or script. -`foreach` and `when` do not create build-time branches. They decide which -manifest entries exist while Netsuke loads the manifest. The generated Ninja -file contains only the selected targets and actions, so a skipped entry cannot -contribute rules, outputs, dependencies, defaults, or command text later in the -build pipeline. +Top-level actions support the same `foreach` and `when` keys. -When a decision must happen while a target is being built, put that branching -inside the recipe command or script: +### Define reusable macros -```yaml -targets: - - name: report - script: | - if test -f report.in; then - ./render report.in > report - else - ./render-empty > report - fi -``` +Macros return rendered text and can accept default arguments: -A future runtime-condition feature may model build-time branching directly, but -current manifests should treat recipe commands and scripts as the build-time -decision point. + -### User-defined Macros +```yaml +netsuke_version: "1.0.0" -Define reusable Jinja logic in the top-level `macros` section. +vars: + greeting: Hello -```yaml macros: - - signature: "cc_cmd(src, obj, flags='')" # Jinja macro signature - body: | # Multi-line body - {{ cc }} {{ flags }} -c {{ src | shell_escape }} \ - -o {{ obj | shell_escape }} + - signature: "say(name, punctuation='!')" + body: | + {{ greeting }}, {{ name }}{{ punctuation }} targets: - - name: build/main.o - command: "{{ cc_cmd('src/main.c', 'build/main.o', flags=cflags) }}" - sources: src/main.c + - name: greeting.txt + command: "echo '{{ say('Netsuke') }}' > {{ outs }}" +defaults: + - greeting.txt ``` -## 7\. Netsuke Standard Library (Stdlib) +### Select optional tools -Netsuke provides a rich set of built-in Jinja functions, filters, and tests to -simplify common build tasks. These are automatically available in your manifest -templates. +`which(name, **kwargs)` returns an executable path and fails when the command +is absent. The same helper is also available as a filter. -### Key Functions +`command_available(name, **kwargs)` returns a boolean and is better for +complementary branches: -- `env(name, default=None)`: Reads an environment variable. Fails if `name` - is unset and no `default` is provided. Example: `{{ env('CC', 'gcc') }}` + -- `glob(pattern)`: Expands a filesystem glob pattern into a sorted list of - *files* (directories are excluded). Handles `*`, `**`, `?`, `[]`. - Case-sensitive. Example: `{{ glob('src/**/*.c') }}` - -- `fetch(url, cache=False)`: Downloads content from a URL. If `cache=True`, - caches the result in `.netsuke/fetch` within the workspace based on URL hash. - Enforces a configurable maximum response size (default 8 MiB); requests abort - with an error quoting the configured threshold when the limit is exceeded. - Cached downloads stream directly to disk and remove partial files on error. - Configure the limit with `StdlibConfig::with_fetch_max_response_bytes`. Marks - template as impure. - -- `now(offset=None)`: Returns the current time as a timezone-aware object - (defaults to UTC). `offset` can be '+HH:MM' or 'Z'. Exposes `.iso8601`, - `.unix_timestamp`, `.offset`. - -- `timedelta(...)`: Creates a duration object (e.g., for age comparisons). - Accepts `weeks`, `days`, `hours`, `minutes`, `seconds`, `milliseconds`, - `microseconds`, `nanoseconds`. Exposes `.iso8601`, `.seconds`, `.nanoseconds`. - -### Key Filters +```yaml +netsuke_version: "1.0.0" -Apply filters using the pipe `|` operator: `{{ value | filter_name(args...) }}` +actions: + - name: test-fast + command: "cargo nextest run" + when: command_available("cargo-nextest") -**Path & File Filters:** + - name: test-fast + command: "cargo test" + when: not command_available("cargo-nextest") -- `basename`: `{{ 'path/to/file.txt' | basename }}` -> `"file.txt"` +targets: [] -- `dirname`: `{{ 'path/to/file.txt' | dirname }}` -> `"path/to"` +defaults: + - test-fast +``` -- `with_suffix(new_suffix, count=1, sep='.')`: Replaces the last `count` - dot-separated extensions. `{{ 'archive.tar.gz' | with_suffix('.zip', 2) }}` -> - `"archive.zip"` +Both helpers accept: -- `relative_to(base_path)`: Makes a path relative. - `{{ '/a/b/c' | relative_to('/a/b') }}` -> `"c"` +- `all=true`: return all `which` matches. It does not change the boolean result + from `command_available`. +- `canonical=true`: canonicalize matching paths. +- `fresh=true`: bypass the resolver cache for this lookup. +- `cwd_mode="auto"|"always"|"never"`: control bounded project-directory + fallback searching. -- `realpath`: Canonicalizes path, resolving symlinks. +The `env(name)` function reads one required environment variable. v0.1.0 does +not accept a default argument; an absent or non-Unicode value is an error. -- `expanduser`: Expands `~` to the home directory. +## Use the template standard library -- `contents(encoding='utf-8')`: Reads file content as a string. +Netsuke registers focused path, collection, command, network and time helpers +alongside MiniJinja's built-ins. -- `size`: File size in bytes. +### Path filters -- `linecount`: Number of lines in a text file. +The path filters are: -- `hash(alg='sha256')`: Full hex digest of file content (supports - `sha256`, `sha512`; `md5`, `sha1` if `legacy-digests` feature enabled). +- `basename` +- `dirname` +- `with_suffix(suffix[, count[, separator]])` +- `relative_to(root)` +- `realpath` +- `expanduser` +- `contents([encoding])` +- `size` +- `linecount` +- `hash([algorithm])` +- `digest([length[, algorithm]])` -- `digest(len=8, alg='sha256')`: Truncated hex digest. +`with_suffix` defaults to replacing one dot-separated suffix. `contents` +defaults to UTF-8, `hash` defaults to SHA-256, and `digest` defaults to the +first eight characters of a SHA-256 digest. Hashing supports SHA-256 and +SHA-512. MD5 and SHA-1 require the `legacy-digests` Cargo feature. -- `shell_escape`: **Crucial for security.** Safely quotes a string for use as - a single shell argument. *Use this whenever interpolating paths or variables - into commands unless you are certain they are safe.* +### Collection filters -**Collection Filters:** +Netsuke adds `uniq`, `flatten` and `group_by(attribute)`. MiniJinja also +provides general filters such as `join`, `map`, `select` and `sort`. -- `uniq`: Removes duplicate items from a list, preserving order. +This complete manifest exercises string-only helpers without depending on +external files: -- `flatten`: Flattens a nested list. `{{ [[1], [2, 3]] | flatten }}` -> - `[1, 2, 3]` + -- `group_by(attribute)`: Groups items in a list of dicts/objects by an - attribute's value. +```yaml +netsuke_version: "1.0.0" -- `map(attribute='...')` / `map('filter', ...)`: Applies attribute access or - another filter to each item in a list. +vars: + names: + - alpha + - alpha + - beta -- `filter(attribute='...')` / `filter('test', ...)`: Selects items based on - attribute value or a test function. +targets: + - name: "{{ 'report.tmp' | with_suffix('.txt') }}" + command: "echo {{ names | uniq | join(',') }} > {{ outs }}" -- `join(sep)`: Joins list items into a string. +defaults: + - report.txt +``` -- `sort`: Sorts list items. +### File tests -**Command Filters (Impure):** +Jinja `is` expressions can test `file`, `dir`, `symlink`, `pipe`, +`block_device`, `char_device` and `device`. Filesystem tests inspect the host, +so results depend on the current workspace and platform. -- `shell(command_string)`: Pipes the input value (string or bytes) as stdin - to `command_string` executed via the system shell (`sh -c` or `cmd /C`). - Returns stdout. **Marks the template as impure.** Example: - `{{ user_list | shell('grep admin') }}`. The captured stdout is limited to 1 - MiB by default; configure a different budget with - `StdlibConfig::with_command_max_output_bytes`. Exceeding the limit raises an - `InvalidOperation` error that quotes the configured threshold. Templates can - pass an options mapping such as `{'mode': 'tempfile'}` to stream stdout into - a temporary file instead. The file path is returned to the template and - remains bounded by `StdlibConfig::with_command_max_stream_bytes` (default 64 - MiB). +### Time helpers -- `grep(pattern, flags=None)`: Filters input lines matching `pattern`. - `flags` can be a string (e.g., `'-i'`) or list of strings. Implemented via - `shell`. Marks template as impure. The same output and streaming limits apply - when `grep` emits large result sets. +`now(offset=...)` returns the current timestamp, optionally at an offset such as +`"+01:00"`. `timedelta(...)` constructs a duration from keyword components: +weeks, days, hours, minutes, seconds, milliseconds, microseconds and +nanoseconds. These helpers read the clock or perform duration arithmetic; they +do not schedule work. -#### Executable discovery +### Impure helpers -The `which` filter and function resolve executables using the current `PATH` -without marking the template as impure. `{{ 'clang++' | which }}` returns the -first matching binary; `{{ which('clang++') }}` is available when a function -call is clearer than piping. Missing executables raise -`netsuke::jinja::which::not_found` with a preview of the searched directories. +The following helpers can observe or modify the outside world: -Use `command_available(name, **kwargs)` when absence should select another -manifest-time branch instead of failing the render. It uses the same resolver -and cache as `which`, returns `true` when at least one executable is found, and -returns `false` for absent commands. Invalid arguments still raise -`netsuke::jinja::which::args`. +- `fetch(url, cache=false)` retrieves a URL. HTTPS is the only allowed scheme + by default. +- `value | shell(command, options)` sends a value to a host shell command. +- `value | grep(pattern, flags, options)` filters lines through `grep`. +- `now(offset=...)` reads the clock. +- File-reading and path-canonicalization filters inspect the filesystem. -| Kwarg | Default | Effect on `command_available` | -| ----------- | ------- | ---------------------------------------------------------------------------- | -| `all` | `false` | Accepted for kwarg symmetry with `which`; does not change the bool return. | -| `canonical` | `false` | Canonicalizes discovered paths before deciding whether any match exists. | -| `fresh` | `false` | Bypasses the resolver cache for this lookup and refreshes the cached result. | -| `cwd_mode` | `auto` | Controls current-directory search: `auto`, `always`, or `never`. | +`fetch`, `shell` and `grep` enforce bounded output. These internal limits are +not currently user-configurable from `Netsukefile`. -When `PATH` is empty and `cwd_mode="auto"`, Netsuke can still discover a -project-local executable through the bounded workspace fallback: +Use impure helpers only in trusted manifests. Netsuke does not sandbox them. -```yaml -actions: - - name: lint - command: ./tools/project-lint - when: command_available("project-lint", cwd_mode="auto") - - name: lint - command: cargo clippy --workspace --all-targets --all-features -- -D warnings - when: not command_available("project-lint", cwd_mode="auto") -``` +## Use the command-line interface -The `which` filter remains the right choice when a missing tool should stop the -manifest render. The predicate is for optional toolchains and complementary -branches. +The top-level command shape is: -Use `command_available` in manifest-time `when` clauses when optional tooling -selects between actions: + -```yaml -actions: - - name: test-fast - command: cargo nextest run - when: command_available("cargo-nextest") - - name: test-fast - command: cargo test - when: not command_available("cargo-nextest") +```plaintext +netsuke [OPTIONS] [COMMAND] +netsuke [OPTIONS] build [TARGETS]... ``` -Only the selected action reaches the typed manifest and generated Ninja file. -Top-level actions selected this way still keep the normal implicit -`phony: true` behaviour. +Global options must appear before the subcommand. For example, +`netsuke --colour-policy always build` is valid; +`netsuke build --colour-policy always` is not. -**Impurity:** Filters like `shell` and functions like `fetch` interact with the -outside world. Netsuke tracks this "impurity". Impure templates might affect -caching or reproducibility analysis in future versions. Use impure helpers -judiciously. +The commands are: -### Key Tests +- `build [TARGETS]...`: generate Ninja and build the named targets. With no + targets, use configured defaults and then manifest defaults. +- `clean`: generate a temporary Ninja file and run `ninja -t clean`. +- `graph`: render the build graph as DOT or self-contained HTML without + invoking Ninja. +- `manifest `: write Ninja without invoking it. Use `-` for stdout. -Use tests with the `is` keyword: `{% if path is file %}` +Running `netsuke` without a subcommand is the same as `netsuke build` with no +explicit targets. A bare target such as `netsuke hello` is not accepted; use +`netsuke build hello`. -- `file`, `dir`, `symlink`: Checks filesystem object type (without following - links). +Important global options include: -- `readable`, `writable`, `executable`: Checks permissions for the current - user. +- `-f, --file ` +- `-C, --directory ` +- `--config ` +- `-j, --jobs <1..64>` +- `-v, --verbose` +- `--locale ` +- `--accessible ` +- `--progress ` +- `--theme ` +- `--colour-policy ` +- `--spinner-mode ` +- `--diag-json` +- `--default-target ` -- `absolute`, `relative`: Checks path type. +Run `netsuke --help` or `netsuke --help` for the complete current +surface. -## 8\. Command-Line Interface (CLI) +### Anchor a project with `--directory` -Netsuke's CLI provides commands to manage your build. +`--directory` changes manifest lookup, project configuration discovery and +relative output paths: -```text -netsuke [OPTIONS] [COMMAND] [TARGETS...] + +```sh +netsuke --directory /path/to/project build ``` -### Global Options +An explicit `--config` path remains relative to the shell's original working +directory. -- `-f, --file `: Path to the `Netsukefile` (default: `Netsukefile`). +### Generate and inspect artefacts -- `-C, --directory `: Change to directory `DIR` before doing anything. +These commands cover the non-default utility workflows: -- `-j, --jobs `: Set the number of parallel jobs Ninja should run - (default: Ninja's default). + -- `-v, --verbose`: Enable verbose diagnostic logging and completion timing - summaries. - -- `--locale `: Localize CLI help and error messages (for example - `en-US` or `es-ES`). +```sh +netsuke clean +netsuke graph --output build.dot +netsuke graph --html --output graph.html +netsuke manifest - +netsuke build --emit build.ninja +``` -### Network Policy Options +`graph` is rendered in-process and does not require Ninja. DOT goes to stdout +unless `--output` is supplied. HTML output contains a server-rendered SVG, a +textual outline and a `