Skip to content

docs(#1102): document harness host_files execution order constraints #1103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fullsend-ai-coder wants to merge 2 commits into
base: main
Choose a base branch
from
+86 −34
Open

docs(#1102): document harness host_files execution order constraints #1103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
68d854b
docs(#1102): document harness host_files execution order constraints
fullsend-ai-coder[bot] Aug 31, 2026
d72eac4
docs: align phase count and overlay timing across repo
fullsend-ai-coder[bot] Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 25 additions & 25 deletions .skillsaw-baseline.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,8 @@
{
"version": "1",
"generated_by": "skillsaw 0.18.0",
"generated_at": "2026-08-31T00:54:48.875560+00:00",
"generated_at": "2026-08-31T17:40:16.011849+00:00",
"violations": [
{
"fingerprint": "34efa8521ae07039",
"rule_id": "agentskill-name",
"file_path": "skills/github-forge/SKILL.md",
"line": 2,
"message": "Name 'github' does not match directory name 'github-forge'",
"severity": "error"
},
{
"fingerprint": "10342080fcd47b75",
"rule_id": "agentskill-name",
Expand All @@ -20,11 +12,11 @@
"severity": "error"
},
{
"fingerprint": "a44e1496f6fe3e11",
"fingerprint": "34efa8521ae07039",
"rule_id": "agentskill-name",
"file_path": "skills/issue-labels/github/SKILL.md",
"file_path": "skills/github-forge/SKILL.md",
"line": 2,
"message": "Name 'issue-labels' does not match directory name 'github'",
"message": "Name 'github' does not match directory name 'github-forge'",
"severity": "error"
},
{
Expand All @@ -43,6 +35,14 @@
"message": "Name 'issue-labels' does not match directory name 'jira'",
"severity": "error"
},
{
"fingerprint": "a44e1496f6fe3e11",
"rule_id": "agentskill-name",
"file_path": "skills/issue-labels/github/SKILL.md",
"line": 2,
"message": "Name 'issue-labels' does not match directory name 'github'",
"severity": "error"
},
{
"fingerprint": "4aecb9c5f36aa70f",
"rule_id": "agentskill-name",
Expand All @@ -63,25 +63,34 @@
"fingerprint": "c0a7ffe42338cd6f",
"rule_id": "content-weak-language",
"file_path": "skills/code-implementation/SKILL.md",
"line": 161,
"line": 205,
"message": "Weak language (vagueness): 'properly' \u2014 Remove 'properly' \u2014 describe what correct behavior looks like",
"severity": "warning"
},
{
"fingerprint": "808393ad080aa81c",
"rule_id": "content-weak-language",
"file_path": "skills/code-implementation/SKILL.md",
"line": 195,
"line": 240,
"message": "Weak language (non-actionable): 'note that' \u2014 Restructure \u2014 state the constraint directly",
"severity": "warning"
},
{
"fingerprint": "3841c25d67e33e68",
"rule_id": "context-budget",
"file_path": "skills/pr-risk-assessment/SKILL.md",
"message": "Estimated 3,775 tokens exceeds skill warn limit of 3,000",
"severity": "warning",
"value": 3775,
"baseline_mode": "ceiling"
},
{
"fingerprint": "8aba1d160bb4c413",
"rule_id": "context-budget",
"file_path": "skills/code-implementation/SKILL.md",
"message": "Estimated 11,477 tokens exceeds skill error limit of 6,000",
"message": "Estimated 12,054 tokens exceeds skill error limit of 6,000",
"severity": "error",
"value": 11477,
"value": 12054,
"baseline_mode": "ceiling"
},
{
Expand All @@ -101,15 +110,6 @@
"severity": "error",
"value": 15709,
"baseline_mode": "ceiling"
},
{
"fingerprint": "3841c25d67e33e68",
"rule_id": "context-budget",
"file_path": "skills/pr-risk-assessment/SKILL.md",
"message": "Estimated 3,775 tokens exceeds skill warn limit of 3,000",
"severity": "warning",
"value": 3775,
"baseline_mode": "ceiling"
}
]
}
9 changes: 5 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,11 +37,12 @@ eval/ Functional eval harness and default online-scoring manifests

## Architecture

Agents run inside sandboxed containers with strict filesystem, network, and binary restrictions. Each agent follows a three-phase pipeline:
Agents run inside sandboxed containers with strict filesystem, network, and binary restrictions. Each agent follows a four-phase pipeline:

1. **Pre-script** — runs on the CI runner to validate inputs and prepare the environment
2. **Sandbox** — runs the agent with restricted permissions; the agent writes code and produces structured JSON output
3. **Post-script** — runs on the runner with elevated permissions to perform forge mutations (pushing branches, creating PRs/MRs, posting comments, applying labels)
1. **Config load** — validates harness YAML and checks `host_files` entries for existence before any scripts run
2. **Pre-script** — runs on the CI runner to validate inputs and prepare the environment
3. **Sandbox** — runs the agent with restricted permissions; the agent writes code and produces structured JSON output
4. **Post-script** — runs on the runner with elevated permissions to perform forge mutations (pushing branches, creating PRs/MRs, posting comments, applying labels)

The agent never has direct write access to the repository. All mutations flow through post-scripts.

Expand Down
9 changes: 5 additions & 4 deletions docs/code.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,12 @@ See [Customizing with AGENTS.md](https://fullsend.sh/docs/guides/user/customizin

## How the agent works

The code agent follows a three-phase pipeline: pre-script, sandbox execution, post-script.
The code agent follows a four-phase pipeline: config load, pre-script, sandbox execution, post-script.

1. **Pre-script** validates inputs on the runner before sandbox creation. It also checks for open PRs linked to the issue.
2. **Sandbox** — the agent reads the issue, explores the codebase, writes code, runs tests and linters, and commits locally. It has restricted network access (enforced by OpenShell).
3. **Post-script** runs on the runner: it performs protected path checks, secret scanning, pre-commit checks, pushes the branch, creates the PR, and best-effort assigns the PR to a human owner (latest `/fs-code` invoker, else issue assignee, else issue author).
1. **Config load** validates harness YAML and checks `host_files` entries for existence. This runs before pre-scripts — files created by pre-scripts must use `optional: true` and dynamic paths (`${RUNNER_TEMP}`) to avoid loader rejection. See the [code-implementation skill](../skills/code-implementation/SKILL.md#harness-execution-lifecycle) for details.
2. **Pre-script** validates inputs on the runner before sandbox creation. It also checks for open PRs linked to the issue.
3. **Sandbox** — the agent reads the issue, explores the codebase, writes code, runs tests and linters, and commits locally. It has restricted network access (enforced by OpenShell).
4. **Post-script** runs on the runner: it performs protected path checks, secret scanning, pre-commit checks, pushes the branch, creates the PR, and best-effort assigns the PR to a human owner (latest `/fs-code` invoker, else issue assignee, else issue author).

This separation ensures the agent never has direct write access to the repository.

Expand Down
3 changes: 2 additions & 1 deletion harness/code.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
---
# harness/code.yaml — code agent with pre/post script pipeline.
#
# Flow: pre_script → sandbox (agent) → post_script
# Runtime flow: pre_script → sandbox (agent) → post_script
# (Config load + validation occurs before this flow — see docs/code.md.)
# pre_script : validates inputs on the runner BEFORE sandbox creation
# agent : reads the issue, implements, tests, scans, commits locally
# post_script : secret scan, pre-commit, push branch, create PR/MR, assign human
Expand Down
49 changes: 49 additions & 0 deletions skills/code-implementation/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,55 @@ fractions of the budget so they scale to any timeout value):
gitlint validation and commit immediately. A commit that fails gitlint
CI is better than no commit at all.

## Harness execution lifecycle

When modifying harness configuration files (e.g., `harness/code.yaml`),
understand the order in which the harness processes them:

1. **Config load + validation** — harness YAML is parsed and validated.
`host_files` entries are checked for existence at this stage.
2. **Pre-scripts execute** — runner-side scripts (`pre_script`) run on
the CI runner. These can create files, fetch external data, and set
up context.
3. **Sandbox setup** — the sandbox container is created and `host_files`
are copied into it.
4. **Agent execution** — the agent runs inside the sandbox.

### `host_files` timing constraint

Top-level `host_files` entries are validated at config load time (step 1).
Any file that does not yet exist will cause the harness loader to reject
the entire agent — before pre-scripts even run. Files created by

@fullsend-ai-review fullsend-ai-review Bot Aug 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] internal-consistency

The documentation states that host_files entries are validated at config load time and that files created by pre-scripts must use optional: true. However, the Jira overlay in harness/code.yaml (lines 140-142) omits optional: true on its ${RUNNER_TEMP}/jira-issue-context.json entry — intentionally removed in commit 6ca729e. Overlay host_files (gated by when conditions) are likely validated after overlay resolution and pre-script execution, not at initial config load. The documentation oversimplifies by not distinguishing between top-level and overlay host_files validation timing.

Suggested fix: Clarify the documentation to note that the config-load validation applies to top-level host_files entries, and that conditional overlay entries may be validated after overlay resolution.

pre-scripts must be marked `optional: true` so the loader does not
reject the config when the file is absent at load time.

Conditional overlay `host_files` entries (those under a `when` guard)
are resolved after overlay evaluation. These entries may have different
validation timing — see the Jira-source overlay in `harness/code.yaml`
for an example of an overlay `host_files` entry without `optional: true`.

Additionally, use dynamic path variables like `${RUNNER_TEMP}` instead
of hardcoded paths like `/tmp`. Hardcoded paths are not portable across
CI runners and may collide with other jobs.

**Correct pattern** for pre-script-generated context files:

```yaml
host_files:
- src: ${RUNNER_TEMP}/context.json
dest: /sandbox/workspace/.context.json
optional: true
```

**Incorrect pattern** — will cause loader rejection:

```yaml
# BAD: file does not exist at config load time, no optional flag
host_files:
- src: /tmp/context.json
dest: /sandbox/workspace/.context.json
```

## Process

Follow these steps in order. Do not skip steps — with one exception,
Expand Down
Loading