Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .claude/skills/fullsend/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@ For user-facing fullsend commands (trigger, inspect, watch, help), use the upstr

## Essential Principles

1. **Customized files are full replacements.** When a repo places a file in `.fullsend/customized/harness/`, fullsend uses it *instead of* the scaffold version — not merged, not overlaid. Any field omitted from the customized file is silently dropped.
2. **Upstream scaffold is source of truth.** The canonical harness and env definitions live in the scaffold repo. Customizations must track upstream changes or they drift.
3. **Always diff before deploying.** Never commit a customized harness without comparing it field-by-field against the current upstream version.
1. **`base:` composition, not `customized/` overlay.** ADR 0064 removed `.fullsend/customized/`. Repo-owned agents live under `.fullsend/rhdh/` and register in `.fullsend/config.yaml`. Harnesses inherit upstream via `base:` and override only the fields that differ.
2. **`repos.yaml` is the fleet source of truth.** Version pins, mint URL, and the managed repo list live at the root of this repo. Roll a new fullsend version by bumping `github.fullsend_ref` and running `fullsend repos install -f repos.yaml` (PRs only, never `--direct`).
3. **Always diff before deploying.** Thin `base:` harnesses still need a field-by-field check against the upstream file they inherit.
4. **Docker ENV is dead at runtime.** OpenShell strips Containerfile `ENV` directives — they exist during `docker build` but not in the sandbox. All runtime env vars must go through `.env.d/` files.
5. **Path flattening.** Fullsend overlays `.fullsend/customized/env/` → `env/`, `.fullsend/customized/harness/` → `harness/`, etc. Harness `host_files.src` paths are always relative to the **flattened** working dir (e.g., `env/foo.env`, never `customized/env/foo.env`).
6. **Confirm before mutating.** Commands that write to GitHub (`comment`, `label`) must confirm with the user before acting. These are shared-state actions visible to the whole team.
5. **Harness paths are relative to `.fullsend/`.** `host_files.src` values such as `rhdh/env/yarn-proxy.env` resolve from the per-repo `.fullsend/` root, not from a flattened `customized/` overlay.
6. **Confirm before mutating.** Commands that write to GitHub (`comment`, `label`) must confirm with the user before acting. These are shared-state actions visible to the whole team. Scaffold upgrades go through PRs.

</essential_principles>

Expand Down Expand Up @@ -91,7 +91,7 @@ To add a variable, create an env file and wire it via `host_files` in the harnes
| `debug <#issue> [--repo]` | Run sandbox diagnostics (shortcut for `trigger debug`) |
| `comment <#issue> <message> [--repo]` | Post a comment on an issue or PR |
| `label <#issue> <add\|remove> <label> [--repo]` | Add or remove a label on an issue or PR |
| `upgrade [version]` | Upgrade CLI, scaffold files, and dispatch workflows to a new fullsend release |
| `upgrade [version]` | Upgrade CLI, bump `repos.yaml`, and converge target repos via PRs |
| `custom-agents` | Guide for building custom standalone agents (scaffold, dispatch, security) |
| `local-setup` | Guide for running fullsend agents locally on a Mac |

Expand Down Expand Up @@ -215,7 +215,7 @@ agent-<agent>-<issue>-<timestamp>/

## Sandbox Image

The custom image (`ghcr.io/redhat-developer/rhdh-fullsend-code:latest`) is built from `images/code/Containerfile` in rhdh-fullsend. Auto-builds on push to main when `images/code/**` changes.
The custom image (`ghcr.io/redhat-developer/rhdh-fullsend-code:latest`) is built from `images/code/Containerfile` in rhdh-fullsend. Auto-builds on push to `main` when `images/code/**` or `repos.yaml` changes (a `fullsend_ref` bump rebuilds so `:latest` picks up the new upstream `fullsend-code` base). `workflow_dispatch` remains for an out-of-band rebuild.

### What's in it (on top of upstream fullsend-code)

Expand Down
200 changes: 52 additions & 148 deletions .claude/skills/fullsend/references/upgrade.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,43 @@
# upgrade

Upgrade customized scaffold files, dispatch workflows, and the CLI binary
to a new fullsend release. Produces one PR per repo with all changes.
Upgrade the fullsend CLI, then roll the scaffold workflow ref through
`repos.yaml`. Every forge change lands as a PR — never `--direct`.

## Usage

```
/fullsend upgrade [target-version]
```

- `target-version`: e.g. `v0.18.0`. Default: latest release on fullsend-ai/fullsend.
- `target-version`: e.g. `v0.37.0`. Default: latest release on fullsend-ai/fullsend.

## Prerequisites

| Gate | Check | If fail |
|------|-------|---------|
| Scaffold dir | `$FULLSEND_SCAFFOLD_DIR` or `../asdlc-lab/resources/fullsend-ai/fullsend/` | Ask user to clone `asdlc-lab` |
| `repos.yaml` | this repo's fleet manifest | Stop — the file is required |
| `gh` CLI | `gh auth status` | Ask user to authenticate |
| `fullsend` CLI | `fullsend --version` | Suggest downloading from GitHub releases |
| `fullsend` CLI | `fullsend --version` | Download from GitHub releases |

Official path: [Rolling out a new fullsend version](https://fullsend.sh/docs/guides/getting-started/repo-management#rolling-out-a-new-fullsend-version).

## Procedure

### 1. Determine versions

```bash
# Current CLI version
fullsend --version

# Fetch latest tags from upstream
cd <fullsend-repo> && git fetch --tags

# Latest upstream release
gh release list --repo fullsend-ai/fullsend --limit 3

# Current version from our customized files (read forked-from stamp)
head -5 <target-repo>/.fullsend/customized/harness/code.yaml
grep fullsend_ref repos.yaml
```

Show the user: current CLI version, current forked-from version (from stamps),
and target version. Confirm before proceeding.
Show the user: current CLI version, current `github.fullsend_ref`, and target
version. Confirm before proceeding.

### 2. Upgrade CLI binary

```bash
VERSION=<target>
VERSION=<target> # without the leading v, e.g. 0.37.0
gh release download "v${VERSION}" --repo fullsend-ai/fullsend \
--pattern "fullsend_${VERSION}_darwin_arm64.tar.gz" -D /tmp

Expand All @@ -55,127 +49,57 @@ chmod +x ~/.local/bin/fullsend
fullsend --version
```

### 3. Generate upstream changelog

Diff the scaffold between the old and new versions to understand what changed:

```bash
OLD=v0.17.0 # from forked-from stamps
NEW=v0.18.0 # target

cd <fullsend-repo>
git log --oneline $OLD..$NEW --no-merges | head -30

# Scaffold-specific changes
git diff $OLD..$NEW -- internal/scaffold/fullsend-repo/harness/
git diff $OLD..$NEW -- internal/scaffold/fullsend-repo/agents/
git diff $OLD..$NEW -- internal/scaffold/fullsend-repo/scripts/
git diff $OLD..$NEW -- internal/scaffold/fullsend-repo/policies/
git diff $OLD..$NEW -- internal/scaffold/fullsend-repo/templates/shim-per-repo.yaml
```
### 3. Bump the fleet manifest

Present a summary of changes to the user. Classify each as:
- **Must adopt**: path changes, new required fields, security fixes
- **Should adopt**: new optional fields, improved defaults
- **Informational**: script changes (inherited automatically if not customized)
In `repos.yaml`, set `github.fullsend_ref` to the target tag (e.g. `v0.37.0`).
Open a PR in **this** repo (`rhdh-fullsend`). Do not push to `main`.

### 4. Per-repo upgrade
Keep `github.mint_mode: private` and the self-hosted `mint_url`. Do not let
setup/install fall through to `https://mint.fullsend.sh`.

For each repo with `.fullsend/customized/`:
### 4. Converge target repos (PRs only)

#### 4a. Sync fork
Dry-run first, then install **without** `--direct`:

```bash
git fetch upstream
git checkout main
git merge upstream/main --ff-only
git push origin main
fullsend repos install -f repos.yaml --dry-run
fullsend repos install -f repos.yaml redhat-developer/rhdh-agentic
```

If ff-merge fails, the fork has diverged. Investigate before proceeding.
Omit the repo filter to converge every entry in the manifest.

#### 4b. Diff all customized files
`repos install` on an already-installed repo:

```bash
UPSTREAM="<fullsend-repo>/internal/scaffold/fullsend-repo"
OURS="<target-repo>/.fullsend/customized"

for f in $(find "$OURS" -type f ! -name ".gitkeep" | sort); do
rel="${f#$OURS/}"
upstream_file="$UPSTREAM/$rel"
echo "--- $rel ---"
if [ -f "$upstream_file" ]; then
diff -u "$upstream_file" "$f" | head -40
else
echo "(custom — no upstream equivalent)"
fi
done
```

For each file, record a decision:
- **ADOPT**: apply upstream change to our customized file
- **SKIP**: upstream changed but our customization is intentional — no action
- **STAMP**: no changes needed — just update the version stamp

Present the decision table to the user before making changes.
- Refreshes `.github/workflows/fullsend.yaml` and thin callers (`prioritize.yml`)
- Upgrades the `reusable-dispatch.yml` pin (`@<sha> # vX.Y.Z` stays SHA-pinned)
- Reconciles mint URL / region variables from the manifest
- **Does not rewrite** `.fullsend/config.yaml` (custom `agents:` stay)
- **Does not delete** leftover `.fullsend/customized/` trees — remove those
in the same scaffold PR if they are empty `.gitkeep` placeholders (ADR 0064)

#### 4c. Apply changes
Review each scaffold PR before merge. Do not use `--direct`.

Create a single branch per repo. Include ALL changes in one PR:
- Scaffold file updates (harness, agents, policies)
- Dispatch workflow sync (`.github/workflows/fullsend.yaml`)
- Version stamp updates on every customized file
### 5. Rebuild sandbox image if needed

**Important constraints:**
- Workflow files (`.github/workflows/`) need human push — the fs-code agent
token lacks `workflows` permission. Always include these in the manual PR.
- Agent prompt `.md` files may reference harness paths (e.g.,
`/sandbox/workspace/prior-review.txt`). When harness `host_files[].dest`
paths change, grep agent prompts for the old paths.

#### 4d. Update version stamps

Every customized file gets updated stamps:
```yaml
# forked-from: fullsend v0.18.0 scaffold
# last-synced: 2026-07-15
```
Check whether the upstream base image changed:

For custom files with no upstream equivalent:
```yaml
# forked-from: custom (no upstream equivalent)
# last-synced: 2026-07-15
```

### 5. Rebuild sandbox image

Check if the upstream base image versions changed:
```bash
git diff $OLD..$NEW -- images/code/Containerfile
git diff $OLD..$NEW -- images/sandbox/Containerfile
git -C /Users/mhild/src/fullsend-ai/fullsend diff $OLD..$NEW -- images/code/Containerfile
```

Our Containerfile extends `ghcr.io/fullsend-ai/fullsend-code:latest`.
Tool upgrades (Go, gopls, tirith) come from the base image automatically.

Only change our Containerfile if:
- The pinned yarn version changed in the target repo's `package.json`
- We need to add/remove tools (e.g., openspec)

To rebuild:
```bash
podman build -t rhdh-fullsend-code:local \
-f images/code/Containerfile images/code/
```
Our Containerfile extends `ghcr.io/fullsend-ai/fullsend-code:latest`. Tool
upgrades come from the base image automatically. Only change our Containerfile
if the pinned yarn version changed or we need to add/remove tools.

CI auto-builds on push to main when `images/code/**` changes. If no
Containerfile changes are needed, trigger manually via `workflow_dispatch`
on the sandbox-images workflow to pick up the new base.
CI auto-builds on push to `main` when `images/code/**` **or** `repos.yaml`
changes. Bumping `github.fullsend_ref` is therefore enough to pick up the new
upstream `fullsend-code` base — no Containerfile change and no manual
`workflow_dispatch` required. Keep `workflow_dispatch` for an out-of-band
rebuild (e.g. upstream published a new `:latest` without a ref bump).

### 6. Smoke test

After PRs are merged, create a test issue on rhdh-agentic to verify
the agent pipeline works with the upgraded scaffold and image.
After the rhdh-agentic scaffold PR is merged, create a test issue:

```bash
gh issue create --repo redhat-developer/rhdh-agentic \
Expand All @@ -185,37 +109,17 @@ Expected: triage agent picks up this issue, classifies it, posts status comment.
Close this issue if triage succeeds."
```

The `issues: opened` event auto-triggers triage. Watch the run:
```bash
/fullsend watch rhdh-agentic issue <N>
```

What to check:
- Route job succeeds (dispatch workflow changes work)
- Triage sandbox starts (image pulls correctly)
- No credential or path errors in logs
- Status comment posted on the issue

If triage succeeds, close the issue:
```bash
gh issue close <N> --repo redhat-developer/rhdh-agentic \
--comment "Smoke test passed — triage ran successfully on <version>."
```

If it fails, inspect with `/fullsend inspect` and check the logs for
path mismatches, credential delivery failures, or toolchain errors.
Watch the run. If triage succeeds, close the issue.

## Known gotchas

1. **Stale local checkout**: always `git fetch --tags` before diffing.
2. **Silent path breakage**: mount path changes silently fail — files mount
but the agent can't find them at the old path.
3. **Agent prompts reference harness paths**: grep `.md` files for old paths
when `host_files[].dest` changes.
4. **Concurrency group cancellation**: `/fs-code` runs can be cancelled by
triage bot comments. Re-trigger after triage completes.
5. **Agent skips branch creation**: for trivial changes, add explicit branch
instructions in the `/fs-code` comment.
6. **Fork sync before branching**: stale forks cause phantom diffs in PRs.
7. **Workflow files need `workflows` token scope**: the fs-code agent cannot
push `.github/workflows/` changes. Include in the manual PR.
1. **Self-hosted mint.** Manifest `mint_mode` must stay `private` with the GCP
mint URL. `github setup` without `--mint-url` writes `mint.fullsend.sh`.
2. **Config-targeting flags rewrite `config.yaml`.** `--runtime`, `--agents`,
`--mint-url`, `--inference-*` re-serialize the overlay (comments lost,
agents kept). Prefer `repos install` over `github setup` for upgrades.
3. **Empty `customized/` dirs are leftover.** ADR 0064 removed the overlay;
custom agents live under `.fullsend/rhdh/` with `base:` composition.
4. **Workflow files need `workflows` token scope.** The fs-code agent cannot
push `.github/workflows/` — that is why upgrades go through `repos install` PRs.
5. **Do not `--direct`.** All scaffold and manifest changes land as PRs.
2 changes: 1 addition & 1 deletion .claude/skills/fullsend/scripts/command-metadata.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
"argumentHint": "<#issue> <add|remove> <label> [--repo owner/name]"
},
"upgrade": {
"description": "Upgrade CLI, scaffold files, and dispatch workflows to a new fullsend release.",
"description": "Upgrade CLI, bump repos.yaml, and converge target repos via PRs.",
"argumentHint": "[version]"
}
}
4 changes: 4 additions & 0 deletions .github/workflows/sandbox-images.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,13 @@ on:
- main
paths:
- "images/code/**"
# Bumping github.fullsend_ref rebuilds so :latest picks up the new
# upstream fullsend-code base (FROM ghcr.io/fullsend-ai/fullsend-code:latest).
- "repos.yaml"
pull_request:
paths:
- "images/code/**"
- "repos.yaml"
workflow_dispatch:

# Cancel in-progress runs for the same branch/PR.
Expand Down
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ skill for the RHDH team's agent infrastructure.

| Component | Purpose |
|-----------|---------|
| **`repos.yaml`** | Fleet manifest for per-repo fullsend installs (version pin, mint, repo list) |
| **Sandbox image** | Extends upstream `fullsend-code` with corepack + yarn for JS monorepos |
| **Deployment docs** | GCP setup, repo onboarding, sandbox networking, known issues |
| **`/fullsend` skill** | RHDH-specific Claude Code skill for validating configs, debugging sandboxes, and building custom agents |
Expand All @@ -22,6 +23,7 @@ New to fullsend? Start here:

| Doc | What it covers |
|-----|---------------|
| [`repos.yaml`](repos.yaml) | Fleet manifest — bump `github.fullsend_ref` then `fullsend repos install -f repos.yaml`. Merging a pin bump also rebuilds `rhdh-fullsend-code`. |
| [GCP Infrastructure](docs/gcp-infrastructure.md) | GCP project, WIF providers, IAM, service accounts |
| [Sandbox Networking](docs/sandbox-networking.md) | DNS inside OpenShell sandboxes — why it fails, workarounds |

Expand Down Expand Up @@ -67,14 +69,14 @@ ghcr.io/fullsend-ai/fullsend-code:latest (upstream)
| `X.Y` | Tag push `v*` | Floating minor for auto-patch |
| `<sha>` | Every non-PR build | Debugging and rollback |

PRs build but don't push (validation only).
PRs that touch `images/code/**` or `repos.yaml` build but don't push (validation only). Merging a `fullsend_ref` bump to `main` rebuilds and pushes `:latest`.

## Usage

Reference in your fullsend harness config:

```yaml
# .fullsend/customized/harness/code.yaml
# .fullsend/rhdh/harness/code.yaml
image: ghcr.io/redhat-developer/rhdh-fullsend-code:latest
```

Expand Down
24 changes: 24 additions & 0 deletions repos.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Fleet manifest for RHDH fullsend per-repo installations.
# https://fullsend.sh/docs/guides/getting-started/repo-management
#
# Roll out a new fullsend version (creates PRs, never pushes to default):
# 1. Bump github.fullsend_ref below
# 2. fullsend repos install -f repos.yaml
# 3. Review and merge the scaffold PRs in each target repo
#
# Converge a single repo:
# fullsend repos install -f repos.yaml redhat-developer/rhdh-agentic
version: 1
defaults:
runtime: claude
allowed_remote_resources:
- https://raw.githubusercontent.com/fullsend-ai/fullsend/
- https://raw.githubusercontent.com/fullsend-ai/agents/
github:
mint_mode: private
mint_url: https://fullsend-mint-gljhbkcloq-uc.a.run.app
fullsend_ref: v0.37.0
repos:
- name: redhat-developer/rhdh-agentic
- name: redhat-developer/rhdh-plugins
- name: redhat-developer/rhdh-plugin-export-overlays