diff --git a/references/apigee-skills-serving/README.md b/references/apigee-skills-serving/README.md index 64ac9b60..482c78c6 100644 --- a/references/apigee-skills-serving/README.md +++ b/references/apigee-skills-serving/README.md @@ -7,11 +7,12 @@ custom MCP hosts, etc.) can discover and install. This reference implementation shows how to: -1. **Author** a skill (a `SKILL.md` + optional `scripts/`) and describe it - with a `manifest.yaml` against a locked schema. +1. **Author** a skill (a `SKILL.md` + optional `scripts/`) and describe + it with a `manifest.yaml` against a locked schema. 2. **Sign** the skill bundle with an Ed25519 key so consumers can verify integrity at install time. -3. **Upload** the signed `.skill` archive to a Google Cloud Storage bucket. +3. **Upload** the signed `.skill` archive to a Google Cloud Storage + bucket. 4. **Register** the skill (and its API hub attribute taxonomy) so it is discoverable through API hub's search and attribute filters. 5. **Install** a skill on the consumer side: search API hub, fetch the @@ -21,6 +22,20 @@ This reference implementation shows how to: The whole loop runs on standard Apigee X / hybrid plus API hub — no custom infrastructure. +Two ready-to-run agent skills wrap the reference implementation into +a one-line install for end users: + +- **`skills/skill-finder/`** — consumer-side discovery client. Queries + API hub for signed skills, verifies their Ed25519 signatures against + locally-installed per-deployment trust roots, and installs matching + skills into the agent runtime's skills directory. +- **`skills/skill-publisher/`** — author-side pipeline orchestrator. + Wraps pack → sign → upload → register into a single skill an agent + can invoke on a source skill directory. + +Both are installed by `bin/install-skill-finder.sh` (which installs +both by default). + ## Why Apigee API hub for skills? Agent skills are, structurally, just metadata-tagged content addressed @@ -48,12 +63,17 @@ references/apigee-skills-serving/ ├── pytest.ini test configuration ├── docs/ │ ├── architecture.md design overview and trust model -│ ├── publish-and-install.md end-to-end walkthrough +│ ├── provisioning.md customer-onboarding runbook (five-minute path) +│ ├── publish-and-install.md detailed end-to-end walkthrough (under-the-hood) │ └── policy-skill-catalog.md about the apigee-policy-top10 example ├── bin/ │ ├── check-prerequisites.sh pre-flight environment validator -│ ├── demo-setup.sh env export + readiness print -│ └── demo-cleanup.sh remove demo artifacts +│ ├── demo-setup.sh env export + readiness print (legacy) +│ ├── demo-cleanup.sh remove locally-installed skills +│ ├── install-skill-finder.sh one-line installer for skill-finder + skill-publisher +│ ├── install-skill-publisher.sh install skill-publisher only +│ └── provision.sh one-line customer bootstrap (APIs, bucket, +│ taxonomy, key gen, publish, trust root) ├── schema/ │ └── skill-manifest.schema.yaml locked v1 manifest schema ├── scripts/ publisher-side toolchain @@ -63,8 +83,10 @@ references/apigee-skills-serving/ │ ├── register_skill.py register the manifest with API hub │ ├── update_taxonomy.py create/update API hub attribute taxonomy │ └── common/ shared libraries (retry, IAM, schema) -├── skills/ example skills -│ ├── apigee-policy-top10/ skill that documents Apigee policy patterns +├── skills/ shipped skills +│ ├── skill-finder/ consumer-side discovery + install client +│ ├── skill-publisher/ author-side pipeline orchestrator +│ ├── apigee-policy-top10/ example skill that reports top Apigee policies │ ├── currency-converter/ minimal example │ └── weather-lookup/ minimal example ├── examples/ @@ -93,42 +115,54 @@ references/apigee-skills-serving/ 6. The roles `roles/apihub.editor` and `roles/storage.objectCreator` on the target project. -## Quickstart +## Quickstart (five minutes) + +The fast path uses two shell scripts: ```bash -# 1. Clone and enter the reference +# 1. Install skill-finder + skill-publisher on your machine. +curl -fsSL https://raw.githubusercontent.com/apigee/devrel/main/\ +references/apigee-skills-serving/bin/install-skill-finder.sh \ + | bash -s -- --runtime gemini # or --runtime opencode / antigravity + +# 2. Clone the reference and provision your GCP project. git clone https://github.com/apigee/devrel.git cd devrel/references/apigee-skills-serving +bash bin/provision.sh --project MY_PROJECT --yes +``` -# 2. Install Python dependencies into a virtualenv -python3 -m venv .venv -. .venv/bin/activate -pip install -r requirements.txt +`provision.sh` enables the required GCP APIs, creates a GCS +bucket, sets up the API hub attribute taxonomy, generates a +fresh Ed25519 signing keypair on your machine (private key +stays there), and publishes the three example skills. Every +step is idempotent — re-run any time to reconcile drift. + +Full runbook (including troubleshooting and key rotation): +[`docs/provisioning.md`](docs/provisioning.md). + +### Under the hood -# 3. Set up environment variables (edit values to match your project) -cp env.sh.example env.sh -$EDITOR env.sh -. ./env.sh - -# 4. Verify prerequisites -./bin/check-prerequisites.sh - -# 5. (One-time) Create the API hub attribute taxonomy -python3 scripts/update_taxonomy.py \ - --project "$APIHUB_PROJECT" \ - --location "$APIHUB_LOCATION" - -# 6. Pack, sign, upload, register a skill -python3 scripts/pack_skill.py skills/currency-converter /tmp/cc.skill -python3 scripts/sign_skill.py /tmp/cc.skill --key-file ./signing.key -python3 scripts/upload_skill.py /tmp/cc.skill --bucket "$GCS_BUCKET" -python3 scripts/register_skill.py \ - --project "$APIHUB_PROJECT" --location "$APIHUB_LOCATION" \ - --manifest /tmp/cc.skill +The scripts above chain the five underlying operations that +this reference exists to demonstrate: + +```bash +# For each skill: +python3 -m scripts.pack_skill --src skills/ --out /tmp/.skill +python3 -m scripts.upload_skill --zip /tmp/.skill --bucket "$GCS_BUCKET" +python3 -m scripts.sign_skill --manifest \ + --zip /tmp/.skill \ + --priv-key \ + --out +python3 -m scripts.register_skill --manifest \ + --project "$APIHUB_PROJECT" \ + --location "$APIHUB_LOCATION" ``` -Full walkthrough: [`docs/publish-and-install.md`](docs/publish-and-install.md). -Design rationale and trust model: [`docs/architecture.md`](docs/architecture.md). +Full walkthrough: +[`docs/publish-and-install.md`](docs/publish-and-install.md). + +Design rationale and trust model: +[`docs/architecture.md`](docs/architecture.md). ## Running the tests @@ -144,26 +178,41 @@ pytest -q `pipeline.sh` runs the same suite and is what apigee/devrel CI invokes nightly. -## Example skills +## Shipped skills -| Skill | Purpose | -| ---------------------- | --------------------------------------------------------------------------------------------------- | -| `currency-converter` | Minimal example. A `SKILL.md` plus a `manifest.yaml`. Useful as a copy-and-edit starting point. | -| `weather-lookup` | Minimal example demonstrating a skill with API-key-based external HTTP calls. | -| `apigee-policy-top10` | A skill that documents the ten most useful Apigee policy patterns, with a script that enumerates | -| | the policies present in your org. See [`docs/policy-skill-catalog.md`](docs/policy-skill-catalog.md). | -| `examples/apigee-proxy-skill` | A complete, production-shaped skill: 18 MCP tools, 25 Jinja2 policy templates, full manifest. | +| Skill | Purpose | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `skill-finder` | Consumer-side. Discovers, verifies, and installs signed skills from the customer's API hub. Multi-key trust root; per-deployment provisioning. | +| `skill-publisher` | Author-side. Wraps pack → sign → upload → register into a single skill an agent can invoke on a source skill directory. | +| `currency-converter` | Minimal example. A `SKILL.md` plus a `manifest.yaml`. Useful as a copy-and-edit starting point. | +| `weather-lookup` | Minimal example demonstrating a skill with API-key-based external HTTP calls. | +| `apigee-policy-top10` | A skill that documents the ten most useful Apigee policy patterns, with a script that enumerates the policies present in your org. See [`docs/policy-skill-catalog.md`](docs/policy-skill-catalog.md). | +| `examples/apigee-proxy-skill` | A complete, production-shaped skill: 18 MCP tools, 25 Jinja2 policy templates, full manifest. | ## Limitations and non-goals -- The skill registry uses API hub's standard search; ranking is keyword - overlap, not semantic. For semantic ranking, integrate a vector - search component separately. -- The publisher and consumer share an Ed25519 trust root. Key rotation - is a manual operator workflow; this reference does not implement - automatic rotation. -- Skills are sandboxed by the consumer runtime (OpenCode, agent host). - This reference does not introduce additional sandboxing on top. +- The skill registry uses API hub's standard search; ranking is + keyword overlap, not semantic. For semantic ranking, integrate + a vector search component separately. +- The trust root is **per-deployment**, not shared. Each customer + who runs `provision.sh` generates their own Ed25519 keypair on + their machine. The `apigee/devrel` repo intentionally contains + no signing key material and no signed release bundles; + skill-finder itself is trusted by provenance (TLS + `apigee` + GitHub org). See + [`docs/architecture.md#trust-model`](docs/architecture.md#trust-model) + for the full rationale. +- Key rotation is a supported operator workflow (multi-key + trust root; add-then-remove pattern with no install-time gap), + but is not automated. See + [`docs/provisioning.md#key-rotation-with-zero-downtime`](docs/provisioning.md#key-rotation-with-zero-downtime). +- The GCS bucket for signed bundles must be public-read because + skill-finder downloads via anonymous HTTPS. This is a demo + choice; production would use signed URLs or an authenticated + consumer. +- Skills are sandboxed by the consumer runtime (OpenCode, Gemini + CLI, Antigravity, custom MCP hosts). This reference does not + introduce additional sandboxing on top. ## License diff --git a/references/apigee-skills-serving/bin/demo-setup.sh b/references/apigee-skills-serving/bin/demo-setup.sh index 4f6ab456..59a4a770 100755 --- a/references/apigee-skills-serving/bin/demo-setup.sh +++ b/references/apigee-skills-serving/bin/demo-setup.sh @@ -52,9 +52,9 @@ REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" # If you re-target the demo at a different GCP project, edit # these four lines. Everything else flows from here. -readonly DEMO_APIHUB_PROJECT="apigee-product-demo" +readonly DEMO_APIHUB_PROJECT="geirs-spaces-demo" readonly DEMO_APIHUB_LOCATION="us-west1" -readonly DEMO_APIGEE_ORG="apigee-product-demo" +readonly DEMO_APIGEE_ORG="geirs-spaces-demo" readonly DEMO_KEYWORD_OVERLAP="1" # ---- Color codes ----------------------------------------- @@ -185,7 +185,14 @@ while [ $# -gt 0 ]; do esac done -# ---- Step 1: prereq check + ADC -------------------------- +# ---- Step 1: ADC ---------------------------------------- +# +# ADC is checked FIRST because check-prerequisites.sh also +# checks it and we want a specific, actionable failure line +# (`gcloud auth application-default login`) before the more +# generic prereq output. If ADC is broken, the prereq check +# would spend time on env-var checks whose answers don't +# matter until ADC is fixed. echo "[setup] Verifying Application Default Credentials..." if gcloud auth application-default print-access-token \ @@ -199,24 +206,22 @@ else exit 1 fi -if [ "$skip_preflight" -eq 0 ]; then - echo - echo "[setup] Running pre-flight checker..." - if bash "${REPO_ROOT}/bin/check-prerequisites.sh"; then - _check "All required prerequisites met" 0 - else - _check "Pre-flight failed" 1 \ - "see [prereq] lines above" - echo - echo "Fix the failing prereqs, then re-run:" - echo " ./bin/demo-setup.sh" - exit 1 - fi -else - echo "[setup] --skip-preflight given; pre-flight check skipped." -fi - # ---- Step 2: env export + persistent config -------------- +# +# We MUST export the demo env vars BEFORE running the +# pre-flight checker. The checker reads APIHUB_PROJECT, +# APIHUB_LOCATION, APIGEE_ORG, and APIGEE_SKILLS_MIN_KEYWORD_OVERLAP +# from the current process environment. In a fresh shell (the +# common case -- an operator has just opened a terminal to +# start the demo), none of those are set yet. If we ran the +# checker before the export, it would fail on 3 of 4 required +# variables and abort, even though the demo is fully +# self-configuring and would work if we let it run. +# +# The persistent config file at ~/.config/apigee-skills-demo/ +# is written here for the same reason: subprocesses launched +# without the exports still resolve the values via +# scripts/common/config.py's fallback. _export_demo_env @@ -233,6 +238,31 @@ echo "[setup] in the calling process)..." _write_demo_config_file echo +# ---- Step 3: pre-flight checker ------------------------- +# +# Runs AFTER the export so the checker sees the demo values. +# The checker still enforces the invariant on behalf of an +# operator who prefers to source env.sh manually -- it just +# won't spuriously fail when the operator relied on this +# script to configure the shell. + +if [ "$skip_preflight" -eq 0 ]; then + echo "[setup] Running pre-flight checker..." + if bash "${REPO_ROOT}/bin/check-prerequisites.sh"; then + _check "All required prerequisites met" 0 + else + _check "Pre-flight failed" 1 \ + "see [prereq] lines above" + echo + echo "Fix the failing prereqs, then re-run:" + echo " ./bin/demo-setup.sh" + exit 1 + fi +else + echo "[setup] --skip-preflight given; pre-flight check skipped." +fi +echo + # ---- Step 3: ready -------------------------------------- echo -e "${GREEN}[setup] READY.${NC}" diff --git a/references/apigee-skills-serving/bin/install-skill-finder.sh b/references/apigee-skills-serving/bin/install-skill-finder.sh new file mode 100755 index 00000000..61a53558 --- /dev/null +++ b/references/apigee-skills-serving/bin/install-skill-finder.sh @@ -0,0 +1,546 @@ +#!/usr/bin/env bash +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# install-skill-finder.sh +# +# One-line installer for the skill-finder agentic skill from +# apigee/devrel. Fetches source from a specific git ref of the +# apigee/devrel repository, creates a per-user Python venv, and +# installs into the operator's chosen agent runtime. +# +# The skill-finder skill itself is trusted by provenance -- the +# operator fetched THIS script from a known-good URL over TLS +# (raw.githubusercontent.com/apigee/devrel) and the tarball comes +# from the same origin. There are no signed release bundles and +# no sha256 pins: apigee/devrel does not publish signed release +# assets, and adding a signing infrastructure to the repo would +# require organisation-level key management that is out of scope +# for the reference implementation. +# +# Payload skills that skill-finder subsequently installs from the +# customer's own API hub ARE verified by ed25519 signature. Those +# signatures are checked against per-deployment trust roots that +# the operator populates via bin/provision.sh (bootstrap) or by +# dropping a PEM into ~/./skills/skill-finder/keys/. +# +# Usage: +# install-skill-finder.sh [--runtime opencode|gemini|antigravity] +# [--install-root ] +# [--ref ] +# [--repo ] +# [--venv-dir ] +# [--use-uv] +# [--skip-publisher] +# [--force] +# [--dry-run] +# [-h | --help] +# +# Required tools on PATH: bash >= 3.2, curl, tar, python3 >= 3.10, +# the `venv` stdlib module (or `uv` on PATH if --use-uv is passed). +# +# Default behaviour: installs BOTH skill-finder (catalog discovery +# client) AND skill-publisher (author-side publishing tool). Pass +# --skip-publisher to install only the discovery client. +# +# Python dependencies are installed into a per-user venv at +# ~/.local/share/skill-finder/venv (override with --venv-dir). This +# is necessary on distros that enforce PEP 668 (Debian 12+, +# Ubuntu 23.04+, recent macOS Homebrew) where system `pip install` +# is refused. The venv is also safer everywhere else: it isolates +# skill-finder's deps from other Python projects. The same venv is +# shared between skill-finder and skill-publisher. +# +# Exit codes: +# 0 install OK +# 1 user error (bad flag, missing dep, invalid runtime) +# 2 network failure +# 3 install-target write failure + +set -u +# We intentionally do NOT set -e. Every step checks its own exit +# code so the error message identifies which step failed. + +# =============================================================== +# Defaults +# =============================================================== + +DEFAULT_REF="main" +DEFAULT_REPO="apigee/devrel" +# Path INSIDE the devrel monorepo where the skills live. +SKILLS_SUBPATH="references/apigee-skills-serving/skills" + +# Python runtime deps needed by find_install.py, list_skills.py, +# and publish.sh (via sign_skill.py which is not shipped in the +# skill bundle -- but skill-publisher shells out to the four +# publisher Python modules that share the same deps). +PY_DEPS=(cryptography google-auth requests pyyaml) + +VENV_DIR="$HOME/.local/share/skill-finder/venv" + +# =============================================================== +# Arg parsing +# =============================================================== + +RUNTIME="" +INSTALL_ROOT="" +REF="$DEFAULT_REF" +REPO="$DEFAULT_REPO" +USE_UV=0 +SKIP_PUBLISHER=0 +FORCE=0 +DRY_RUN=0 + +usage() { + cat <<'USAGE' +Usage: + install-skill-finder.sh [--runtime opencode|gemini|antigravity] + [--install-root ] + [--ref ] + [--repo ] + [--venv-dir ] + [--use-uv] + [--skip-publisher] + [--force] + [--dry-run] + +Flags: + --runtime Runtime to install into. Auto-detected if omitted. + One of: opencode | gemini | antigravity. + --install-root

Override the runtime's default skills root. + --ref Git ref (branch, tag, or commit SHA) to fetch + source from. Default: main. + --repo GitHub repo to fetch from. Default: apigee/devrel. + --venv-dir

Per-user venv location. Default: + ~/.local/share/skill-finder/venv + --use-uv Use `uv` instead of `python3 -m venv`. + --skip-publisher Install skill-finder only (skip skill-publisher). + --force Overwrite an existing install without prompting. + --dry-run Print what would happen, do nothing. + -h, --help Show this help. +USAGE +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --runtime) RUNTIME="${2:-}"; shift 2 ;; + --install-root) INSTALL_ROOT="${2:-}"; shift 2 ;; + --ref) REF="${2:-}"; shift 2 ;; + --repo) REPO="${2:-}"; shift 2 ;; + --venv-dir) VENV_DIR="${2:-}"; shift 2 ;; + --use-uv) USE_UV=1; shift ;; + --skip-publisher) SKIP_PUBLISHER=1; shift ;; + --force) FORCE=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + -h|--help) usage; exit 0 ;; + *) + echo "[install] FATAL: unknown flag: $1" >&2 + echo "[install] run --help for usage" >&2 + exit 1 + ;; + esac +done + +log() { echo "[install] $*"; } +err() { echo "[install] $*" >&2; } + +# =============================================================== +# Step 1: detect runtime and resolve install root +# =============================================================== + +if [ -z "$RUNTIME" ]; then + # Heuristic detection. Operator can always override with --runtime. + # Detection order: + # 1. OpenCode if its skills dir exists. + # 2. Antigravity if its browser-profile marker exists. + # 3. Gemini CLI: canonical user-skills root is ~/.gemini/skills. + # 4. Fallback to opencode. + if [ -d "$HOME/.config/opencode/skills" ]; then + RUNTIME="opencode" + elif [ -d "$HOME/.gemini/antigravity-browser-profile" ]; then + RUNTIME="antigravity" + elif [ -d "$HOME/.gemini" ]; then + RUNTIME="gemini" + else + RUNTIME="opencode" + fi + log "auto-detected runtime: $RUNTIME (override with --runtime)" +fi + +case "$RUNTIME" in + opencode) + DEFAULT_INSTALL_ROOT="$HOME/.config/opencode/skills" + ;; + gemini) + # Per Gemini CLI docs (docs/cli/skills.md), user skills live at + # ~/.gemini/skills/ -- NOT ~/.gemini/config/skills/ (that's the + # Antigravity path). + DEFAULT_INSTALL_ROOT="$HOME/.gemini/skills" + ;; + antigravity) + # Antigravity's global install root is ~/.gemini/antigravity/skills/. + DEFAULT_INSTALL_ROOT="$HOME/.gemini/antigravity/skills" + ;; + *) + err "FATAL: --runtime must be opencode | gemini | antigravity (got: $RUNTIME)" + exit 1 + ;; +esac + +if [ -z "$INSTALL_ROOT" ]; then + INSTALL_ROOT="$DEFAULT_INSTALL_ROOT" +fi + +log "runtime: $RUNTIME" +log "install root: $INSTALL_ROOT" +log "source repo: $REPO" +log "source ref: $REF" +log "venv dir: $VENV_DIR" +if [ "$USE_UV" -eq 1 ]; then + log "venv tool: uv (forced via --use-uv)" +else + log "venv tool: python3 -m venv (stdlib)" +fi +if [ "$SKIP_PUBLISHER" -eq 1 ]; then + log "skills: skill-finder only (--skip-publisher set)" +else + log "skills: skill-finder + skill-publisher (default)" +fi + +# =============================================================== +# Step 2: tool preflight +# =============================================================== + +need_tool() { + local tool="$1" + if ! command -v "$tool" >/dev/null 2>&1; then + err "FATAL: required tool not on PATH: $tool" + exit 1 + fi +} + +need_tool curl +need_tool tar +need_tool python3 +if [ "$USE_UV" -eq 1 ]; then + need_tool uv +fi + +# python3 must be >= 3.10 (matches the venv skill runtimes). +PY_VER=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') +case "$PY_VER" in + 3.10|3.11|3.12|3.13|3.14|3.15|3.16|3.17|3.18|3.19) + log "python: ${PY_VER} (OK)" + ;; + *) + err "FATAL: python3 must be >= 3.10 (found: ${PY_VER})" + exit 1 + ;; +esac + +# =============================================================== +# Step 3: create the per-user venv and install runtime deps +# =============================================================== + +log "step 3: setting up Python runtime" +log " venv dir: $VENV_DIR" + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping venv create + dep install" +else + if ! mkdir -p "$(dirname "$VENV_DIR")"; then + err "FATAL: cannot create venv parent dir: $(dirname "$VENV_DIR")" + exit 3 + fi + + # Reuse an existing venv only if it's healthy (has bin/python + # AND a working pip). A half-built venv from a prior failed run + # would cascade into 'No module named pip' errors. In that case + # rebuild from scratch. + VENV_OK=0 + if [ -x "$VENV_DIR/bin/python" ]; then + if "$VENV_DIR/bin/python" -m pip --version >/dev/null 2>&1; then + VENV_OK=1 + fi + fi + if [ "$VENV_OK" -eq 0 ]; then + if [ -d "$VENV_DIR" ]; then + log " existing venv at $VENV_DIR is broken; removing and recreating" + rm -rf "$VENV_DIR" + else + log " creating venv at $VENV_DIR" + fi + if [ "$USE_UV" -eq 1 ]; then + if ! uv venv --quiet "$VENV_DIR"; then + err "FATAL: uv venv failed" + exit 1 + fi + else + if ! python3 -m venv "$VENV_DIR"; then + err "FATAL: python3 -m venv failed" + exit 1 + fi + fi + else + log " reusing existing venv (pip is healthy)" + fi + + log " installing deps: ${PY_DEPS[*]}" + if [ "$USE_UV" -eq 1 ]; then + if ! uv pip install --quiet --python "$VENV_DIR/bin/python" \ + --upgrade "${PY_DEPS[@]}"; then + err "FATAL: uv pip install failed for runtime deps" + exit 1 + fi + else + if ! "$VENV_DIR/bin/python" -m pip install --quiet --upgrade \ + "${PY_DEPS[@]}"; then + err "FATAL: pip install failed for runtime deps in venv" + exit 1 + fi + fi +fi + +# =============================================================== +# Step 4: fetch source from apigee/devrel +# =============================================================== +# +# Strategy: fetch the entire repo as a tarball at $REF and extract +# only the two skill directories we care about. This avoids +# requiring `git` on the operator's machine (curl + tar are more +# portable) and avoids fetching the whole history. +# +# GitHub tarball URL format: +# https://codeload.github.com/{repo}/tar.gz/refs/heads/{branch} +# https://codeload.github.com/{repo}/tar.gz/refs/tags/{tag} +# https://codeload.github.com/{repo}/tar.gz/{sha} +# +# We probe the tag URL first, then the branch URL, then the raw +# SHA URL. Whichever returns 200 wins. + +TARBALL_TMP=$(mktemp -t skill-finder-source-XXXXXX.tar.gz) +# Clean up the tarball on exit no matter how we got here. +trap 'rm -f "$TARBALL_TMP"' EXIT + +CANDIDATES=( + "https://codeload.github.com/${REPO}/tar.gz/refs/tags/${REF}" + "https://codeload.github.com/${REPO}/tar.gz/refs/heads/${REF}" + "https://codeload.github.com/${REPO}/tar.gz/${REF}" +) + +log "step 4: fetching source from ${REPO} at ref '${REF}'" +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping source fetch" +else + FETCHED=0 + for URL in "${CANDIDATES[@]}"; do + log " trying: $URL" + if curl -fsSL -o "$TARBALL_TMP" "$URL"; then + log " OK: fetched $(wc -c <"$TARBALL_TMP") bytes" + FETCHED=1 + break + fi + done + if [ "$FETCHED" -eq 0 ]; then + err "FATAL: could not fetch source from any of:" + for URL in "${CANDIDATES[@]}"; do err " $URL"; done + err "Check that --repo and --ref name a real ref, and that" + err "you have network access to codeload.github.com." + exit 2 + fi +fi + +# Extract into a staging dir. GitHub tarballs have a top-level +# directory named {repo-name}-{sha-or-ref}; we don't care about +# that name, just the subpath we want. +EXTRACT_TMP=$(mktemp -d -t skill-finder-extract-XXXXXX) +trap 'rm -f "$TARBALL_TMP"; rm -rf "$EXTRACT_TMP"' EXIT + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping tarball extract" +else + if ! tar -xzf "$TARBALL_TMP" -C "$EXTRACT_TMP"; then + err "FATAL: could not extract tarball" + exit 2 + fi + # Find the extracted top-level dir (there's exactly one). + TOPDIR=$(find "$EXTRACT_TMP" -mindepth 1 -maxdepth 1 -type d) + if [ -z "$TOPDIR" ] || [ ! -d "$TOPDIR/${SKILLS_SUBPATH}/skill-finder" ]; then + err "FATAL: tarball does not contain ${SKILLS_SUBPATH}/skill-finder." + err "Wrong --ref or wrong --repo? Extracted structure:" + ls -la "$EXTRACT_TMP" >&2 + exit 2 + fi + SRC_SKILL_FINDER="$TOPDIR/${SKILLS_SUBPATH}/skill-finder" + # Publisher may not exist in older refs; check separately. + if [ -d "$TOPDIR/${SKILLS_SUBPATH}/skill-publisher" ]; then + SRC_SKILL_PUBLISHER="$TOPDIR/${SKILLS_SUBPATH}/skill-publisher" + else + SRC_SKILL_PUBLISHER="" + if [ "$SKIP_PUBLISHER" -eq 0 ]; then + log " warning: ${SKILLS_SUBPATH}/skill-publisher not found at ref '$REF'" + log " installing skill-finder only" + SKIP_PUBLISHER=1 + fi + fi + # skill-publisher also needs the scripts/common/ package (same + # code the devrel publisher pipeline uses). Locate it once so + # install_one_skill can splice it in. + if [ -d "$TOPDIR/references/apigee-skills-serving/scripts/common" ]; then + SRC_COMMON_PACKAGE="$TOPDIR/references/apigee-skills-serving/scripts/common" + else + SRC_COMMON_PACKAGE="" + fi +fi + +# =============================================================== +# Step 5: install skills into the runtime's skills root +# =============================================================== + +install_one_skill() { + local skill_name="$1" + local skill_src="$2" + local target_dir="$INSTALL_ROOT/$skill_name" + + log "step 5.$skill_name: installing to $target_dir" + + # Refuse to overwrite an existing install unless --force. + if [ -d "$target_dir" ]; then + if [ "$FORCE" -eq 0 ]; then + err "FATAL: $target_dir already exists (pass --force to overwrite)" + exit 3 + fi + log " removing existing install at $target_dir (--force)" + if [ "$DRY_RUN" -eq 0 ]; then + rm -rf "$target_dir" + fi + fi + + if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping copy + venv-wrapper install" + return + fi + + # Atomic install: build in a staging dir, mv into place. + local staging + staging=$(mktemp -d -p "$INSTALL_ROOT" ".staging-XXXXXX") + if ! cp -r "$skill_src/." "$staging/"; then + err "FATAL: cp failed from $skill_src to $staging" + rm -rf "$staging" + exit 3 + fi + + # Splice in scripts/common/ so `from common.X import Y` resolves + # at runtime. skill-finder and skill-publisher both share the + # same common package. + if [ -n "$SRC_COMMON_PACKAGE" ] && [ -d "$staging/scripts" ]; then + if ! cp -r "$SRC_COMMON_PACKAGE" "$staging/scripts/common"; then + err "FATAL: cp of scripts/common failed" + rm -rf "$staging" + exit 3 + fi + fi + + # Install the venv wrapper. The wrapper is a tiny shim that + # invokes $VENV_DIR/bin/python on the script passed as argv[1]. + # SKILL.md invocations are rewritten to go through it so agents + # that would otherwise use system python (which may lack our + # deps) instead use our venv. + mkdir -p "$staging/bin" + cat >"$staging/bin/run-with-venv.sh" < ${SKILL_DIR}/bin/run-with-venv.sh ${SKILL_DIR}/scripts/X.py + if command -v python3 >/dev/null 2>&1; then + python3 - "$staging/SKILL.md" <<'PYREWRITE' +import sys +path = sys.argv[1] +old = open(path).read() +new = old.replace( + 'python3 ${SKILL_DIR}/scripts/', + '${SKILL_DIR}/bin/run-with-venv.sh ${SKILL_DIR}/scripts/', +) +new = new.replace( + 'bash ${SKILL_DIR}/scripts/', + '${SKILL_DIR}/bin/run-with-venv.sh ${SKILL_DIR}/scripts/', +) +if new != old: + open(path, 'w').write(new) + print(f' rewrote {path} to invoke scripts via the venv wrapper') +else: + print(f' SKILL.md at {path} needs no rewrite (unrecognised invocation shape)') +PYREWRITE + fi + fi + + # Atomic mv into place. + mkdir -p "$INSTALL_ROOT" + if ! mv "$staging" "$target_dir"; then + err "FATAL: mv into place failed: $staging -> $target_dir" + rm -rf "$staging" + exit 3 + fi + log " installed: $target_dir" +} + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: would install skill-finder to $INSTALL_ROOT/skill-finder" + if [ "$SKIP_PUBLISHER" -eq 0 ]; then + log "DRY RUN: would install skill-publisher to $INSTALL_ROOT/skill-publisher" + fi +else + install_one_skill "skill-finder" "$SRC_SKILL_FINDER" + if [ "$SKIP_PUBLISHER" -eq 0 ] && [ -n "${SRC_SKILL_PUBLISHER:-}" ]; then + install_one_skill "skill-publisher" "$SRC_SKILL_PUBLISHER" + fi +fi + +# =============================================================== +# Step 6: next-steps message +# =============================================================== + +log "" +log "skill-finder installed successfully to $INSTALL_ROOT/skill-finder" +if [ "$SKIP_PUBLISHER" -eq 0 ] && [ -n "${SRC_SKILL_PUBLISHER:-}" ]; then + log "skill-publisher installed to $INSTALL_ROOT/skill-publisher" +fi +log "" +log "IMPORTANT: skill-finder cannot verify any downloaded skill" +log "manifests until at least one trust root PEM is present in" +log "$INSTALL_ROOT/skill-finder/keys/. Two ways to populate it:" +log "" +log " 1. Run bin/provision.sh from the apigee/devrel checkout" +log " to generate a fresh ed25519 keypair, publish the initial" +log " example skills to your API hub signed with the new key," +log " and drop the corresponding public key into keys/." +log "" +log " 2. Drop your org's existing public key PEM into" +log " $INSTALL_ROOT/skill-finder/keys/.pem" +log " manually. Any file matching keys/*.pem is loaded." +log "" +log "See docs/trust-root.md for the full trust-model discussion." diff --git a/references/apigee-skills-serving/bin/install-skill-publisher.sh b/references/apigee-skills-serving/bin/install-skill-publisher.sh new file mode 100755 index 00000000..85d1758c --- /dev/null +++ b/references/apigee-skills-serving/bin/install-skill-publisher.sh @@ -0,0 +1,346 @@ +#!/usr/bin/env bash +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# install-skill-publisher.sh +# +# One-line installer for the skill-publisher agentic skill from +# apigee/devrel. Same fetch model as install-skill-finder.sh +# (source from a git ref, no signed release assets, trust by +# provenance) but installs only the publisher skill. +# +# skill-publisher is the author-side companion to skill-finder: +# it orchestrates pack -> sign -> upload -> register for a source +# skill directory. Unlike skill-finder, it does NOT verify +# signatures at runtime -- the author IS the signer -- so no +# trust-root population step is needed. +# +# If you want both skills, prefer install-skill-finder.sh which +# installs both by default (this script's behaviour is a subset). +# +# Usage: +# install-skill-publisher.sh [--runtime opencode|gemini|antigravity] +# [--install-root

] +# [--ref ] +# [--repo ] +# [--venv-dir ] +# [--use-uv] +# [--force] +# [--dry-run] +# [-h | --help] +# +# See install-skill-finder.sh for the full trust-model rationale +# and PEP-668 explanation. This script is intentionally a thin +# wrapper around install-skill-finder.sh --skip-publisher --skip-finder +# would be prettier, but that would require adding a --skip-finder +# flag to the finder installer -- keeping them separate lets +# operators install only the piece they need without cross-flags. +# +# Exit codes: +# 0 install OK +# 1 user error +# 2 network failure +# 3 install-target write failure + +set -u + +DEFAULT_REF="main" +DEFAULT_REPO="apigee/devrel" +SKILLS_SUBPATH="references/apigee-skills-serving/skills" + +# skill-publisher shells out to the publisher-side Python +# modules (pack_skill, sign_skill, upload_skill, register_skill) +# which need cryptography + google-auth + requests + pyyaml. +PY_DEPS=(cryptography google-auth requests pyyaml) + +VENV_DIR="$HOME/.local/share/skill-finder/venv" + +RUNTIME="" +INSTALL_ROOT="" +REF="$DEFAULT_REF" +REPO="$DEFAULT_REPO" +USE_UV=0 +FORCE=0 +DRY_RUN=0 + +usage() { + cat <<'USAGE' +Usage: + install-skill-publisher.sh [--runtime opencode|gemini|antigravity] + [--install-root ] + [--ref ] + [--repo ] + [--venv-dir ] + [--use-uv] + [--force] + [--dry-run] + +Flags: + --runtime Runtime to install into. Auto-detected if omitted. + --install-root

Override the runtime's default skills root. + --ref Git ref (branch, tag, or commit SHA). Default: main. + --repo GitHub repo. Default: apigee/devrel. + --venv-dir

Per-user venv. Default: ~/.local/share/skill-finder/venv + --use-uv Use `uv` instead of `python3 -m venv`. + --force Overwrite an existing install. + --dry-run Print what would happen, do nothing. + -h, --help Show this help. +USAGE +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --runtime) RUNTIME="${2:-}"; shift 2 ;; + --install-root) INSTALL_ROOT="${2:-}"; shift 2 ;; + --ref) REF="${2:-}"; shift 2 ;; + --repo) REPO="${2:-}"; shift 2 ;; + --venv-dir) VENV_DIR="${2:-}"; shift 2 ;; + --use-uv) USE_UV=1; shift ;; + --force) FORCE=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + -h|--help) usage; exit 0 ;; + *) + echo "[install] FATAL: unknown flag: $1" >&2 + echo "[install] run --help for usage" >&2 + exit 1 + ;; + esac +done + +log() { echo "[install] $*"; } +err() { echo "[install] $*" >&2; } + +# =============================================================== +# Step 1: detect runtime, resolve install root +# =============================================================== + +if [ -z "$RUNTIME" ]; then + if [ -d "$HOME/.config/opencode/skills" ]; then + RUNTIME="opencode" + elif [ -d "$HOME/.gemini/antigravity-browser-profile" ]; then + RUNTIME="antigravity" + elif [ -d "$HOME/.gemini" ]; then + RUNTIME="gemini" + else + RUNTIME="opencode" + fi + log "auto-detected runtime: $RUNTIME (override with --runtime)" +fi + +case "$RUNTIME" in + opencode) DEFAULT_INSTALL_ROOT="$HOME/.config/opencode/skills" ;; + gemini) DEFAULT_INSTALL_ROOT="$HOME/.gemini/skills" ;; + antigravity) DEFAULT_INSTALL_ROOT="$HOME/.gemini/antigravity/skills" ;; + *) + err "FATAL: --runtime must be opencode | gemini | antigravity (got: $RUNTIME)" + exit 1 + ;; +esac + +if [ -z "$INSTALL_ROOT" ]; then + INSTALL_ROOT="$DEFAULT_INSTALL_ROOT" +fi + +log "runtime: $RUNTIME" +log "install root: $INSTALL_ROOT" +log "source repo: $REPO" +log "source ref: $REF" +log "venv dir: $VENV_DIR" + +# =============================================================== +# Step 2: tool preflight +# =============================================================== + +for tool in curl tar python3; do + if ! command -v "$tool" >/dev/null 2>&1; then + err "FATAL: required tool not on PATH: $tool" + exit 1 + fi +done +if [ "$USE_UV" -eq 1 ] && ! command -v uv >/dev/null 2>&1; then + err "FATAL: --use-uv passed but uv is not on PATH" + exit 1 +fi + +PY_VER=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")') +case "$PY_VER" in + 3.10|3.11|3.12|3.13|3.14|3.15|3.16|3.17|3.18|3.19) + log "python: ${PY_VER} (OK)" + ;; + *) + err "FATAL: python3 must be >= 3.10 (found: ${PY_VER})" + exit 1 + ;; +esac + +# =============================================================== +# Step 3: create venv + install deps (idempotent; shared with +# skill-finder's venv) +# =============================================================== + +log "step 3: setting up Python runtime" + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping venv create + dep install" +else + mkdir -p "$(dirname "$VENV_DIR")" || { + err "FATAL: cannot create venv parent dir"; exit 3; + } + VENV_OK=0 + if [ -x "$VENV_DIR/bin/python" ] \ + && "$VENV_DIR/bin/python" -m pip --version >/dev/null 2>&1; then + VENV_OK=1 + fi + if [ "$VENV_OK" -eq 0 ]; then + [ -d "$VENV_DIR" ] && rm -rf "$VENV_DIR" + log " creating venv at $VENV_DIR" + if [ "$USE_UV" -eq 1 ]; then + uv venv --quiet "$VENV_DIR" || { err "FATAL: uv venv failed"; exit 1; } + else + python3 -m venv "$VENV_DIR" \ + || { err "FATAL: python3 -m venv failed"; exit 1; } + fi + else + log " reusing existing venv (pip is healthy)" + fi + log " installing deps: ${PY_DEPS[*]}" + if [ "$USE_UV" -eq 1 ]; then + uv pip install --quiet --python "$VENV_DIR/bin/python" \ + --upgrade "${PY_DEPS[@]}" \ + || { err "FATAL: uv pip install failed"; exit 1; } + else + "$VENV_DIR/bin/python" -m pip install --quiet --upgrade \ + "${PY_DEPS[@]}" \ + || { err "FATAL: pip install failed"; exit 1; } + fi +fi + +# =============================================================== +# Step 4: fetch source from apigee/devrel +# =============================================================== + +TARBALL_TMP=$(mktemp -t skill-publisher-source-XXXXXX.tar.gz) +trap 'rm -f "$TARBALL_TMP"' EXIT + +CANDIDATES=( + "https://codeload.github.com/${REPO}/tar.gz/refs/tags/${REF}" + "https://codeload.github.com/${REPO}/tar.gz/refs/heads/${REF}" + "https://codeload.github.com/${REPO}/tar.gz/${REF}" +) + +log "step 4: fetching source from ${REPO} at ref '${REF}'" +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping source fetch" +else + FETCHED=0 + for URL in "${CANDIDATES[@]}"; do + log " trying: $URL" + if curl -fsSL -o "$TARBALL_TMP" "$URL"; then + log " OK: fetched $(wc -c <"$TARBALL_TMP") bytes" + FETCHED=1 + break + fi + done + if [ "$FETCHED" -eq 0 ]; then + err "FATAL: could not fetch source from any of:" + for URL in "${CANDIDATES[@]}"; do err " $URL"; done + exit 2 + fi +fi + +EXTRACT_TMP=$(mktemp -d -t skill-publisher-extract-XXXXXX) +trap 'rm -f "$TARBALL_TMP"; rm -rf "$EXTRACT_TMP"' EXIT + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: skipping tarball extract" +else + tar -xzf "$TARBALL_TMP" -C "$EXTRACT_TMP" \ + || { err "FATAL: tar extract failed"; exit 2; } + TOPDIR=$(find "$EXTRACT_TMP" -mindepth 1 -maxdepth 1 -type d) + SRC_SKILL_PUBLISHER="$TOPDIR/${SKILLS_SUBPATH}/skill-publisher" + if [ ! -d "$SRC_SKILL_PUBLISHER" ]; then + err "FATAL: tarball does not contain ${SKILLS_SUBPATH}/skill-publisher" + err "Wrong --ref or wrong --repo?" + exit 2 + fi + if [ -d "$TOPDIR/references/apigee-skills-serving/scripts/common" ]; then + SRC_COMMON_PACKAGE="$TOPDIR/references/apigee-skills-serving/scripts/common" + else + SRC_COMMON_PACKAGE="" + fi +fi + +# =============================================================== +# Step 5: install +# =============================================================== + +TARGET_DIR="$INSTALL_ROOT/skill-publisher" +log "step 5: installing to $TARGET_DIR" + +if [ -d "$TARGET_DIR" ] && [ "$FORCE" -eq 0 ]; then + err "FATAL: $TARGET_DIR already exists (pass --force to overwrite)" + exit 3 +fi + +if [ "$DRY_RUN" -eq 1 ]; then + log "DRY RUN: would install to $TARGET_DIR" +else + [ -d "$TARGET_DIR" ] && rm -rf "$TARGET_DIR" + mkdir -p "$INSTALL_ROOT" + STAGING=$(mktemp -d -p "$INSTALL_ROOT" ".staging-XXXXXX") + cp -r "$SRC_SKILL_PUBLISHER/." "$STAGING/" \ + || { err "FATAL: cp failed"; rm -rf "$STAGING"; exit 3; } + if [ -n "$SRC_COMMON_PACKAGE" ] && [ -d "$STAGING/scripts" ]; then + cp -r "$SRC_COMMON_PACKAGE" "$STAGING/scripts/common" \ + || { err "FATAL: cp of common/ failed"; rm -rf "$STAGING"; exit 3; } + fi + mkdir -p "$STAGING/bin" + cat >"$STAGING/bin/run-with-venv.sh" < \\" +log " --bucket --priv-key \\" +log " --project --location " diff --git a/references/apigee-skills-serving/bin/provision.sh b/references/apigee-skills-serving/bin/provision.sh new file mode 100755 index 00000000..de4312c6 --- /dev/null +++ b/references/apigee-skills-serving/bin/provision.sh @@ -0,0 +1,807 @@ +#!/usr/bin/env bash +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# provision.sh +# +# End-to-end bootstrap for the apigee-skills-serving reference +# implementation. Given an empty GCP project (with an existing +# Apigee organisation and an API hub instance), this script: +# +# 1. Enables the GCP APIs skill-finder + skill-publisher need. +# 2. Creates a public-read GCS bucket for signed .skill bundles. +# 3. Creates the API hub custom attribute taxonomy (agentic_skill, +# keywords, gs_uri, signing_key_id) and patches the +# skill-spec value into system-spec-type. +# 4. Generates a fresh ed25519 signing keypair on THIS machine. +# The private key never leaves this machine and is never +# committed to any repo. +# 5. Packs + signs + uploads + registers the three example +# payload skills (currency-converter, weather-lookup, +# apigee-policy-top10) using the fresh key. +# 6. Installs the public key into an already-installed +# skill-finder's keys/ directory so it will trust the +# just-published catalog. Silently skips this step if +# skill-finder is not installed yet -- the operator can +# install skill-finder later and drop the printed key into +# keys/ by hand. +# +# Every step is idempotent. Re-running against a partially- +# provisioned project detects what exists and no-ops those +# steps. Selective skip flags (--skip-apis, --skip-bucket, etc.) +# let you re-run one narrow step for troubleshooting. +# +# NOT provisioned by this script (out of scope): +# +# - The API hub INSTANCE itself. Provisioning takes 40+ minutes, +# requires several console-only decisions (CMEK vs GMEK, +# Vertex region), and is a customer-lifecycle decision. +# provision.sh REQUIRES an existing API hub instance and +# fails fast with a clear message if none exists. +# +# - The Apigee organisation. Same reasons. +# +# - IAM bindings for the demo caller. The caller is expected to +# have roles/owner (or the fine-grained equivalent). The script +# tests every API call and surfaces the specific missing perm +# if any call fails with 403. +# +# Usage: +# bin/provision.sh --project [flags] +# +# See --help for the full flag list. +# +# Exit codes: +# 0 success +# 1 user error (bad flag, missing tool, missing prereq) +# 2 network / GCP API failure +# 3 cryptographic error (key generation failed) +# 4 publish pipeline failure (pack / sign / upload / register) + +set -u + +# =============================================================== +# Locate self, discover devrel checkout +# =============================================================== +# +# provision.sh depends on sibling scripts in ../scripts/*.py and +# on the three example skills in ../skills/. We resolve relative +# to $0 so the script works from any cwd as long as it's run +# from a devrel checkout. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)" + +if [ ! -d "${REPO_ROOT}/scripts" ] \ + || [ ! -d "${REPO_ROOT}/skills" ]; then + echo "[provision] FATAL: cannot find scripts/ and skills/ next to bin/" >&2 + echo "[provision] REPO_ROOT resolved to: ${REPO_ROOT}" >&2 + echo "[provision] Run this from a devrel checkout of" >&2 + echo "[provision] references/apigee-skills-serving/." >&2 + exit 1 +fi + +# =============================================================== +# Defaults +# =============================================================== + +DEFAULT_LOCATION="us-west1" +DEFAULT_KEY_PATH="${HOME}/.config/apigee-skills-demo/signing.raw" +EXAMPLE_SKILLS=(currency-converter weather-lookup apigee-policy-top10) + +# =============================================================== +# Arg parsing +# =============================================================== + +PROJECT="" +LOCATION="$DEFAULT_LOCATION" +BUCKET="" +APIGEE_ORG="" +KEY_PATH="$DEFAULT_KEY_PATH" +RUNTIME="" +SKIP_APIS=0 +SKIP_BUCKET=0 +SKIP_TAXONOMY=0 +SKIP_SKILLS=0 +SKIP_KEY=0 +SKIP_TRUST_ROOT_INSTALL=0 +CHECK_ONLY=0 +DRY_RUN=0 +FORCE=0 +YES=0 + +usage() { + cat < [flags] + +Required: + --project

GCP project id to provision. + +Optional: + --location API hub / GCS bucket location. + Default: ${DEFAULT_LOCATION} + --bucket GCS bucket name for signed .skill + bundles. Default: + -skills- + --apigee-org Apigee organisation used by the + apigee-policy-top10 example skill. + Default: same as --project. + --key-path

Path to write / read the ed25519 + private key. Default: ${DEFAULT_KEY_PATH} + --runtime Agent runtime whose skill-finder + install should receive the trust root + PEM. One of opencode | gemini | + antigravity. Auto-detected if omitted. + +Selective skips (idempotent re-runs): + --skip-apis Skip enabling GCP APIs. + --skip-bucket Skip GCS bucket create/config. + --skip-taxonomy Skip API hub attribute taxonomy. + --skip-skills Skip publishing the 3 example skills. + --skip-key Skip key generation (reuse existing). + --skip-trust-root-install Skip dropping the pubkey into + skill-finder's keys/ (e.g. if + skill-finder is not installed yet). + +Modes: + --check-only Read-only preflight. Reports what's + missing without making any changes. + --dry-run Print what would be done, do nothing. + --force Overwrite an existing signing key + (default: reuse existing to avoid + invalidating prior signatures). + --yes No interactive prompts. + + -h, --help Show this help. +USAGE +} + +while [ "$#" -gt 0 ]; do + case "$1" in + --project) PROJECT="${2:-}"; shift 2 ;; + --location) LOCATION="${2:-}"; shift 2 ;; + --bucket) BUCKET="${2:-}"; shift 2 ;; + --apigee-org) APIGEE_ORG="${2:-}"; shift 2 ;; + --key-path) KEY_PATH="${2:-}"; shift 2 ;; + --runtime) RUNTIME="${2:-}"; shift 2 ;; + --skip-apis) SKIP_APIS=1; shift ;; + --skip-bucket) SKIP_BUCKET=1; shift ;; + --skip-taxonomy) SKIP_TAXONOMY=1; shift ;; + --skip-skills) SKIP_SKILLS=1; shift ;; + --skip-key) SKIP_KEY=1; shift ;; + --skip-trust-root-install) SKIP_TRUST_ROOT_INSTALL=1; shift ;; + --check-only) CHECK_ONLY=1; shift ;; + --dry-run) DRY_RUN=1; shift ;; + --force) FORCE=1; shift ;; + --yes) YES=1; shift ;; + -h|--help) usage; exit 0 ;; + *) + echo "[provision] FATAL: unknown flag: $1" >&2 + echo "[provision] run --help for usage" >&2 + exit 1 + ;; + esac +done + +# Resolve default project from gcloud if not passed. +if [ -z "$PROJECT" ]; then + PROJECT=$(gcloud config get-value project 2>/dev/null || echo "") + if [ -z "$PROJECT" ]; then + echo "[provision] FATAL: --project not given and no default in gcloud config." >&2 + echo "[provision] Either pass --project or run:" >&2 + echo "[provision] gcloud config set project " >&2 + exit 1 + fi +fi + +# Derive defaults that depend on --project / --location. +if [ -z "$BUCKET" ]; then + BUCKET="${PROJECT}-skills-${LOCATION}" +fi +if [ -z "$APIGEE_ORG" ]; then + APIGEE_ORG="$PROJECT" +fi + +# =============================================================== +# Colour + logging helpers +# =============================================================== + +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +log() { echo "[provision] $*"; } +ok() { echo -e "[provision] ${GREEN}OK${NC}: $*"; } +warn() { echo -e "[provision] ${YELLOW}WARN${NC}: $*"; } +err() { echo -e "[provision] ${RED}FAIL${NC}: $*" >&2; } + +step() { + echo + echo "===============================================" + echo " $*" + echo "===============================================" +} + +confirm() { + # Ask the user y/N. Auto-yes if --yes was passed. Returns 0 for + # yes, 1 for no. + local prompt="$1" + if [ "$YES" -eq 1 ]; then + log " auto-yes (--yes): $prompt" + return 0 + fi + read -r -p "[provision] $prompt [y/N]: " ans + case "$ans" in + y|Y|yes|YES) return 0 ;; + *) return 1 ;; + esac +} + +# =============================================================== +# Header +# =============================================================== + +step "apigee-skills-serving provisioner" +log "project: $PROJECT" +log "location: $LOCATION" +log "bucket: $BUCKET" +log "apigee org: $APIGEE_ORG" +log "key path: $KEY_PATH" +if [ -n "$RUNTIME" ]; then + log "runtime: $RUNTIME (explicit)" +fi +if [ "$CHECK_ONLY" -eq 1 ]; then + log "MODE: --check-only (read-only preflight)" +elif [ "$DRY_RUN" -eq 1 ]; then + log "MODE: --dry-run (no changes will be made)" +fi + +# =============================================================== +# Step 0: preflight (tools, ADC, venv) +# =============================================================== + +step "Step 0: preflight" + +# Required tools. +for tool in gcloud python3 curl; do + if ! command -v "$tool" >/dev/null 2>&1; then + err "required tool missing: $tool" + exit 1 + fi +done +ok "required tools present (gcloud, python3, curl)" + +# ADC. +if ! gcloud auth application-default print-access-token >/dev/null 2>&1; then + err "ADC token not available. Run:" + err " gcloud auth application-default login" + exit 1 +fi +ok "ADC token available" + +# Confirm the caller's identity so we don't accidentally provision +# under the wrong account. +ACTIVE_ACCOUNT=$(gcloud auth list --filter=status:ACTIVE --format="value(account)" 2>/dev/null | head -1) +ok "active gcloud account: ${ACTIVE_ACCOUNT}" + +# Python venv for the publisher scripts. Reuse skill-finder's +# venv if it exists (installed by install-skill-finder.sh). +VENV_DIR="${HOME}/.local/share/skill-finder/venv" +if [ -x "${VENV_DIR}/bin/python" ] \ + && "${VENV_DIR}/bin/python" -m pip --version >/dev/null 2>&1; then + PY="${VENV_DIR}/bin/python" + ok "reusing skill-finder venv at ${VENV_DIR}" +else + # Bootstrap a fresh venv if skill-finder isn't installed yet. + # We need one for pack_skill.py / sign_skill.py (they import + # cryptography and pyyaml). + log " creating temporary venv at ${VENV_DIR}" + if [ "$CHECK_ONLY" -eq 0 ] && [ "$DRY_RUN" -eq 0 ]; then + mkdir -p "$(dirname "${VENV_DIR}")" + python3 -m venv "${VENV_DIR}" \ + || { err "python3 -m venv failed"; exit 1; } + "${VENV_DIR}/bin/python" -m pip install --quiet --upgrade \ + cryptography google-auth requests pyyaml \ + || { err "pip install of venv deps failed"; exit 1; } + fi + PY="${VENV_DIR}/bin/python" + ok "created venv at ${VENV_DIR}" +fi + +# Every python invocation needs PYTHONPATH so `from scripts.common +# import X` resolves. +export PYTHONPATH="${REPO_ROOT}" + +# =============================================================== +# Step 1: enable APIs +# =============================================================== + +step "Step 1: enable GCP APIs" + +REQUIRED_APIS=( + apigee.googleapis.com + apihub.googleapis.com + storage.googleapis.com + iam.googleapis.com +) + +if [ "$SKIP_APIS" -eq 1 ]; then + log " skipped (--skip-apis)" +else + ENABLED_APIS=$(gcloud services list --enabled \ + --project="$PROJECT" --format="value(config.name)" 2>/dev/null || echo "") + TO_ENABLE=() + for api in "${REQUIRED_APIS[@]}"; do + if echo "$ENABLED_APIS" | grep -qFx "$api"; then + ok "$api already enabled" + else + TO_ENABLE+=("$api") + fi + done + if [ "${#TO_ENABLE[@]}" -gt 0 ]; then + if [ "$CHECK_ONLY" -eq 1 ]; then + warn "would enable: ${TO_ENABLE[*]}" + elif [ "$DRY_RUN" -eq 1 ]; then + log " DRY RUN: would enable ${TO_ENABLE[*]}" + else + log " enabling: ${TO_ENABLE[*]}" + gcloud services enable "${TO_ENABLE[@]}" --project="$PROJECT" \ + || { err "gcloud services enable failed"; exit 2; } + ok "enabled ${TO_ENABLE[*]}" + fi + fi +fi + +# =============================================================== +# Step 2: verify API hub instance + Apigee org exist +# =============================================================== + +step "Step 2: verify prereqs (API hub instance, Apigee org)" + +# API hub instance. Creating one takes 40+ min and has console- +# only choices, so we insist it exists rather than automating. +APIHUB_INSTANCE=$(gcloud apihub api-hub-instances lookup \ + --project="$PROJECT" --location="$LOCATION" \ + --format="value(apiHubInstance.name)" 2>/dev/null || echo "") +if [ -z "$APIHUB_INSTANCE" ]; then + err "no API hub instance in ${PROJECT} / ${LOCATION}" + err "Provision one in the Cloud Console:" + err " https://console.cloud.google.com/apigee/api-hub/setup?project=${PROJECT}" + err "This takes 40+ min. Re-run provision.sh when it's ACTIVE." + exit 1 +fi +ok "API hub instance: ${APIHUB_INSTANCE}" + +# Apigee organisation. +TOKEN=$(gcloud auth application-default print-access-token) +ORG_JSON=$(curl -sf -H "Authorization: Bearer $TOKEN" \ + "https://apigee.googleapis.com/v1/organizations/${APIGEE_ORG}" 2>/dev/null || echo "") +if [ -z "$ORG_JSON" ]; then + err "Apigee organisation '${APIGEE_ORG}' not accessible." + err "Either --apigee-org is wrong, or the caller lacks" + err "apigee.organizations.get permission on it." + err "The apigee-policy-top10 skill needs this org; other" + err "skills do not. Pass --skip-skills to skip publishing" + err "if you only want the infrastructure provisioned." + exit 1 +fi +ok "Apigee organisation: ${APIGEE_ORG}" + +# =============================================================== +# Step 3: create GCS bucket +# =============================================================== + +step "Step 3: GCS bucket for signed skill bundles" + +if [ "$SKIP_BUCKET" -eq 1 ]; then + log " skipped (--skip-bucket)" +else + BUCKET_URI="gs://${BUCKET}" + if gcloud storage buckets describe "$BUCKET_URI" \ + --project="$PROJECT" >/dev/null 2>&1; then + ok "bucket ${BUCKET_URI} exists" + else + if [ "$CHECK_ONLY" -eq 1 ]; then + warn "would create bucket ${BUCKET_URI}" + elif [ "$DRY_RUN" -eq 1 ]; then + log " DRY RUN: would create ${BUCKET_URI}" + else + log " creating ${BUCKET_URI}" + gcloud storage buckets create "$BUCKET_URI" \ + --project="$PROJECT" --location="$LOCATION" \ + --uniform-bucket-level-access \ + --no-public-access-prevention 2>&1 | tail -3 + ok "created ${BUCKET_URI}" + fi + fi + + # Grant allUsers reader. This is a demo-only choice -- + # skill-finder does anonymous HTTPS GET for the .skill zips, + # so the bucket must be public-read. In production you'd + # bind roles/storage.objectViewer to a specific service + # account (or use signed URLs) instead. + if [ "$CHECK_ONLY" -eq 0 ] && [ "$DRY_RUN" -eq 0 ]; then + log " granting allUsers:objectViewer (needed for skill-finder" + log " anonymous downloads -- demo choice)" + gcloud storage buckets add-iam-policy-binding "$BUCKET_URI" \ + --member=allUsers --role=roles/storage.objectViewer \ + >/dev/null 2>&1 || warn "allUsers binding may already exist" + ok "public-read enabled on ${BUCKET_URI}" + fi +fi + +# =============================================================== +# Step 4: API hub attribute taxonomy +# =============================================================== + +step "Step 4: API hub attribute taxonomy" + +if [ "$SKIP_TAXONOMY" -eq 1 ]; then + log " skipped (--skip-taxonomy)" +else + # Detect existing state before deciding whether to create. + # Query the four user-defined attrs and system-spec-type. + ATTRS_URL="https://apihub.googleapis.com/v1/projects/${PROJECT}/locations/${LOCATION}/attributes" + EXPECTED_ATTRS=(agentic_skill keywords gs_uri signing_key_id) + MISSING_ATTRS=() + for a in "${EXPECTED_ATTRS[@]}"; do + if ! curl -sf -H "Authorization: Bearer $TOKEN" "${ATTRS_URL}/${a}" \ + >/dev/null 2>&1; then + MISSING_ATTRS+=("$a") + fi + done + # Check that system-spec-type includes skill-spec. + SST_JSON=$(curl -sf -H "Authorization: Bearer $TOKEN" \ + "${ATTRS_URL}/system-spec-type" 2>/dev/null || echo "") + if echo "$SST_JSON" | grep -q '"id": "skill-spec"'; then + HAS_SKILL_SPEC=1 + else + HAS_SKILL_SPEC=0 + fi + + if [ "${#MISSING_ATTRS[@]}" -eq 0 ] && [ "$HAS_SKILL_SPEC" -eq 1 ]; then + ok "user-defined attributes: all 4 present" + ok "system-spec-type includes skill-spec" + elif [ "$CHECK_ONLY" -eq 1 ]; then + if [ "${#MISSING_ATTRS[@]}" -gt 0 ]; then + warn "would create missing user-defined attrs: ${MISSING_ATTRS[*]}" + fi + if [ "$HAS_SKILL_SPEC" -eq 0 ]; then + warn "would add 'skill-spec' to system-spec-type enum" + fi + elif [ "$DRY_RUN" -eq 1 ]; then + log " DRY RUN: would create ${MISSING_ATTRS[*]}, patch system-spec-type" + else + log " creating user-defined attributes (idempotent)" + "$PY" -m scripts.update_taxonomy \ + --project "$PROJECT" --location "$LOCATION" \ + || { err "update_taxonomy.py failed"; exit 2; } + ok "user-defined attributes created/verified" + + log " patching skill-spec into system-spec-type enum" + # Idempotent: PATCHing with the same allowed_values list is + # a no-op if the value already exists. + curl -sf -X PATCH \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: application/json" \ + "https://apihub.googleapis.com/v1/projects/${PROJECT}/locations/${LOCATION}/attributes/system-spec-type?updateMask=allowed_values" \ + -d '{ + "allowedValues": [ + {"id":"openapi","displayName":"OpenAPI Spec","description":"OpenAPI Spec","immutable":true}, + {"id":"proto","displayName":"Proto","description":"Proto","immutable":true}, + {"id":"wsdl","displayName":"WSDL","description":"WSDL","immutable":true}, + {"id":"mcp-spec","displayName":"MCP Spec","description":"MCP Spec","immutable":true}, + {"id":"skill-spec","displayName":"Skill Spec","description":"Signed agent skill manifest (apigee-skills-serving reference)"} + ] + }' >/dev/null \ + || { err "system-spec-type PATCH failed"; exit 2; } + ok "skill-spec added to system-spec-type" + fi +fi + +# =============================================================== +# Step 5: generate signing keypair +# =============================================================== + +step "Step 5: ed25519 signing keypair" + +if [ "$SKIP_KEY" -eq 1 ]; then + log " skipped (--skip-key); reusing existing key at ${KEY_PATH}" + if [ ! -f "$KEY_PATH" ]; then + err "--skip-key passed but no key found at ${KEY_PATH}" + exit 1 + fi +elif [ -f "$KEY_PATH" ] && [ "$FORCE" -eq 0 ]; then + ok "existing key at ${KEY_PATH} (reusing; pass --force to regenerate)" +else + if [ "$CHECK_ONLY" -eq 1 ]; then + warn "would generate keypair at ${KEY_PATH}" + elif [ "$DRY_RUN" -eq 1 ]; then + log " DRY RUN: would generate keypair" + else + if [ -f "$KEY_PATH" ] && [ "$FORCE" -eq 1 ]; then + warn "--force: overwriting existing key at ${KEY_PATH}" + warn " Prior signatures made with the old key will no longer" + warn " verify against the new one -- re-publish all skills" + warn " after this step." + if ! confirm "Proceed with key regeneration?"; then + err "aborted at key regeneration" + exit 1 + fi + fi + log " generating fresh ed25519 keypair at ${KEY_PATH}" + mkdir -p "$(dirname "$KEY_PATH")" + "$PY" </dev/null \ + | "$PY" -c " +import json, sys +try: + d = json.load(sys.stdin) +except Exception: + sys.exit(0) +for a in d.get('apis', []): + print(a.get('displayName', '')) +" 2>/dev/null || echo "") + MISSING_SKILLS=() + PRESENT_SKILLS=() + for skill in "${EXAMPLE_SKILLS[@]}"; do + if echo "$EXISTING_APIS" | grep -qFx "$skill"; then + PRESENT_SKILLS+=("$skill") + else + MISSING_SKILLS+=("$skill") + fi + done + + if [ "${#PRESENT_SKILLS[@]}" -gt 0 ]; then + ok "already published: ${PRESENT_SKILLS[*]}" + fi + if [ "${#MISSING_SKILLS[@]}" -eq 0 ]; then + ok "all example skills already published" + # Skip the entire publish loop. + SKIP_SKILLS=1 + fi +fi + +if [ "$SKIP_SKILLS" -eq 1 ]; then + : # already handled above +elif [ "$CHECK_ONLY" -eq 1 ]; then + warn "would publish: ${MISSING_SKILLS[*]}" +elif [ "$DRY_RUN" -eq 1 ]; then + log " DRY RUN: would publish ${MISSING_SKILLS[*]}" +else + # Working dir for patched manifests + signed manifests. Kept + # outside the source tree so the source tree stays clean. + WORKDIR="${HOME}/.local/share/apigee-skills-serving/publish-workdir" + mkdir -p "$WORKDIR" + + for skill in "${MISSING_SKILLS[@]}"; do + log " --- publishing $skill ---" + SRC_MANIFEST="${REPO_ROOT}/skills/${skill}/manifest.yaml" + WORK_MANIFEST="${WORKDIR}/${skill}.manifest.yaml" + SIGNED_MANIFEST="${WORKDIR}/${skill}.signed.yaml" + SKILL_ZIP="/tmp/${skill}.skill" + + # Patch gs_uri (source manifest has no gs_uri; publisher + # scripts require it to be present before signing). + cp "$SRC_MANIFEST" "$WORK_MANIFEST" + "$PY" -c " +import yaml +p = '${WORK_MANIFEST}' +m = yaml.safe_load(open(p)) +m['gs_uri'] = 'gs://${BUCKET}/${skill}.skill' +open(p, 'w').write(yaml.safe_dump(m, sort_keys=True)) +" + # Pack. + "$PY" -m scripts.pack_skill \ + --src "${REPO_ROOT}/skills/${skill}" \ + --out "$SKILL_ZIP" \ + || { err "pack failed for $skill"; exit 4; } + # Upload. + "$PY" -m scripts.upload_skill \ + --zip "$SKILL_ZIP" --bucket "$BUCKET" \ + || { err "upload failed for $skill"; exit 4; } + # Sign. + "$PY" -m scripts.sign_skill \ + --manifest "$WORK_MANIFEST" \ + --zip "$SKILL_ZIP" \ + --priv-key "$KEY_PATH" \ + --out "$SIGNED_MANIFEST" \ + || { err "sign failed for $skill"; exit 4; } + # Register. + "$PY" -m scripts.register_skill \ + --manifest "$SIGNED_MANIFEST" \ + --project "$PROJECT" --location "$LOCATION" \ + || { err "register failed for $skill"; exit 4; } + ok "$skill published" + done +fi + +# =============================================================== +# Step 7: install trust root into skill-finder's keys/ +# =============================================================== + +step "Step 7: install trust root into skill-finder" + +if [ "$SKIP_TRUST_ROOT_INSTALL" -eq 1 ]; then + log " skipped (--skip-trust-root-install)" +elif [ "$CHECK_ONLY" -eq 1 ] || [ "$DRY_RUN" -eq 1 ]; then + log " would install ${FINGERPRINT:-}.pem into skill-finder/keys/" +else + # Auto-detect runtime if not passed. + if [ -z "$RUNTIME" ]; then + if [ -d "$HOME/.gemini/skills/skill-finder" ]; then + RUNTIME="gemini" + elif [ -d "$HOME/.gemini/antigravity/skills/skill-finder" ]; then + RUNTIME="antigravity" + elif [ -d "$HOME/.config/opencode/skills/skill-finder" ]; then + RUNTIME="opencode" + else + warn "skill-finder is not installed in any known runtime." + warn "The example skills ARE published, but skill-finder" + warn "won't trust them until you install skill-finder and" + warn "drop the following PEM into keys/:" + warn "" + "$PY" <.pem" + warn " fingerprint: ${FINGERPRINT}" + RUNTIME="" + fi + fi + + if [ -n "$RUNTIME" ]; then + case "$RUNTIME" in + opencode) SKILLS_ROOT="$HOME/.config/opencode/skills" ;; + gemini) SKILLS_ROOT="$HOME/.gemini/skills" ;; + antigravity) SKILLS_ROOT="$HOME/.gemini/antigravity/skills" ;; + esac + KEYS_DIR="${SKILLS_ROOT}/skill-finder/keys" + if [ ! -d "$KEYS_DIR" ]; then + warn "skill-finder is not installed at ${SKILLS_ROOT}/skill-finder" + warn "Install it first with:" + warn " bash bin/install-skill-finder.sh --runtime ${RUNTIME}" + else + # .pem + FP_STEM="${FINGERPRINT#sha256:}" + PEM_PATH="${KEYS_DIR}/${FP_STEM}.pem" + if [ -f "$PEM_PATH" ]; then + ok "trust root already installed: ${PEM_PATH}" + elif [ "$CHECK_ONLY" -eq 1 ] || [ "$DRY_RUN" -eq 1 ]; then + log " would install trust root to: ${PEM_PATH}" + else + "$PY" <}" +if [ -n "${RUNTIME:-}" ]; then + log "Runtime: ${RUNTIME}" +fi +log "" +log "Next steps:" +log " 1. If skill-finder is not yet installed on this machine:" +log " bash bin/install-skill-finder.sh --runtime " +log " Then re-run: bash bin/provision.sh --project ${PROJECT} \\" +log " --skip-apis --skip-bucket --skip-taxonomy --skip-skills" +log " to install just the trust root." +log "" +log " 2. Launch your agent runtime and ask a question that matches" +log " one of the published skill keywords (e.g. 'currency'," +log " 'weather', 'apigee policies'). skill-finder will find the" +log " matching skill in API hub, verify its signature against" +log " the trust root we just installed, and install it into" +log " the runtime skills dir." +log "" +log " 3. Re-run this script any time to reconcile drift. All" +log " steps are idempotent. Use --skip-* flags to narrow the" +log " re-run to specific steps." diff --git a/references/apigee-skills-serving/docs/architecture.md b/references/apigee-skills-serving/docs/architecture.md index bb01d18e..f20725ee 100644 --- a/references/apigee-skills-serving/docs/architecture.md +++ b/references/apigee-skills-serving/docs/architecture.md @@ -90,19 +90,68 @@ verifying one manifest but installing a different bundle. ### What the consumer verifies -At install time, the consumer: +The reference consumer implementation is the `skill-finder` skill +(installed via `bin/install-skill-finder.sh`). At install time, +`find_install.py` performs a locked 16-step contract: 1. Fetches the manifest from API hub. 2. Looks up the publisher's public key by the `signing_key_id` - fingerprint (typically out-of-band; the consumer is configured with - a list of trusted publisher keys). + fingerprint against **the trust roots installed on this + machine** (see "Per-deployment trust roots" below). 3. Re-canonicalises the manifest (excluding signature fields). 4. Verifies the Ed25519 signature. 5. Downloads the `.skill` zip from `gs_uri`. -6. Computes SHA-256 of the downloaded zip and compares it against the - manifest's `zip_sha256`. -7. Only after both checks pass does the consumer extract the `.skill` - contents to the local skills directory. +6. Computes SHA-256 of the downloaded zip and compares it against + the manifest's `zip_sha256`. +7. Only after both checks pass does the consumer extract the + `.skill` contents to the local skills directory. + +### Per-deployment trust roots + +The trust root is **not** a shared, repo-baked public key. It is a +per-deployment secret: each customer who runs `bin/provision.sh` +generates a fresh Ed25519 keypair on their own machine, signs +their catalog with it, and drops the corresponding public PEM into +their local skill-finder install's `keys/` directory. The private +key never leaves the customer's machine and is never committed to +any repo (including `apigee/devrel`). + +Consequences: + +- **skill-finder itself is trusted by provenance**, not by + signature. The operator fetched skill-finder from a known-good + URL (`raw.githubusercontent.com/apigee/devrel/main/…`) over TLS + from the `apigee` GitHub organisation. There is no signature to + verify because the code that would verify it *is* skill-finder. + +- **Payload skills** (currency-converter, weather-lookup, + apigee-policy-top10, and anything a customer publishes on top) + ARE cryptographically verified against the per-deployment trust + root. + +### Multi-key trust roots + +skill-finder loads **every** `.pem` file under +`~/./skills/skill-finder/keys/` as an accepted trust +root. A downloaded manifest is accepted if its `signing_key_id` +matches **any** installed key. This supports three operator +workflows: + +- **Multi-publisher orgs**: Alice's team's key and Bob's team's + key can both be trusted simultaneously. Any skill signed by + either team installs cleanly. +- **Zero-downtime key rotation**: add the new key alongside the + old one; publish new skills signed with the new key; wait for + callers to stop using the old key; remove the old key. No + install-time gap. +- **Bootstrap onboarding**: `bin/provision.sh` drops a + `.pem` into `keys/` after the initial key + generation, giving skill-finder its first trust root without a + separate configuration step. + +If `keys/` is empty at verify time, skill-finder emits +`trust root: FAILED — no valid keys loaded from …` and refuses +to install anything. ### Threat model @@ -111,12 +160,27 @@ At install time, the consumer: | Compromised GCS bucket serves a wrong zip | `zip_sha256` mismatch is detected before extraction. | | Modified manifest in API hub | Ed25519 signature mismatch is detected. | | Replay of an old (vulnerable) skill version | Consumer is responsible for tracking versions; manifest carries `version`. | -| Compromised publisher signing key | Out of scope; operator must rotate the trust root. | +| Compromised publisher signing key | Operator rotates by adding a new key to `keys/`, re-publishing all skills, then removing the old key. Multi-key support means no install-time gap. | | Malicious skill body (legitimate signature) | Out of scope; agent runtime sandboxing is the boundary. | - -The reference does **not** introduce a new sandbox. The agent runtime -that loads the skill (OpenCode, Claude Code, etc.) is responsible for -limiting what skill code can do once it runs. +| Compromised skill-finder installer over TLS | Out of scope for v1 (relies on TLS + apigee org provenance). Operators can pin `--ref ` in the installer invocation to reduce exposure. | + +The reference does **not** introduce a new sandbox. The agent +runtime that loads the skill (OpenCode, Gemini CLI, Antigravity, +Claude Code, custom MCP hosts) is responsible for limiting what +skill code can do once it runs. + +### Trust-root discovery in v2 (not implemented) + +A future iteration could store the trust root in API hub itself +as a well-known attribute on a placeholder API resource. This +would let skill-finder discover the trust root from the same +catalog it queries for skills, eliminating the +`bin/provision.sh`-drops-a-PEM step. The first fetch would be +trust-on-first-use (TOFU) against a URL the operator provides; +subsequent fetches would verify against the cached pubkey. This +is a v2 design and is intentionally out of scope for v1: it +requires new attribute conventions, taxonomy changes, and a +bootstrap mode that would add several more moving parts. ## Canonical serialization diff --git a/references/apigee-skills-serving/docs/provisioning.md b/references/apigee-skills-serving/docs/provisioning.md new file mode 100644 index 00000000..03430558 --- /dev/null +++ b/references/apigee-skills-serving/docs/provisioning.md @@ -0,0 +1,300 @@ + + +# Provisioning runbook + +This document walks through provisioning a green-field GCP project +as an apigee-skills-serving reference deployment. The end state: +skill-finder is installed on the operator's machine, the API hub +catalog contains three signed example skills, and a fresh +Ed25519 keypair (owned by the operator) is the deployment's trust +root. + +If you already have a deployment and just want to add another +skill or rotate keys, jump to the [Common workflows](#common-workflows) +section. + +## Prerequisites (must exist before you start) + +Two things `bin/provision.sh` does NOT create for you, because +they have long provisioning times and console-only decisions: + +1. **The Apigee organisation.** Provisioning takes 10-30 minutes + and asks for region, networking, and analytics preferences. + [Enable Apigee](https://cloud.google.com/apigee/docs/api-platform/get-started/provisioning-intro) + before running `provision.sh`. +2. **The API hub instance.** Provisioning takes 40+ minutes and + asks for encryption (CMEK vs GMEK) and Vertex region. + [Enable API hub](https://cloud.google.com/apigee/docs/apihub/provision) + before running `provision.sh`. + +You also need: + +- IAM `roles/owner` on the target project (or the fine-grained + equivalent: `apihub.editor`, `apigee.admin`, + `storage.admin`, `serviceusage.serviceUsageAdmin`). +- `gcloud`, `python3` (≥ 3.10), and `curl` on your machine. +- ADC set up: `gcloud auth application-default login`. +- A clone of `apigee/devrel` — the provisioning script runs from + `references/apigee-skills-serving/`. + +## The five-minute path + +For an operator who wants to get from empty project to working +demo as fast as possible: + +```bash +# 1. Install skill-finder + skill-publisher on your machine. +curl -fsSL https://raw.githubusercontent.com/apigee/devrel/main/\ +references/apigee-skills-serving/bin/install-skill-finder.sh \ + | bash -s -- --runtime gemini # or --runtime opencode / antigravity + +# 2. Clone the reference and run the provisioner. +git clone https://github.com/apigee/devrel.git +cd devrel/references/apigee-skills-serving +bash bin/provision.sh --project MY_PROJECT --yes +``` + +After ~2-3 minutes, `provision.sh` prints a `READY` banner. Launch +your agent and ask a question that matches one of the example +skills' keywords ("convert 100 USD to EUR", "what's the weather in +Tokyo", "what apigee policies should I optimize"). skill-finder +will find, verify, and install the matching skill. + +## What `provision.sh` does step by step + +### Step 0: preflight + +Verifies: +- Required tools (`gcloud`, `python3`, `curl`) are on PATH. +- ADC is set up and returns a token. +- A per-user Python venv exists at + `~/.local/share/skill-finder/venv` (creates one if not; the same + venv is shared with `install-skill-finder.sh`, so installing + skill-finder first speeds provisioning up). + +### Step 1: enable GCP APIs + +Enables (if not already enabled): `apigee.googleapis.com`, +`apihub.googleapis.com`, `storage.googleapis.com`, +`iam.googleapis.com`. Skips APIs that are already on. + +### Step 2: verify prereqs + +Confirms the API hub instance exists in the target project + +location. Confirms the Apigee organisation exists. If either is +missing, fails fast with a link to the console page where you can +provision it. + +### Step 3: GCS bucket + +Creates a public-read GCS bucket named +`-skills-` (or the `--bucket` value). Grants +`allUsers:objectViewer` — the demo requires public read because +skill-finder downloads bundles via anonymous HTTPS (`https:// +storage.googleapis.com//.skill`), no ADC scope. + +**Production note**: this is a demo-only choice. In production +you would prefer signed URLs or an authenticated consumer service +account. skill-finder currently supports only anonymous HTTPS; +adding ADC-authenticated download would be a small change to +`find_install.py`. + +### Step 4: API hub attribute taxonomy + +Creates the four user-defined attributes skill-finder queries and +skill-publisher writes: + +- `agentic_skill` (string, "true" | "false") — the filter + skill-finder uses to enumerate agent-consumable APIs. +- `keywords` (string, multi-valued) — the tokens skill-finder + ranks against the user's query. +- `gs_uri` (string) — GCS URI of the `.skill` bundle. +- `signing_key_id` (string) — sha256 fingerprint of the signing + public key. + +Also patches `skill-spec` into the `system-spec-type` enum. This +is a known gap: `scripts/update_taxonomy.py` alone does not add +`skill-spec` (because it's a system-defined attribute, not user- +defined), but `register_skill.py` requires it. `provision.sh` +patches it via a direct API hub `PATCH` call. + +### Step 5: ed25519 signing keypair + +Generates a fresh keypair on your machine and writes the private +key to `~/.config/apigee-skills-demo/signing.raw` (mode 0600). +Prints the SHA-256 fingerprint of the public key. + +If a key already exists at that path, `provision.sh` reuses it +by default. Pass `--force` to regenerate — this invalidates all +existing signatures and requires re-publishing every skill. + +### Step 6: publish example skills + +For each of `currency-converter`, `weather-lookup`, +`apigee-policy-top10`: + +1. Copies the source manifest into a workdir at + `~/.local/share/apigee-skills-serving/publish-workdir/`. +2. Patches in the `gs_uri` (the source manifest ships without one; + `sign_skill.py` requires it to be present at sign time + because the signature covers `gs_uri`). +3. Runs `pack_skill.py` to produce a `.skill` zip in `/tmp/`. +4. Runs `upload_skill.py` to push the zip to GCS. +5. Runs `sign_skill.py` to produce a signed manifest in the + workdir. +6. Runs `register_skill.py` to register the signed manifest with + API hub as an API + Version + Spec triple. + +Skills that are already registered are detected and skipped — +re-running `provision.sh` against a fully-provisioned project +takes ~10 seconds and produces zero writes. + +### Step 7: install trust root into skill-finder + +Auto-detects the agent runtime (Gemini CLI, OpenCode, or +Antigravity) by looking for +`~/.gemini/skills/skill-finder/`, +`~/.config/opencode/skills/skill-finder/`, or +`~/.gemini/antigravity/skills/skill-finder/`. If any of them +exists, drops the public key PEM (with the fingerprint as the +filename) into that install's `keys/` directory. skill-finder +will trust the just-published catalog on its next invocation. + +If skill-finder isn't installed anywhere yet, `provision.sh` +prints the public PEM to stdout with instructions for the +operator to save it manually. The example skills are still +published — only the local trust-root install is skipped. + +## Common workflows + +### Add another publisher (multi-key trust) + +skill-finder's `keys/` directory is a set: any file matching +`*.pem` is loaded. To add a second publisher without disrupting +the first: + +1. The second publisher generates their own Ed25519 keypair on + their machine. +2. They send you (out of band) the public PEM. +3. You drop it into `~/./skills/skill-finder/keys/.pem`. +4. Every skill they sign with their key now installs cleanly + alongside skills signed by the original key. + +### Key rotation with zero downtime + +1. Generate the new key: `bash bin/provision.sh --project X + --force` (with `--yes` if you want no prompts). This + overwrites `signing.raw`, re-publishes all example skills + signed with the new key, and drops the new pubkey into + `keys/`. The **old** pubkey is still in `keys/` too, so + old-signed manifests keep verifying. +2. Republish any of your own skills with the new key. +3. When you're confident no one is still using old-signed + manifests, delete the old PEM from `keys/`. + +### Rebuild a partially-provisioned project + +`provision.sh` is idempotent. Just re-run it. Every step detects +existing state and skips work that isn't needed. If you want to +force one specific step to re-run: + +```bash +# Force re-publish (drops the "already published" fast-path): +bash bin/provision.sh --project X --skip-apis --skip-bucket \ + --skip-taxonomy --skip-key --skip-trust-root-install --yes +# You'll need to delete + re-register each skill from API hub by +# hand to force a real re-publish; the fast-path skips +# publishing when API hub already lists the skill. Better to +# fix whatever went wrong in-place with individual +# scripts/register_skill.py invocations. +``` + +### Remove a deployment + +`provision.sh` has no `--uninstall`. To tear down: + +```bash +# Delete registered APIs from API hub. +gcloud apihub apis delete currency-converter \ + --project X --location us-west1 +gcloud apihub apis delete weather-lookup \ + --project X --location us-west1 +gcloud apihub apis delete apigee-policy-top10 \ + --project X --location us-west1 + +# Delete .skill objects from GCS. +gcloud storage rm gs:///currency-converter.skill +gcloud storage rm gs:///weather-lookup.skill +gcloud storage rm gs:///apigee-policy-top10.skill + +# Optionally delete the bucket + attributes + APIs. Note that +# deleting an API hub instance is a separate operation that +# takes 40+ minutes. + +# Delete the local trust root. +rm ~/.gemini/skills/skill-finder/keys/*.pem + +# Delete the private key. +rm ~/.config/apigee-skills-demo/signing.raw +``` + +## Troubleshooting + +### `provision.sh` says "no API hub instance in ..." + +The instance doesn't exist yet in that project + location. Follow +the console link in the error message; provisioning takes 40+ +minutes. Re-run `provision.sh` when the instance state is `ACTIVE`. + +### `provision.sh` says "Apigee organisation ... not accessible" + +Either the org doesn't exist in that project, or your caller +lacks `apigee.organizations.get`. If you're not planning to use +the `apigee-policy-top10` skill (which is the only one that +needs an Apigee org), pass `--skip-skills` to provision the +infrastructure without publishing that skill. + +### skill-finder says `trust root: FAILED — no valid keys loaded` + +`~/./skills/skill-finder/keys/` is empty. Either: + +- Re-run `provision.sh` (it will detect that skill-finder is + now installed and drop the PEM in during Step 7), or +- Manually copy the PEM from another operator's install. + +### skill-finder says `key-id check: FAILED — manifest declares X, but skill-finder only trusts: Y` + +The signing key that signed the manifest is different from any +key in your local `keys/`. Either your `provision.sh` used a +different key (check the fingerprint in the READY banner), or a +different publisher signed the skill. Get their public PEM, +drop it into `keys/`, and retry. + +### `register_skill.py` in Step 6 fails with `attribute system-spec-type does not have any allowed value with id: skill-spec` + +Step 4 didn't run (or was skipped). Re-run `provision.sh` without +`--skip-taxonomy`; it will PATCH the enum to include `skill-spec`. + +### After provisioning, my agent still doesn't find any skills + +Two possibilities: + +- skill-finder isn't installed in your agent runtime. Run + `bash bin/install-skill-finder.sh --runtime `. +- The trust root PEM isn't in the runtime's skills dir. Re-run + `provision.sh` with `--skip-apis --skip-bucket --skip-taxonomy + --skip-skills --skip-key` (i.e. run only Step 7). diff --git a/references/apigee-skills-serving/docs/publish-and-install.md b/references/apigee-skills-serving/docs/publish-and-install.md index d1bf9dad..5f001a24 100644 --- a/references/apigee-skills-serving/docs/publish-and-install.md +++ b/references/apigee-skills-serving/docs/publish-and-install.md @@ -1,9 +1,41 @@ # Publish and install walkthrough This walkthrough takes you from a fresh checkout to a signed, -registered skill that an agent runtime can install from API hub. The -examples use the bundled `skills/currency-converter` as the skill being -published. +registered skill that an agent runtime can install from API hub. +The examples use the bundled `skills/currency-converter` as the +skill being published. + +## For most people: use `bin/provision.sh` + +If you just want a working reference deployment, skip the rest +of this document and read [`provisioning.md`](provisioning.md). +The five-minute path is: + +```bash +# 1. Install skill-finder on your machine +curl -fsSL https://raw.githubusercontent.com/apigee/devrel/main/\ +references/apigee-skills-serving/bin/install-skill-finder.sh \ + | bash -s -- --runtime gemini + +# 2. Clone the reference and provision your project +git clone https://github.com/apigee/devrel.git +cd devrel/references/apigee-skills-serving +bash bin/provision.sh --project MY_PROJECT --yes +``` + +`provision.sh` runs everything in this walkthrough end to end: +enables APIs, creates the GCS bucket, sets up the taxonomy, +generates a fresh signing keypair on your machine, and publishes +the three example skills. It is idempotent — re-run it any time +to reconcile drift. + +**Keep reading this document if:** + +- You want to understand what `provision.sh` does under the hood. +- You want to publish your own skill (not one of the bundled + examples). See especially section 6 (Pack) and section 7 (Sign) + for the per-skill inputs. +- You are troubleshooting a failure at a specific step. ## 1. Prerequisites @@ -187,38 +219,65 @@ You should see your `currency-converter` entry in the list. ## 11. Consumer side: install the skill -A consumer (an agent runtime, or a developer running `pip install` -analog for skills) performs the inverse flow: +The consumer-side reference implementation lives at +`skills/skill-finder/scripts/find_install.py`. It performs the +inverse flow: -1. Searches API hub by keyword overlap on the skill description, or - by attribute (e.g., "every skill with `skill-runtime-iam` containing - `apigee.proxies.create`"). -2. Fetches the chosen entry's manifest from API hub. +1. Searches API hub for APIs where the `agentic_skill` attribute + is `"true"`, ranks results by keyword overlap on the user's + query, picks the top match. +2. Fetches the chosen entry's manifest (base64-encoded in the + API hub Spec resource). 3. Re-canonicalises the manifest (excluding signature fields), - verifies the Ed25519 signature against the publisher's known public - key. -4. Downloads the `.skill` zip from `gs_uri`. -5. Computes SHA-256 and matches it against `zip_sha256` from the - manifest. -6. Extracts the `.skill` into the consumer's skills directory - (e.g., `~/.config/opencode/skills/{skill-name}/`). - -The consumer-side `find_install.py` reference implementation is **not** -included in this PR; the contract it consumes is fully documented above -and exercised by the test suite. See -`tests/test_register_fetch_integration.py` for a stateful in-process -example. + verifies the Ed25519 signature against **every trust root** + under `~/./skills/skill-finder/keys/*.pem`. Accepts + the manifest if the `signing_key_id` matches any installed + trust root (multi-key model — see + [`architecture.md#multi-key-trust-roots`](architecture.md#multi-key-trust-roots)). +4. Runs an IAM pre-flight against the skill's declared + `runtime_iam` permissions — fails fast if the caller lacks + any of them. +5. Downloads the `.skill` zip from `gs_uri` via anonymous HTTPS + (the bucket must be public-read; see + [`architecture.md#trust-model`](architecture.md#trust-model)). +6. Computes SHA-256 and matches it against `zip_sha256` from + the manifest. +7. Extracts the `.skill` into the consumer's skills directory + (`~/.gemini/skills/{skill-name}/`, + `~/.config/opencode/skills/{skill-name}/`, or + `~/.gemini/antigravity/skills/{skill-name}/` per runtime). + +The whole flow is emitted as a locked 16-line +[Hyrum's Law contract](https://www.hyrumslaw.com/) on stdout so +the agent runtime can surface each verification step to the user +verbatim. + +To install skill-finder itself, use +`bin/install-skill-finder.sh` (see +[`provisioning.md`](provisioning.md)). ## Cleanup -To remove the demo artifacts: +To remove the demo artifacts installed by skill-finder: ```bash ./bin/demo-cleanup.sh ``` -This removes locally extracted skills under +This removes locally extracted PAYLOAD skills under `~/.config/opencode/skills/{currency-converter,weather-lookup, -apigee-policy-top10}/`. It does **not** delete anything from API hub -or your GCS bucket — those are remote and the cleanup intentionally -stays local to avoid surprising side-effects in shared projects. +apigee-policy-top10}/`. It does **not** delete anything from API +hub or your GCS bucket, and it does **not** remove skill-finder +or skill-publisher themselves — those are meant to persist +across demo runs. + +To remove skill-finder + skill-publisher too: + +```bash +rm -rf ~/.gemini/skills/skill-finder ~/.gemini/skills/skill-publisher +# or for opencode: +rm -rf ~/.config/opencode/skills/skill-finder ~/.config/opencode/skills/skill-publisher +``` + +For a full teardown (including remote resources), see +[`provisioning.md#remove-a-deployment`](provisioning.md#remove-a-deployment). diff --git a/references/apigee-skills-serving/skills/skill-finder/SKILL.md b/references/apigee-skills-serving/skills/skill-finder/SKILL.md new file mode 100644 index 00000000..e74aad95 --- /dev/null +++ b/references/apigee-skills-serving/skills/skill-finder/SKILL.md @@ -0,0 +1,178 @@ +--- +name: skill-finder +description: Discovers and installs agentic skills from the customer + API hub. Two modes — (1) SEARCH AND INSTALL one matching skill by + natural-language query, verifying its ed25519 signature against + the trust root(s) installed under keys/; (2) LIST every available + skill in the catalog as a markdown table, with cursor pagination, + so the user can browse what's offered before installing. +license: Apache-2.0 +compatibility: opencode, antigravity, gemini-cli +metadata: + # Trust roots are per-deployment: each install has one or more + # .pem files under keys/, populated at install time + # by bin/provision.sh or by the operator. See keys/README.md. + trusted_signing_key_ids: keys/*.pem +--- + +# skill-finder + +## ⚠️ Runtime requirements (read this first) + +This skill **requires** its bundled venv wrapper. The Python +scripts depend on `cryptography`, `google-auth`, `requests`, and +`pyyaml`, which are installed into a per-user venv by the +installer (`bin/install-skill-finder.sh`). They are **not** +available to the system `python3`. + +**Always invoke scripts via the wrapper, never via bare +`python3`:** + +```bash +# CORRECT - invokes the venv Python via the wrapper: +${SKILL_DIR}/bin/run-with-venv.sh ${SKILL_DIR}/scripts/