diff --git a/CHANGELOG.md b/CHANGELOG.md index 5650d62..57b713f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ First OpenCodeHighEnd release. New product on OpenCode 2. Not OpenCodeBestFriend - Installer fails closed on OpenCode 1.x. Gate is major `>= 2`. - New identity: CLI `opencode-he`, overlay `~/.config/opencode/highend`, share `~/.local/share/opencode-highend`, AGENTS markers `OPENCODEHIGHEND:BEGIN/END`. - V1 plugins are not copied. `lsp` is not ported. Design Bank / Design V2 / SmartDoc remain user data, never git media. +- Design Bank bootstrap: local valid bank first; `OPENCODE_DESIGN_BANK_URL` + `OPENCODE_DESIGN_BANK_SHA256` (URL without SHA fails closed); default Drive ZIP pin; GitHub `Design-bank.tgz` fallback. Uninstall never deletes `~/Design` or `~/DesignV2`. - Legacy V1 plugins (`impeccable-live-poll.ts`) quarantined to `~/.local/share/opencode-highend/quarantine/plugins/`; doctor reports `V1_PLUGIN_LEFTOVER`. - FOREIGN_ON_DEMAND MCP stay enable-gated: serena, stitch, reticle, ui-skills, markitdown; exa is never added/removed/overwritten. - Retired twins stay retired: `ask-matt`, `grilling`, `wait-what`, `matt-implement`. diff --git a/README.md b/README.md index a9b295d..9c9e972 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,8 @@ # OpenCodeHighEnd -Production-ready capability layer for OpenCode: -62 routed skills (core + Wave 2/3 warehouse specialists), MCP, Codebase Memory, -Design Bank, Design Intelligence, SmartDoc, browser and verification tooling. +OpenCode 2 overlay: 62 frozen routed skills, thin `AGENTS.md`, `opencode-he`. -OpenCodeHighEnd is an installer and runtime overlay for [OpenCode 2](https://opencode.ai/v2/docs/). It is **not** Claude Code, **not** GrokBuild, **not** OpenCodeBestFriend runtime, **not** a model provider, and **not** a dump of a developer home directory. +Installer and runtime overlay for [OpenCode 2](https://opencode.ai/v2/docs/). It is **not** Claude Code, **not** GrokBuild, **not** OpenCodeBestFriend runtime, **not** a model provider, and **not** a dump of a developer home directory. Version **0.1.0**. The 62-skill catalog is inherited from OpenCodeBestFriend 1.8.6 (`67142e4` / PR #29) and stays frozen. This is a new product on a new host. @@ -39,6 +37,9 @@ cd OpenCodeHighEnd # optional: acquire the full user-owned Design Bank and build DesignV2 ./install.sh --with-design-bank +# or after install: +# opencode-he design bootstrap +# OPENCODE_DESIGN_BANK_URL=... OPENCODE_DESIGN_BANK_SHA256=... opencode-he design bootstrap # pick up OPENCODE_DISABLE_CLAUDE_CODE=1 exec "$SHELL" @@ -182,11 +183,18 @@ Normal `./install.sh` installs the engine only and never starts the multi-gigaby ```bash ./install.sh --with-design-bank -# or after installation opencode-he design bootstrap +OPENCODE_DESIGN_BANK_URL=... OPENCODE_DESIGN_BANK_SHA256=... opencode-he design bootstrap ``` -Bootstrap resolves `OPENCODE_DESIGN_BANK` → existing pointer → `~/Design`. It downloads the declared public artifact with curl, verifies SHA-256, safely extracts into a temporary directory, validates all four catalogs, and commits the bank without merging into an existing directory. `~/Design` and `~/DesignV2` are user data and uninstall never removes them. Google Drive is contacted only by bootstrap; retrieval remains offline. +Two download sources (SHA-256 fail-closed; URL without SHA is refused): + +1. **Default Drive pin** in `lib/design_v2/bootstrap_sources.json` (ZIP). Google Drive is contacted only during bootstrap. +2. **Fallback GitHub artifact** in `vendor/sources.json` (`GrokBestFriend` `Design-bank.tgz`, sha256 `9866f5a8…`). Used when the Drive pin is unavailable. + +Operator override: `OPENCODE_DESIGN_BANK_URL` + `OPENCODE_DESIGN_BANK_SHA256`. Drive view links (`/file/d/ID/view`) resolve to `uc?export=download`. A valid local bank (four catalogs at `OPENCODE_DESIGN_BANK` or `~/Design`) is used as-is — no download. + +Bootstrap verifies SHA-256, extracts to a temp directory, validates 21st / Aura / Refero / Motionsites catalogs, then commits into `~/Design` or `OPENCODE_DESIGN_BANK`. Uninstall never deletes `~/Design` or `~/DesignV2`. After bootstrap, retrieval stays offline. ## Design Intelligence diff --git a/docs/design-bank.md b/docs/design-bank.md index 68eebcf..ad02f48 100644 --- a/docs/design-bank.md +++ b/docs/design-bank.md @@ -11,23 +11,51 @@ motionsites/library/catalog.json Normal `./install.sh` installs the engine only. It does not download Design Bank media. -Run the optional bootstrap during or after install: - ```bash ./install.sh --with-design-bank opencode-he design bootstrap +OPENCODE_DESIGN_BANK_URL=... OPENCODE_DESIGN_BANK_SHA256=... opencode-he design bootstrap +``` + +## Priority + +1. Valid local bank (`OPENCODE_DESIGN_BANK` or `~/Design` with all four catalogs) → `already_present`, no download. +2. `OPENCODE_DESIGN_BANK_URL` + `OPENCODE_DESIGN_BANK_SHA256`. URL without SHA-256 fails closed; nothing is downloaded. +3. Default Google Drive ZIP pin in `lib/design_v2/bootstrap_sources.json`. +4. Fallback GitHub `.tgz` in `vendor/sources.json` (`design-bank.artifactUrl` + `artifactSha256`). + +The target and generated `~/DesignV2` are user data, not installer-owned. Uninstall never deletes them. + +## Operator Drive archive + +Publish the ZIP (or `.tgz`) on Google Drive as anyone-with-the-link, or keep it restricted and download it yourself then point `OPENCODE_DESIGN_BANK` at the extracted tree. + +Compute the digest of the **archive file**, not of a folder: + +```bash +sha256sum OpenCodeHighEnd-DesignBank-v1.zip ``` -Bootstrap target order: `OPENCODE_DESIGN_BANK` → existing supported pointer → `~/Design`. The target and generated `~/DesignV2` are user data, not installer-owned. +Accepted URL shapes: + +- `https://drive.google.com/uc?export=download&id=FILE_ID` +- `https://drive.google.com/file/d/FILE_ID/view` +- a direct `https://` artifact -The source declaration is `lib/design_v2/bootstrap_sources.json`. Network access is limited to checksum and archive acquisition. The archive is downloaded with curl, checked against both the downloaded checksum and the pinned SHA-256, bounded and checked for unsafe ZIP members, extracted to a temporary sibling, validated, then atomically committed. A healthy existing target returns `already_present`; an incompatible existing target is never overwritten. +View links are rewritten to `uc?export=download`. Large Drive files may hit a virus-scan confirm page; bootstrap follows that token **once** with a temporary cookie jar. The jar is never written into git or `~/.config`. -The local pointer remains: +Drive is first-hop only. After the bank is committed, search, shortlist, inspect, doctor, dedupe, and rebuild stay offline. Aura and 21st live sites are never fetched. + +There is no gcloud user OAuth in the installer. Do not put the tarball in this git repository. + +## Pointer ```text ~/.config/opencode/highend/config/design-bank.json ``` -After commit, existing DesignV2 APIs pointer-ingest Refero, Motionsites, 21st, and Aura, then dedupe, rebuild, and doctor. Preview media remains only under the Design root. Search, shortlist, inspect, doctor, dedupe, and rebuild do not contact the network. +Missing bank is `DEGRADED` in `opencode-he doctor`, not `FAIL`. + +After commit, Design V2 pointer-ingests Refero, Motionsites, 21st, and Aura, then dedupes, rebuilds, and doctors. Preview media remains only under the Design root. Design Intelligence ships in-tree (`design-intelligence/`) and stays lazy inside Impeccable. diff --git a/lib/design_v2/bootstrap.py b/lib/design_v2/bootstrap.py index 4ac9548..b5c4580 100644 --- a/lib/design_v2/bootstrap.py +++ b/lib/design_v2/bootstrap.py @@ -1,20 +1,23 @@ from __future__ import annotations +import html import json import os import re import shutil import stat import subprocess +import tarfile import tempfile import zipfile from collections.abc import Callable from dataclasses import dataclass from pathlib import Path from typing import Any -from urllib.parse import urlencode +from urllib.parse import urlencode, urlparse -from ..common import he_dir, home, sha256_file, share_dir, write_json +from ..common import he_dir, home, repo_root, sha256_file, share_dir, write_json +from ..paths import tar_member_ok from . import FTS_SCHEMA_VERSION, PACKAGE_DIR from .bank import DesignV2Error, resolve_design_v2_root from .commands import bank_health, doctor_rows @@ -33,8 +36,18 @@ SOURCE_CONFIG = PACKAGE_DIR / "bootstrap_sources.json" SOURCE_NAME_RE = re.compile(r"^[a-z0-9][a-z0-9._-]{0,63}$") DRIVE_FILE_ID_RE = re.compile(r"^[A-Za-z0-9_-]{10,128}$") +SHA256_HEX_RE = re.compile(r"^[0-9A-Fa-f]{64}$") SHA256_RE = re.compile(r"^([0-9A-Fa-f]{64})[ \t]+\*?([^\r\n]+)$") +DRIVE_VIEW_RE = re.compile(r"https?://(?:drive|docs)\.google\.com/file/d/([A-Za-z0-9_-]{10,128})") +DRIVE_ID_QUERY_RE = re.compile(r"[?&]id=([A-Za-z0-9_-]{10,128})") +DRIVE_CONFIRM_RE = re.compile(r"[?&]confirm=([0-9A-Za-z_-]+)") +DRIVE_CONFIRM_INPUT_RE = re.compile( + r'name=["\']confirm["\'][^>]*value=["\']([^"\']+)["\']|value=["\']([^"\']+)["\'][^>]*name=["\']confirm["\']', + re.I, +) ZIP_MAGIC = (b"PK\x03\x04", b"PK\x05\x06", b"PK\x07\x08") +GZIP_MAGIC = b"\x1f\x8b" +DRIVE_HOSTS = ("drive.google.com", "drive.usercontent.google.com", "docs.google.com") BOOTSTRAP_ZIP_LIMITS = { "max_members": 150_000, "max_member_uncompressed": 1 << 30, @@ -160,15 +173,141 @@ def google_drive_public_url(file_id: str) -> str: return f"https://drive.usercontent.google.com/download?{query}" -def _curl_download(url: str, destination: Path) -> None: - curl = shutil.which("curl") - if not curl: - raise BootstrapError("PREFLIGHT", "curl is required", code="CURL_MISSING") +def resolve_operator_url(url: str) -> str: + raw = (url or "").strip() + if not raw.startswith("https://"): + raise BootstrapError("SOURCE_RESOLVED", "URL must be https", code="DESIGN_BANK_URL_INVALID") + view = DRIVE_VIEW_RE.search(raw) + if view: + return f"https://drive.google.com/uc?export=download&id={view.group(1)}" + host = (urlparse(raw).hostname or "").lower() + if host in DRIVE_HOSTS: + found = DRIVE_ID_QUERY_RE.search(raw) + if found: + return f"https://drive.google.com/uc?export=download&id={found.group(1)}" + return raw + + +def env_design_bank_override() -> tuple[str, str] | None: + url = (os.environ.get("OPENCODE_DESIGN_BANK_URL") or "").strip() + sha = (os.environ.get("OPENCODE_DESIGN_BANK_SHA256") or "").strip() + if not url: + return None + if not SHA256_HEX_RE.fullmatch(sha): + raise BootstrapError( + "SOURCE_RESOLVED", + "OPENCODE_DESIGN_BANK_SHA256 is required", + code="SHA256_REQUIRED", + ) + return url, sha.lower() + + +def _archive_name_from_url(url: str, default: str) -> str: + host = (urlparse(url).hostname or "").lower() + if host in DRIVE_HOSTS: + return default + name = Path(urlparse(url).path).name + if not name or name in {".", "..", "uc", "download", "view"} or "/" in name or "\\" in name: + return default + return name + + +def operator_url_source(url: str, sha256: str) -> tuple[BootstrapSource, str]: + resolved = resolve_operator_url(url) + return ( + BootstrapSource( + name="env-url", + source_type="operator-url", + bank_version="env", + archive_name=_archive_name_from_url(resolved, "design-bank.bin"), + archive_file_id="", + checksum_file_id="", + pinned_sha256=sha256.lower(), + ), + resolved, + ) + + +def github_fallback_source() -> tuple[BootstrapSource, str]: + path = repo_root() / "vendor" / "sources.json" + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise BootstrapError("SOURCE_RESOLVED", "vendor sources unreadable", code="BOOTSTRAP_SOURCE_INVALID") from exc + block = ((payload.get("sources") or {}) if isinstance(payload, dict) else {}).get("design-bank") + if not isinstance(block, dict): + raise BootstrapError("SOURCE_RESOLVED", "github fallback missing", code="BOOTSTRAP_SOURCE_INVALID") + url = block.get("artifactUrl") + sha = block.get("artifactSha256") + if not isinstance(url, str) or not url.startswith("https://"): + raise BootstrapError("SOURCE_RESOLVED", "github fallback URL", code="BOOTSTRAP_SOURCE_INVALID") + if not isinstance(sha, str) or not SHA256_HEX_RE.fullmatch(sha): + raise BootstrapError("SOURCE_RESOLVED", "github fallback SHA-256", code="BOOTSTRAP_SOURCE_INVALID") + return ( + BootstrapSource( + name="github-release-fallback", + source_type="https-artifact", + bank_version=str(block.get("version") or "fallback"), + archive_name=_archive_name_from_url(url, "Design-bank.tgz"), + archive_file_id="", + checksum_file_id="", + pinned_sha256=sha.lower(), + ), + url, + ) + + +def select_remote_source( + source_name: str | None = None, *, config_path: Path | None = None +) -> tuple[BootstrapSource, str | None, str]: + env = env_design_bank_override() + if env: + source, url = operator_url_source(env[0], env[1]) + return source, url, "curl-operator-url" + try: + source = resolve_bootstrap_source(source_name, config_path=config_path) + return source, None, "curl-google-drive-public" + except BootstrapError: + if source_name: + raise + source, url = github_fallback_source() + return source, url, "curl-github-release" + + +def _is_html_file(path: Path) -> bool: + try: + head = path.read_bytes()[:512].lstrip().lower() + except OSError: + return False + return head.startswith(b" str | None: + text = html.unescape(page) + tokens = DRIVE_CONFIRM_RE.findall(text) + for match in DRIVE_CONFIRM_INPUT_RE.findall(text): + tokens.extend(part for part in match if part) + confirm = next((token for token in tokens if token and token != "t"), tokens[0] if tokens else None) + if not confirm: + return None + found = DRIVE_ID_QUERY_RE.search(original_url) or DRIVE_ID_QUERY_RE.search(text) + file_id = found.group(1) if found else None + if file_id and DRIVE_FILE_ID_RE.fullmatch(file_id): + return f"https://drive.google.com/uc?export=download&id={file_id}&confirm={confirm}" + sep = "&" if "?" in original_url else "?" + return f"{original_url}{sep}confirm={confirm}" + + +def _is_drive_url(url: str) -> bool: + host = (urlparse(url).hostname or "").lower() + return host in DRIVE_HOSTS + + +def _run_curl(curl: str, url: str, destination: Path, cookie_jar: Path | None) -> None: destination.parent.mkdir(parents=True, exist_ok=True) partial = destination.with_name(destination.name + ".part") command = [ curl, - "--fail", "--location", "--silent", "--show-error", @@ -181,23 +320,17 @@ def _curl_download(url: str, destination: Path) -> None: "30", "--output", str(partial), - url, ] - if partial.is_file() and partial.stat().st_size: - command[1:1] = ["--continue-at", "-"] + if cookie_jar is not None: + command.extend(["--cookie", str(cookie_jar), "--cookie-jar", str(cookie_jar)]) + command.append(url) try: result = subprocess.run(command, capture_output=True, text=True) except OSError as exc: partial.unlink(missing_ok=True) raise BootstrapError("ARCHIVE_DOWNLOADED", type(exc).__name__, code="DOWNLOAD_FAILED") from exc - if result.returncode != 0 and "--continue-at" in command: - partial.unlink(missing_ok=True) - command[1:3] = [] - try: - result = subprocess.run(command, capture_output=True, text=True) - except OSError as exc: - raise BootstrapError("ARCHIVE_DOWNLOADED", type(exc).__name__, code="DOWNLOAD_FAILED") from exc if result.returncode != 0: + partial.unlink(missing_ok=True) detail = (result.stderr or "curl failed").strip().splitlines()[-1] raise BootstrapError("ARCHIVE_DOWNLOADED", detail, code="DOWNLOAD_FAILED") if partial.is_symlink() or not partial.is_file() or partial.stat().st_size == 0: @@ -206,6 +339,33 @@ def _curl_download(url: str, destination: Path) -> None: os.replace(partial, destination) +def _curl_download(url: str, destination: Path) -> None: + curl = shutil.which("curl") + if not curl: + raise BootstrapError("PREFLIGHT", "curl is required", code="CURL_MISSING") + if not _is_drive_url(url): + _run_curl(curl, url, destination, None) + return + with tempfile.TemporaryDirectory(prefix="opencode-he-drive-") as tmp: + cookie_jar = Path(tmp) / "cookies" + _run_curl(curl, url, destination, cookie_jar) + if not _is_html_file(destination): + return + try: + page = destination.read_text(encoding="utf-8", errors="ignore") + except OSError as exc: + destination.unlink(missing_ok=True) + raise BootstrapError("ARCHIVE_DOWNLOADED", "unreadable download", code="DOWNLOAD_FAILED") from exc + confirm = drive_confirm_url(url, page) + destination.unlink(missing_ok=True) + if not confirm: + raise BootstrapError("ARCHIVE_DOWNLOADED", "Google Drive confirm page", code="DOWNLOAD_FAILED") + _run_curl(curl, confirm, destination, cookie_jar) + if _is_html_file(destination): + destination.unlink(missing_ok=True) + raise BootstrapError("ARCHIVE_DOWNLOADED", "Google Drive confirm page", code="DOWNLOAD_FAILED") + + def download_public_file(url: str, destination: Path) -> None: _curl_download(url, destination) @@ -320,8 +480,129 @@ def safe_extract_bootstrap_zip(path: Path, destination: Path) -> dict[str, int]: return stats +def inspect_bootstrap_tar(path: Path) -> dict[str, int]: + try: + handle = tarfile.open(path, mode="r:*") + except (OSError, tarfile.TarError) as exc: + raise BootstrapError("ARCHIVE_INSPECTED", "invalid tar archive", code="ARCHIVE_INVALID") from exc + limits = BOOTSTRAP_ZIP_LIMITS + try: + members = handle.getmembers() + if not members or len(members) > int(limits["max_members"]): + raise BootstrapError("ARCHIVE_INSPECTED", "tar member limit", code="ARCHIVE_UNSAFE") + seen: set[str] = set() + total = 0 + files = 0 + for member in members: + name = (member.name or "").replace("\\", "/") + normalized = name.rstrip("/") + if normalized in {"", "."}: + continue + if "\x00" in name or not member_ok(normalized) or ".." in Path(name).parts: + raise BootstrapError("ARCHIVE_INSPECTED", f"unsafe tar path {name}", code="ARCHIVE_UNSAFE") + if len(name) > int(limits["max_path_length"]) or len(Path(name).parts) > int(limits["max_path_depth"]): + raise BootstrapError("ARCHIVE_INSPECTED", "tar path limit", code="ARCHIVE_UNSAFE") + if normalized in seen: + raise BootstrapError("ARCHIVE_INSPECTED", f"duplicate tar path {normalized}", code="ARCHIVE_UNSAFE") + seen.add(normalized) + if member.issym() or member.islnk() or not (member.isfile() or member.isdir()): + raise BootstrapError("ARCHIVE_INSPECTED", f"special tar member {name}", code="ARCHIVE_UNSAFE") + if member.isdir(): + continue + files += 1 + uncompressed = int(member.size) + if uncompressed > int(limits["max_member_uncompressed"]): + raise BootstrapError("ARCHIVE_INSPECTED", "tar member size limit", code="ARCHIVE_UNSAFE") + total += uncompressed + if total > int(limits["max_total_uncompressed"]): + raise BootstrapError("ARCHIVE_INSPECTED", "tar total size limit", code="ARCHIVE_UNSAFE") + return {"members": len(members), "files": files, "uncompressed_bytes": total} + finally: + handle.close() + + +def safe_extract_bootstrap_tar(path: Path, destination: Path) -> dict[str, int]: + stats = inspect_bootstrap_tar(path) + destination.mkdir(parents=True, exist_ok=False) + base = destination.resolve() + try: + with tarfile.open(path, mode="r:*") as handle: + for member in handle.getmembers(): + name = (member.name or "").replace("\\", "/") + if name.rstrip("/") in {"", "."}: + continue + if not tar_member_ok(destination, name): + raise BootstrapError( + "EXTRACTED_TO_TEMP", f"unsafe tar path {name}", code="ARCHIVE_EXTRACTION_FAILED" + ) + target = destination / name.rstrip("/") + try: + target.resolve(strict=False).relative_to(base) + except (OSError, ValueError) as exc: + raise BootstrapError( + "EXTRACTED_TO_TEMP", f"unsafe tar path {name}", code="ARCHIVE_EXTRACTION_FAILED" + ) from exc + if member.isdir(): + target.mkdir(parents=True, exist_ok=True) + continue + if not member.isfile(): + raise BootstrapError( + "EXTRACTED_TO_TEMP", f"special tar member {name}", code="ARCHIVE_EXTRACTION_FAILED" + ) + target.parent.mkdir(parents=True, exist_ok=True) + reader = handle.extractfile(member) + if reader is None: + raise BootstrapError( + "EXTRACTED_TO_TEMP", f"failed to extract {name}", code="ARCHIVE_EXTRACTION_FAILED" + ) + try: + with reader, target.open("xb") as writer: + shutil.copyfileobj(reader, writer, length=1 << 20) + except OSError as exc: + raise BootstrapError( + "EXTRACTED_TO_TEMP", f"failed to extract {name}", code="ARCHIVE_EXTRACTION_FAILED" + ) from exc + st = target.lstat() + if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1 or st.st_size != member.size: + raise BootstrapError( + "EXTRACTED_TO_TEMP", f"unsafe extracted file {name}", code="ARCHIVE_EXTRACTION_FAILED" + ) + except BootstrapError: + raise + except (OSError, tarfile.TarError) as exc: + raise BootstrapError("EXTRACTED_TO_TEMP", "tar extraction failed", code="ARCHIVE_EXTRACTION_FAILED") from exc + return stats + + +def _archive_magic(path: Path) -> bytes: + try: + with path.open("rb") as handle: + return handle.read(4) + except OSError as exc: + raise BootstrapError("ARCHIVE_INSPECTED", "archive is unreadable", code="ARCHIVE_INVALID") from exc + + +def inspect_bootstrap_archive(path: Path) -> dict[str, int]: + magic = _archive_magic(path) + if any(magic.startswith(prefix) for prefix in ZIP_MAGIC): + return inspect_bootstrap_zip(path) + if magic.startswith(GZIP_MAGIC): + return inspect_bootstrap_tar(path) + raise BootstrapError("ARCHIVE_INSPECTED", "download is not a ZIP or tar archive", code="ARCHIVE_INVALID") + + +def safe_extract_bootstrap_archive(path: Path, destination: Path) -> dict[str, int]: + magic = _archive_magic(path) + if any(magic.startswith(prefix) for prefix in ZIP_MAGIC): + return safe_extract_bootstrap_zip(path, destination) + if magic.startswith(GZIP_MAGIC): + return safe_extract_bootstrap_tar(path, destination) + raise BootstrapError("ARCHIVE_INSPECTED", "download is not a ZIP or tar archive", code="ARCHIVE_INVALID") + + def validate_design_bank(root: Path) -> dict[str, Any]: - if root.is_symlink() or not root.is_dir(): + check_root = root.resolve() if root.is_symlink() else root + if not check_root.is_dir() or check_root.is_symlink(): raise BootstrapError("BANK_VALIDATED", "Design Bank root is not a directory", code="DESIGN_BANK_INVALID") counts: dict[str, int] = {} sampled: dict[str, int] = {} @@ -505,9 +786,27 @@ def stage(name: str, evidence: str = "") -> None: report(name, evidence) stage("PREFLIGHT") - source = resolve_bootstrap_source(source_name, config_path=config_path) design_target = resolve_design_target(target) v2_root = design_v2_root or resolve_design_v2_root() + existing = False + validation: dict[str, Any] | None = None + if (design_target.exists() or design_target.is_symlink()) and not download_only: + try: + validation = validate_design_bank(design_target) + except BootstrapError as exc: + raise BootstrapError( + "PREFLIGHT", "target exists but is not a compatible Design Bank", code="TARGET_EXISTS" + ) from exc + existing = True + stage("BANK_VALIDATED", "already-present") + stage("BANK_COMMITTED", "already-present") + + if existing: + source = resolve_bootstrap_source(source_name, config_path=config_path) + archive_url: str | None = None + download_method = "curl-google-drive-public" + else: + source, archive_url, download_method = select_remote_source(source_name, config_path=config_path) stage("SOURCE_RESOLVED", source.name) if dry_run: stage("COMPLETE", "dry-run") @@ -519,23 +818,10 @@ def stage(name: str, evidence: str = "") -> None: "source_type": source.source_type, "target": str(design_target), "design_v2_root": str(v2_root), - "download_method": "curl-google-drive-public", + "download_method": download_method, "stages": stages, } - existing = False - validation: dict[str, Any] | None = None - if design_target.exists() or design_target.is_symlink(): - try: - validation = validate_design_bank(design_target) - except BootstrapError as exc: - raise BootstrapError( - "PREFLIGHT", "target exists but is not a compatible Design Bank", code="TARGET_EXISTS" - ) from exc - existing = True - stage("BANK_VALIDATED", "already-present") - stage("BANK_COMMITTED", "already-present") - cache = cache_dir or share_dir() / "cache" / "design-bootstrap" / source.name archive = cache / source.archive_name checksum_file = cache / f"{source.archive_name}.sha256" @@ -545,34 +831,42 @@ def stage(name: str, evidence: str = "") -> None: if cache.is_symlink() or (cache.exists() and not cache.is_dir()): raise BootstrapError("PREFLIGHT", "bootstrap cache is not a safe directory", code="CACHE_UNSAFE") cache.mkdir(parents=True, exist_ok=True) - if checksum_file.is_symlink(): - checksum_file.unlink() - if not checksum_file.is_file(): + if source.checksum_file_id: + if checksum_file.is_symlink(): + checksum_file.unlink() + if not checksum_file.is_file(): + try: + downloader(google_drive_public_url(source.checksum_file_id), checksum_file) + except BootstrapError as exc: + if exc.stage == "PREFLIGHT": + raise + raise BootstrapError("CHECKSUM_FETCHED", exc.detail, code=exc.code) from exc + except Exception as exc: + raise BootstrapError("CHECKSUM_FETCHED", str(exc), code="DOWNLOAD_FAILED") from exc try: - downloader(google_drive_public_url(source.checksum_file_id), checksum_file) - except BootstrapError as exc: - if exc.stage == "PREFLIGHT": - raise - raise BootstrapError("CHECKSUM_FETCHED", exc.detail, code=exc.code) from exc - except Exception as exc: - raise BootstrapError("CHECKSUM_FETCHED", str(exc), code="DOWNLOAD_FAILED") from exc - try: - expected = parse_checksum(checksum_file.read_text(encoding="utf-8"), source.archive_name) - except BootstrapError: - checksum_file.unlink(missing_ok=True) - raise - except (OSError, UnicodeDecodeError) as exc: - checksum_file.unlink(missing_ok=True) - raise BootstrapError("CHECKSUM_FETCHED", "checksum is unreadable", code="CHECKSUM_INVALID") from exc - if source.pinned_sha256 and expected != source.pinned_sha256: - raise BootstrapError("CHECKSUM_FETCHED", "checksum does not match pinned digest", code="CHECKSUM_MISMATCH") + expected = parse_checksum(checksum_file.read_text(encoding="utf-8"), source.archive_name) + except BootstrapError: + checksum_file.unlink(missing_ok=True) + raise + except (OSError, UnicodeDecodeError) as exc: + checksum_file.unlink(missing_ok=True) + raise BootstrapError("CHECKSUM_FETCHED", "checksum is unreadable", code="CHECKSUM_INVALID") from exc + if source.pinned_sha256 and expected != source.pinned_sha256: + raise BootstrapError( + "CHECKSUM_FETCHED", "checksum does not match pinned digest", code="CHECKSUM_MISMATCH" + ) + else: + expected = source.pinned_sha256 + if not expected: + raise BootstrapError("SOURCE_RESOLVED", "SHA-256 is required", code="SHA256_REQUIRED") stage("CHECKSUM_FETCHED", expected) cached_ok = archive.is_file() and not archive.is_symlink() and sha256_file(archive) == expected if not cached_ok: archive.unlink(missing_ok=True) + fetch_url = archive_url or google_drive_public_url(source.archive_file_id) try: - downloader(google_drive_public_url(source.archive_file_id), archive) + downloader(fetch_url, archive) except BootstrapError: raise except Exception as exc: @@ -584,7 +878,7 @@ def stage(name: str, evidence: str = "") -> None: raise BootstrapError("ARCHIVE_VERIFIED", "archive SHA-256 mismatch", code="CHECKSUM_MISMATCH") stage("ARCHIVE_VERIFIED", actual) try: - archive_stats = inspect_bootstrap_zip(archive) + archive_stats = inspect_bootstrap_archive(archive) except BootstrapError: archive.unlink(missing_ok=True) raise @@ -608,7 +902,7 @@ def stage(name: str, evidence: str = "") -> None: workspace = Path(tempfile.mkdtemp(prefix=".opencode-design-bootstrap-", dir=str(design_target.parent))) extracted = workspace / "extract" try: - safe_extract_bootstrap_zip(archive, extracted) + safe_extract_bootstrap_archive(archive, extracted) stage("EXTRACTED_TO_TEMP", str(extracted)) normalized = normalize_extracted_bank(extracted) validation = validate_design_bank(normalized) diff --git a/tests/support.py b/tests/support.py index ca283fd..034967d 100644 --- a/tests/support.py +++ b/tests/support.py @@ -20,6 +20,8 @@ def setUp(self): "OPENCODE_HE_MOCK_OPENCODE", "OPENCODE_HE_TEST_CBM", "OPENCODE_DESIGN_BANK", + "OPENCODE_DESIGN_BANK_URL", + "OPENCODE_DESIGN_BANK_SHA256", "OPENCODE_SMARTDOC", "OPENCODE_DISABLE_CLAUDE_CODE", "OPENCODE_HE_MOCK_MCP_LIST", @@ -40,6 +42,8 @@ def setUp(self): os.environ["OPENCODE_HE_MOCK_OPENCODE"] = str(mock_oc) os.environ["OPENCODE_HE_TEST_CBM"] = str(mock_cbm) os.environ["OPENCODE_DESIGN_BANK"] = str(ROOT / "tests" / "fixtures" / "Design") + os.environ.pop("OPENCODE_DESIGN_BANK_URL", None) + os.environ.pop("OPENCODE_DESIGN_BANK_SHA256", None) os.environ["OPENCODE_DISABLE_CLAUDE_CODE"] = "1" claude = self.tmp / ".claude" claude.mkdir() diff --git a/tests/test_design_bootstrap.py b/tests/test_design_bootstrap.py index c6c9ef8..46fcc63 100644 --- a/tests/test_design_bootstrap.py +++ b/tests/test_design_bootstrap.py @@ -6,6 +6,7 @@ import os import shutil import socket +import tarfile import urllib.request import zipfile from contextlib import redirect_stdout @@ -17,10 +18,12 @@ from lib.design_v2.bootstrap import ( BootstrapError, bootstrap_design_bank, + drive_confirm_url, google_drive_public_url, inspect_bootstrap_zip, load_bootstrap_sources, parse_checksum, + resolve_operator_url, validate_design_bank, ) from lib.design_v2.commands import doctor_rows @@ -303,6 +306,92 @@ def test_archive_inspection_and_bank_validation_are_bounded(self): self.assertEqual(bank["preview_samples"]["21st"], 1) self.assertEqual(bank["preview_samples"]["aura"], 1) + def test_env_url_without_sha256_fails_closed(self): + os.environ["OPENCODE_DESIGN_BANK_URL"] = "https://example.invalid/bank.zip" + os.environ.pop("OPENCODE_DESIGN_BANK_SHA256", None) + with self.assertRaises(BootstrapError) as caught: + self._bootstrap() + self.assertEqual(caught.exception.code, "SHA256_REQUIRED") + self.assertEqual(self.download_calls, []) + self.assertFalse(self.target.exists()) + + def test_env_url_with_sha256_uses_operator_url(self): + os.environ["OPENCODE_DESIGN_BANK_URL"] = ( + "https://drive.google.com/file/d/1QCqajqPkSl95Y2PDsyC5o-SkyGD7FyRw/view" + ) + os.environ["OPENCODE_DESIGN_BANK_SHA256"] = self.digest + payload = self._bootstrap() + self.assertEqual(payload["status"], "ok") + self.assertEqual(payload["source"], "env-url") + self.assertTrue(any("uc?export=download" in url for url in self.download_calls)) + self.assertTrue((self.target / "Refero/bank/catalog.json").is_file()) + + def test_drive_view_link_resolves_to_uc_export(self): + resolved = resolve_operator_url( + "https://drive.google.com/file/d/1QCqajqPkSl95Y2PDsyC5o-SkyGD7FyRw/view?usp=sharing" + ) + self.assertEqual( + resolved, + "https://drive.google.com/uc?export=download&id=1QCqajqPkSl95Y2PDsyC5o-SkyGD7FyRw", + ) + + def test_drive_confirm_token_extracted_once(self): + html = ( + 'download' + ) + original = "https://drive.google.com/uc?export=download&id=1QCqajqPkSl95Y2PDsyC5o-SkyGD7FyRw" + confirm = drive_confirm_url(original, html) + self.assertIsNotNone(confirm) + self.assertIn("confirm=ABCD", confirm) + self.assertIn("id=1QCqajqPkSl95Y2PDsyC5o-SkyGD7FyRw", confirm) + + def test_symlink_valid_bank_is_already_present(self): + real = self.tmp / "real-bank" + shutil.copytree(self.source_tree, real) + self.target.symlink_to(real) + payload = self._bootstrap() + self.assertEqual(payload["status"], "already_present") + self.assertEqual(self.download_calls, []) + + def test_github_tgz_fallback_when_drive_config_missing(self): + tgz = self.tmp / "Design-bank.tgz" + with tarfile.open(tgz, "w:gz") as handle: + handle.add(self.source_tree, arcname=".") + digest = hashlib.sha256(tgz.read_bytes()).hexdigest() + from lib.design_v2.bootstrap import BootstrapSource + + fallback = BootstrapSource( + name="github-release-fallback", + source_type="https-artifact", + bank_version="1.0.0", + archive_name="Design-bank.tgz", + archive_file_id="", + checksum_file_id="", + pinned_sha256=digest, + ) + url = "https://github.com/kuker24/GrokBestFriend/releases/download/v1.0.0/Design-bank.tgz" + + def tgz_downloader(fetch_url: str, destination: Path) -> None: + self.download_calls.append(fetch_url) + destination.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(tgz, destination) + + missing = self.tmp / "missing-drive.json" + missing.write_text('{"schemaVersion":1,"default":"missing","sources":{}}', encoding="utf-8") + with patch("lib.design_v2.bootstrap.github_fallback_source", return_value=(fallback, url)): + payload = bootstrap_design_bank( + target=self.target, + design_v2_root=self.design_v2, + cache_dir=self.cache, + downloader=tgz_downloader, + config_path=missing, + ) + self.assertEqual(payload["status"], "ok") + self.assertEqual(payload["source"], "github-release-fallback") + self.assertEqual(self.download_calls, [url]) + self.assertTrue((self.target / "21st/library/catalog.json").is_file()) + if __name__ == "__main__": import unittest diff --git a/vendor/sources.json b/vendor/sources.json index 4df5cba..99fa6e0 100644 --- a/vendor/sources.json +++ b/vendor/sources.json @@ -38,7 +38,7 @@ "artifactSha256": "9866f5a82f57f84c45f4f82af8092f9bd3a6838b5c12d6b097e3eb4fda5b634c", "ownedDest": "~/.local/share/opencode-highend/design-bank", "redistribution": "not-cleared", - "note": "Not vendored in git. Installer discovers a valid bank or downloads this existing upstream release asset." + "note": "Not vendored in git. Fourth-priority bootstrap fallback after a valid local bank, OPENCODE_DESIGN_BANK_URL+SHA256, and the Drive ZIP pin in lib/design_v2/bootstrap_sources.json." }, "serena": { "repository": "https://github.com/oraios/serena",