Marketplace `add` source parity with the Anthropic spec

[Vicente-Pastor]

Summary

Bring APM's apm marketplace add to parity with Anthropic's spec by accepting all four source shapes:

Source shape	Example	Today
owner/repo (GitHub)	apm marketplace add anthropic/skills	✅
Git URL (any host, with #ref)	apm marketplace add https://gitlab.com/.../plugins.git#v1.0.0	❌
Local path (dir or file)	apm marketplace add ./vendor/internal-marketplace	❌
Remote marketplace.json URL	apm marketplace add https://example.com/marketplace.json	❌

Phase 2 covers auth integration via AuthResolver for private sources (the gap originally raised in #692).

.well-known/agent-skills/ (Cloudflare RFC) is deferred behind a clear external trigger: the RFC reaches v1.0, OR the upstream agentskills.io PR (#254) merges, OR production adoption shows up. See discussion in #676 (comment).

Why

Anthropic's /plugin marketplace add already accepts these four source shapes, and it's the closest thing the plugin-marketplace world has to a settled spec. Three missing source shapes against that spec is the real, achievable gap to close — not a draft RFC. Closing the parity gap unlocks:

• Air-gapped / enterprise workflows: register marketplaces from internal git mirrors, local checkouts, or a hosted JSON file behind corporate auth.
• Non-GitHub publishers (GitLab, Bitbucket, self-hosted): same UX as the GitHub case.
• Lower friction for adopters who don't want to publish a full git repo just to share a curated catalog — a single marketplace.json URL is enough.

Phase 1 — source parity (this issue)

Accept all four source shapes in apm marketplace add. Lockfile records canonical source + content digest for reproducibility.

Building on the foundation in #691 by @Vicente-Pastor:

• MarketplaceSource generalization (marketplace/models.py)
• ETag / conditional-refresh (marketplace/client.py)
• Archive safety / zip-slip protection (marketplace/archive.py)
• Auth-first client design
• Lockfile provenance fields (deps/lockfile.py)
• ~364 tests across 8 test files

The destination contract changes from the agent-skills index format to Anthropic's marketplace.json schema, but most of the plumbing transfers cleanly.

Phase 2 — auth integration (follow-on)

Wire AuthResolver into URL-based marketplace fetch so private git URLs and authenticated marketplace.json endpoints work end-to-end. Tracked in #692.

Reference

• Anthropic plugin marketplace docs: https://code.claude.com/docs/en/discover-plugins
• Anthropic marketplace.json schema: https://code.claude.com/docs/en/plugin-marketplaces#marketplace-schema
• APM discussion Roadmap — Now / Next / Later #116 (roadmap)
• Original framing context: see comment thread below

Related

• [FEATURE] Skills from .well-known URIs #554 — supersedes (close as dup once Phase 1 lands)
• feat: marketplace UX, security hardening, and lockfile provenance (#514) #677 — partial overlap (publish/validate/view UX); see scope-reduction discussion there
• [FEATURE] URL-based marketplace work Extension #692 — Phase 2 scope
• Feature/676 url based marketplace #691 — implementation in flight
• feat: Support Artifactory-hosted marketplace indexes #506 — Artifactory virtual marketplace path (already shipped; complements remote URL source)

[sergio-sisternes-epam]

Great design writeup, @Vicente-Pastor -- this covers UX, update semantics, digest handling, and lockfile provenance thoroughly.

A few notes for context:

• [FEATURE] Skills from .well-known URIs #554 is the original community request from @jonathanhefner for .well-known URI support. This issue supersedes it as the detailed design proposal -- we should close [FEATURE] Skills from .well-known URIs #554 as a duplicate pointing here once we confirm @jonathanhefner agrees.
• feat: Support Artifactory-hosted marketplace indexes #506 (closed) added Artifactory-hosted marketplace indexes. The URL-based backend proposed here generalizes that pattern further -- the implementation should build on the MarketplaceSource abstraction introduced there.
• The MCP .well-known standardization referenced in [FEATURE] Skills from .well-known URIs #554 (SEP-2127: MCP Server Cards - HTTP Server Discovery via .well-known modelcontextprotocol/modelcontextprotocol#2127) may also be relevant as a future extension point.

The design direction of treating this as a new marketplace backend (not a separate install model) is the right call -- it preserves the existing name@marketplace UX and avoids fragmenting the install surface.

[Vicente-Pastor]

Thanks @sergio-sisternes-epam! Good callout on #554 and #506 — I've been building directly on the MarketplaceSource abstraction from #506 as you suggested.

On the extension point you mentioned — I had actually planned ahead for this. The implementation in #691 was deliberately designed to keep the door open for direct URL-based package installs (not just marketplace discovery). I've opened #692 to track that next step explicitly, with the specific design constraints that #676 already satisfies:

• source_type is an open string (not a closed enum) — ready for "url-package" or similar
• _fetch_url_direct() is generic (plain HTTPS GET, no Agent Skills assumptions)
• _detect_index_format() is extensible for apm.yml detection
• LockedDependency uses neutral source_url / source_digest names (not marketplace-specific)
• Archive extraction (archive.py) is decoupled from marketplace logic, ready for reuse by the package installer

The MCP .well-known standardization you referenced (modelcontextprotocol/modelcontextprotocol#2127) would slot in naturally since we already resolve .well-known/agent-skills/index.json — adding another .well-known path would be a small extension.

See #692 for the full design constraints and capability roadmap.

[Vicente-Pastor]

Implementation Report — URL-based Marketplace Registration

Branch: feature/676-url-based-marketplace
PR: #691
Tests: 364 passing (11 test files — 7 new + 4 modified)
Approach: Strict TDD — every feature red→green→refactor, followed by deep code review + external review

---

Design Decisions

Core architecture: new marketplace backend, not a separate install model

URL-based sources are a new source_type on the existing MarketplaceSource model, preserving the skill@marketplace install UX. This builds directly on the abstraction from #506 (Artifactory) and avoids fragmenting the install surface.

• source_type is an open string ("github" / "url"), not a closed enum — future types like "url-package" slot in without refactoring (see [FEATURE] URL-based marketplace work Extension #692)
• owner/repo become optional (default "") for URL sources — no schema migration needed
• to_dict() omits GitHub-only fields for URL sources; from_dict() defaults to "github" for legacy dicts without source_type → full backward compat

URL detection strategy

URL detection happens before OWNER/REPO parsing in the add command — https:// prefix routes to the URL branch immediately. http:// is rejected with a clear HTTPS error. Scheme detection is case-insensitive (HTTPS://, Https:// all accepted).

.well-known auto-resolution

Bare origin URLs (e.g. https://example.com) are automatically resolved to /.well-known/agent-skills/index.json per RFC 8615. URLs that already contain a path are used as-is. Query strings are preserved (supports token-authenticated endpoints like https://cdn.corp.com?token=abc).

Agent Skills index parsing

parse_agent_skills_index() strictly enforces the RFC v0.2.0 schema:

• $schema field MUST be present and match https://schemas.agentskills.io/discovery/0.2.0/schema.json
• Skill names validated per RFC rules: 1-64 chars, lowercase alphanumeric + hyphens, no leading/trailing/consecutive hyphens
• Invalid entries (bad names, missing digests, non-dict entries) are logged and skipped, not fatal
• Both skill-md and archive types accepted

Format auto-detection

_detect_index_format(data) inspects top-level keys: "skills" → Agent Skills, "plugins" → legacy GitHub, else "unknown". _parse_manifest() is a single dispatcher used by all code paths (fresh fetch, cache hit, 304, stale fallback) — consolidating parsing logic was a key review finding.

Fetch layer: _fetch_url_direct()

Deliberately kept generic — plain HTTPS GET, no Agent Skills assumptions. This allows future callers (e.g. URL-based package installs per #692) to reuse it without importing marketplace internals.

• HTTPS scheme enforced at fetch time (not just command layer)
• Post-redirect validation: if a server redirects from HTTPS to HTTP, the redirect is rejected (prevents HTTPS→HTTP downgrade attacks)
• Returns FetchResult(data, digest, etag, last_modified) — frozen dataclass

Caching strategy

• Cache key for URL sources: sha256(url)[:16] (filesystem-safe, collision-resistant)
• Same 1-hour TTL as GitHub sources
• ETag/Last-Modified conditional refresh: sends If-None-Match/If-Modified-Since; on 304, resets TTL without network write
• Stale-while-revalidate: on network error, serves expired cache with warning (prevents hard failures from transient outages)

Digest / integrity verification

• sha256:<64 hex chars> computed on raw response bytes at fetch time
• Stored in .meta.json alongside cache data
• expected_digest parameter on _fetch_url_direct — extension point for lockfile-aware re-fetch
• Per-skill content-level digest stored in MarketplacePlugin.source["digest"] — verified at discovery, full verification deferred to install time

Lockfile provenance

LockedDependency gained source_url and source_digest fields (names intentionally format-neutral, not marketplace-specific) for future traceability and reproducible installs.

Archive support

marketplace/archive.py provides standalone extraction utilities:

• Formats: .tar.gz, .zip (auto-detected from magic bytes + extension)
• Safety: path traversal, absolute paths, Windows drive-letter/UNC paths, symlinks, hardlinks, null bytes, decompression bomb (512 MB limit)
• Decoupled from marketplace logic — reusable by future package installer

Resolver changes

• resolve_plugin_source() handles skill-md/archive types as parallel code paths
• Non-GitHub HTTPS URLs returned as-is (pass-through) instead of rejected
• Empty marketplace_owner guard: raises ValueError immediately rather than allowing downstream crashes

Error handling strategy

Three-tier error handling in the add command:

1. MarketplaceFetchError → displayed directly (fetch-layer message is already user-friendly)
2. ValueError → wrapped with "Invalid index format:" prefix (parser-layer validation)
3. All other exceptions → "Failed to register marketplace:" prefix (unexpected errors)

---

Implementation Steps

Step 1 — Extend MarketplaceSource model (gaps #1, #7)

• url: str = "", source_type: str = "github", is_url_source property
• to_dict()/from_dict() with backward compat
• 14 tests in test_marketplace_url_models.py

Step 2 — URL detection in marketplace add (gap #3)

• _resolve_index_url() helper, http:// rejection, name from hostname
• Tests in test_marketplace_url_commands.py

Step 3 — URL fetch path in client.py (gaps #2, #8)

• _cache_key() SHA-256, _fetch_url_direct(), _detect_index_format(), _parse_manifest()
• FetchResult frozen dataclass, fetch_marketplace() URL branch
• 18 tests in test_marketplace_url_client.py

Step 4 — Agent Skills index parser (gaps #5, #13, #15)

• parse_agent_skills_index(), $schema enforcement, _is_valid_skill_name()
• Tests in test_marketplace_url_models.py

Step 5 — Digest / integrity verification (gap #9)

• FetchResult.digest, _write_cache() stores digest, _is_valid_digest()
• LockedDependency.source_url / source_digest
• 27 tests across digest, lockfile provenance

Step 6 — ETag/conditional refresh (gap #10)

• If-None-Match/If-Modified-Since, 304 handling, _read_stale_meta()
• Stale-while-revalidate fallback
• Parametrized tests for conditional headers + cache

Step 7 — Archive type support (gap #14)

• archive.py: detection, extraction, safety guards
• 43 tests in test_marketplace_archive.py

Step 8 — Fix list/remove/browse/resolver (gaps #4, #6, #12)

• is_url_source display guards, clear_marketplace_cache(source=), resolver URL pass-through
• Tests in URL commands + resolver suites

---

External Review Summary

48 items resolved across bugs, design, tests, exceptions, comments, and naming. Highlights:

• S1: Post-redirect HTTPS enforcement (prevents HTTPS→HTTP downgrade)
• DR1: from_dict() raises ValueError on unknown source_type (fail-fast)
• DR4: Query string preserved in _resolve_index_url()
• DR6: Case-insensitive scheme detection
• All bug fixes, design improvements, docstrings, and naming cleanups verified with tests

5 architecture items deferred to a dedicated refactoring PR:

• N1: Rename archive.py → extraction.py
• N2: Move ArchiveError to errors.py, inherit from MarketplaceError
• N5: Move _resolve_index_url from commands to client.py
• N8: Move _WELL_KNOWN_PATH alongside _MARKETPLACE_PATHS
• G8: Move archive.py from marketplace/ to utils/

---

Test Inventory

File	Count	Coverage
test_marketplace_url_client.py	67	URL fetch, cache, digest, ETag, format detection, HTTPS enforcement, redirect validation
test_marketplace_url_models.py	61	URL source fields, Agent Skills parser, name validation, digest format
test_marketplace_resolver.py	45	All resolver paths (GitHub + URL), skill-md/archive types
test_marketplace_archive.py	43	Archive safety, extraction, format detection, decompression bomb
test_marketplace_models.py	33	GitHub models + parser (pre-existing)
test_marketplace_url_commands.py	29	URL add/list/remove/update, query string, case-insensitive scheme
test_marketplace_client.py	20	GitHub client paths (pre-existing, adapted for tuple return)
test_marketplace_url_resolver.py	15	URL source resolution, PluginNotFoundError
test_marketplace_commands.py	15	GitHub commands (pre-existing)
test_lockfile_provenance.py	14	Lockfile source_url/source_digest round-trips
test_marketplace_registry.py	10	Registry CRUD (pre-existing)
test_marketplace_install_integration.py	7	Install pre-parse, marketplace ref detection
test_marketplace_errors.py	5	Error class hierarchy (pre-existing)
Total	364	364 / 364 passing

---

Files Changed (vs main)

File	Type	Summary
src/apm_cli/marketplace/models.py	Modified	MarketplaceSource URL fields, parse_agent_skills_index(), _is_valid_skill_name(), _is_valid_digest(), manifest provenance fields
src/apm_cli/commands/marketplace.py	Modified	URL detection, _resolve_index_url(), list/remove/update URL display, error handling
src/apm_cli/marketplace/client.py	Modified	_fetch_url_direct(), FetchResult, _detect_index_format(), _parse_manifest(), _cache_key() SHA-256, _read_stale_meta(), ETag/conditional, redirect validation
src/apm_cli/marketplace/resolver.py	Modified	Non-GitHub URL pass-through, skill-md/archive types, empty URL guard
src/apm_cli/marketplace/archive.py	New	Archive detection, extraction, safety guards (path traversal, symlinks, decompression bomb)
src/apm_cli/deps/lockfile.py	Modified	source_url / source_digest on LockedDependency
tests/unit/marketplace/test_marketplace_url_models.py	New	61 tests for URL source model and Agent Skills parser
tests/unit/marketplace/test_marketplace_url_client.py	New	67 tests for URL fetch, cache, digest, format detection
tests/unit/marketplace/test_marketplace_url_commands.py	New	29 tests for URL commands
tests/unit/marketplace/test_marketplace_url_resolver.py	New	15 tests for URL resolver
tests/unit/marketplace/test_marketplace_archive.py	New	43 tests for archive safety and extraction
tests/unit/marketplace/test_lockfile_provenance.py	Modified	7 new provenance round-trip tests
tests/unit/marketplace/test_marketplace_client.py	Modified	Adapted for _read_cache() tuple return
tests/unit/marketplace/test_marketplace_resolver.py	Modified	Updated non-GitHub URL test

---

Future Extension

Issue #692 tracks extending this work to support installing full APM packages directly from a URL. The current implementation was designed to not foreclose this direction — see #692 for specific design constraints already satisfied.