LLM provider management: write tools (BYOM)

[mocha06]

Closes #415. Final phase of milestone 19 (AI Agents: Knowledge Bases, Capabilities & Model Selection); the write counterpart to the provider discovery surface (#411).

Motivation

The discovery phase (#411) made LLM providers readable but not manageable. Bringing your own model (BYOM) still required the Pipefy UI: there was no way to create a custom provider, rotate its credentials, toggle it, or set the organization default from an agent or script. This phase closes that gap while keeping provider secrets out of arguments, logs, errors, and responses.

What ships

Six write tools across SDK → MCP → CLI, plus docs and the AI-agents skill:

MCP	CLI
create_llm_provider	pipefy ai-provider create
update_llm_provider	pipefy ai-provider update
delete_llm_provider	pipefy ai-provider delete
set_llm_provider_active_status	pipefy ai-provider set-active-status
set_default_llm_provider	pipefy ai-provider default set
reset_default_llm_provider	pipefy ai-provider default reset

Proven contract

Verified against the live GraphQL schema and traced through the full delegation chain (GraphQL entry → downstream credential-validating service), per-argument:

• Configuration is secret-bearing and file-only. MCP (configuration_file_path) and CLI (--config-file) accept the provider configuration only via a local JSON file, never inline — so it never becomes a logged tool argument. The provider key selects the vendor; vendor/model membership and credential validity are validated server-side (a live credential test call), so the toolkit treats the object as opaque rather than coupling to the vendor schema.
• Secrets are never returned. create/update deliberately omit configuration from their selection sets even though the payload exposes it; the response carries only id/name/type/active/organizationDefault. File/parse errors report only the path and a structural reason, never file contents.
• Update is a full replacement, and the redaction round-trip is safe. configuration is required on every update. A caller can fetch the (redacted) configuration, edit the non-secret fields, and resubmit: any value left as the __REDACTED__ placeholder is stripped upstream and the stored secret is preserved — confirmed live (an update with the placeholder succeeds; the backend used the stored key, not the literal placeholder). Supplying a new real value rotates the secret.
• Identifier forms (not uniform, following the API): create/update/delete take the organization UUID; set-default/reset take the numeric organization id; set-active-status takes no organization argument; set-default requires exactly one of provider_id / system_provider_id (enforced client-side and downstream).
• Auth: writes require the manage_ai_providers organization permission and an eligible plan — distinct from manage_ai_agents. The read-access probe proves read only, so a green probe does not imply write entitlement.
• Owner-scoped operations need a service-account credential. set-active-status, set-default, reset, and the organization-owner get_default_llm_provider authorize against the credential's own organization context. A service account is bound to one organization and supplies that context; a bare personal token is denied regardless of the organization passed — even when the same token can create/update/delete providers there (those authorize against the passed UUID). Documented in the tool docstrings and docs/mcp/tools/llm-providers.md.

Behavior boundaries

• Default set/reset are organization-scoped (owner is always the organization); assistant/behavior owner scopes are out of scope for the write surface.
• Create/update stay excluded from the remote profile (they read a local file); all six writes are unmarked, so the default-deny remote seed is unchanged.
• CLI create/update are gated on the read-access probe, treated as clean only when it is ok and carries no problem (a present problem is partial denial, never full access); the 2. LLM provider discovery: read tools and access probe #411 probe docs now state this clean-gate contract explicitly.
• Deletes require confirmation (MCP two-step confirm; CLI prompt unless --yes).

Known upstream issue

Provider credential-validation errors are forwarded from the backend and can echo a masked copy of the submitted credential plus an internal error_code blob. This originates upstream; the toolkit surfaces the classified message verbatim (no client-side scrub, by decision) and the upstream fix is tracked separately (here)

Docs and skills touched

docs/mcp/tools/llm-providers.md (writes, secret handling, the update flow, the probe clean-gate contract, service-account requirement), docs/parity.md, README / packages/mcp/README tool counts (reconciled to 182), the AI-agents skill (BYOM provider management flow), and the registry.

Testing

Unit tests assert configuration never appears in responses, logs, or errors; the XOR rule; full-replacement update; the redaction placeholder passthrough; confirm-delete; and the probe clean-gate. Live end-to-end against a lab organization:

#	Scenario	Observed	Verdict
1	Read-access probe	ok, no problem, providers visible	pass
2	Read redaction	secrets return as __REDACTED__	pass
3	create (valid key)	provider created; response carries no configuration	pass
4	Created provider reads redacted	token __REDACTED__	pass
5	update redaction round-trip	success with __REDACTED__; model changed; stored secret preserved (no re-auth failure)	pass
6	create — non-object config	rejected; secret not echoed	pass
7	create — malformed JSON	rejected with position only, no content	pass
8	set-default — both ids / neither	rejected "exactly one"	pass
9	create — invalid key	backend rejects; nothing created	pass
10	set-active-status (service account)	deactivate then reactivate	pass
11	set-default — provider id / system id (service account)	both paths succeed	pass
12	reset-default (service account)	default cleared	pass
13	Owner-scoped ops with a personal token	PERMISSION_DENIED (documented auth boundary)	pass (expected rejection)

[adriannoes]

Summary

Checkout+tests on tip 7b9a81d (87 provider unit tests + ruff green; CI lint/test/skills green). The write surface itself (file-only secrets, omitted configuration on create/update, CLI clean-gate, parity/registry) looks solid against #415. One point-of-use docs gap needs a fix before agents reliably create non-openai BYOM providers: the kebab vs snake vendor vocabularies are listed on different tools with no bridge.

What worked well

• Secret handling is coherent end-to-end: path-only input, no configuration in create/update selection sets, structural-only file errors, and tests that secrets never appear in parse error strings.
• CLI create/update clean-gate (ok and no problem) matches the #411/#415 contract; probe docs state it explicitly.
• Owner-scoped auth (service account) and UUID vs numeric id quirks are documented in MCP tool docstrings and docs/mcp/tools/llm-providers.md.

Required before merge

• Document that configuration.provider (kebab, e.g. oracle-oci) is not interchangeable with ProviderName / get_available_ai_models (provider_name snake, e.g. oracle_oci). Live create with snake fails as invalid adapter; models reject kebab. Fix in the create tool docstring and the configuration section of docs/mcp/tools/llm-providers.md (optional one-line cross-ref on the models tool).

Also noted

• The AI-agents skill BYOM paragraph lists owner-scoped ops (set_llm_provider_active_status, default set/reset) without restating the service-account requirement or UUID vs numeric split. Tool docstrings already cover this; optional skill polish only (not on the #415 acceptance path of validate → create/list → behavior providerId).

Review path

• Checked out PR tip in a worktree, ran provider-focused unit tests and ruff; CI green on tip.
• Live checks confirmed dual vendor vocabularies (config reads oracle-oci; ProviderName is snake-only; create with snake → Invalid adapter).
• Security and architecture passes found no medium+ exploit or facade bypass; related KB probe-gate work stays in #435/#436 after this PR.

[adriannoes]

The create path and the models lookup use different vendor string forms, and the docs list each without saying they are not interchangeable.

configuration.provider in the JSON file for create_llm_provider / update_llm_provider expects kebab-case values such as oracle-oci and amazon-bedrock. Live create rejects snake_case forms as an invalid adapter.

get_available_ai_models takes provider_name in snake_case (oracle_oci, amazon_bedrock, …). Kebab-case is rejected there.

Please add an explicit note in the create tool docstring and in docs/mcp/tools/llm-providers.md (configuration section) that the two vocabularies are surface-specific and must not be copied across. A one-line cross-reference from the models tool Args would help agents avoid a failed round-trip when wiring BYOM.

[mocha06]

Good catch — fixed in 45c013f. The two vocabularies are now explicitly bridged, with the non-interchangeability stated at each point of use:

• create_llm_provider docstring: notes the hyphenated provider strings are file-specific and that get_available_ai_models takes snake_case, with the invalid-adapter consequence.
• get_available_ai_models Args: one-line cross-ref back to the configuration.provider hyphenated form.
• docs/mcp/tools/llm-providers.md (configuration section): a dedicated "surface-specific and not interchangeable" note listing both forms and noting single-word vendors (openai, anthropic, custom) are identical on both.

I also extended it to the CLI, since it's the same footgun there: the shared --config-file help (create/update) and the models --provider-name help now carry the same cross-reference, and the help golden is regenerated.

[mocha06]

Thanks for the thorough pass @adriannoes.

Required (kebab vs snake vocabulary): Done in 45c013f, and replied inline with specifics. Bridged at every point of use across both surfaces — MCP (create_llm_provider docstring, get_available_ai_models Args, the docs configuration section) and CLI (--config-file help, models --provider-name help, regenerated golden). Live behavior confirmed matches your finding: snake_case configuration.provider → invalid adapter; hyphenated provider_name → rejected by the models lookup; single-word vendors identical on both.

Also noted (skill BYOM paragraph): I'm going to leave this one as-is. The service-account requirement and the UUID-vs-numeric split already live where an agent hits them at call time — the tool docstrings and docs/mcp/tools/llm-providers.md — and you flagged it yourself as optional and off the #415 acceptance path (validate → create/list → behavior providerId). The skill's BYOM paragraph is already dense; restating owner-scoping caveats there is redundancy rather than added clarity. Happy to reconsider if you feel the skill is a path an agent would take without the docstrings in view.

[adriannoes]

LGTM