LLM provider discovery: read tools and access probe

[mocha06]

Motivation

AI agent behaviors accept providerId / systemProviderId, but the only way to discover valid IDs was the organization's AI settings in the Pipefy UI. There was also no way to list the models a vendor exposes, see which provider an owner defaults to, check what depends on a provider, or verify that a credential can read the provider surface at all — and failures on this surface came back as opaque server errors. Closes #411.

Outcome

• Five read-only tools with full SDK + MCP + CLI parity: get_llm_providers (custom + Pipefy-managed system providers in one union surface, paginated, only_active), get_available_ai_models, get_default_llm_provider (general owner_type, organization default), get_llm_provider_dependencies, and the validate_llm_provider_access probe. Registry and parity table go 157 → 162; all carry readOnlyHint.
• Providers carry the API's type discriminator (byom / system) and their configuration, which the API redacts server-side — placeholders come back instead of secrets.
• New standalone SDK module pipefy_sdk.graphql_problem classifies GraphQL errors into structured problems (permission denied / not found / invalid arguments / feature / runtime) with code and correlation_id carried through. The MCP tools and the probe consume it today; CLI write gating (6. LLM provider management: write tools (BYOM) #415) reuses it next. It lives in the SDK because the existing MCP enrichment cannot be imported by the SDK/CLI.
• Probe semantics are explicit: a green result proves read access only, never write entitlement; an empty system-provider list can mean system models aren't enabled for the organization rather than a permission problem; partial GraphQL errors alongside data stay visible on the result.
• The organization identifier is not uniform across the API and is documented per tool: list/dependencies take the org UUID, the org default takes the numeric org ID, models takes no org argument; get_organization returns both.
• CLI: pipefy ai-provider list | models | default get | dependencies | validate-access; validate-access exits 1 on a red probe.
• Docs and skills that pointed provider-ID discovery at the Pipefy UI now point at get_llm_providers; new reference doc docs/mcp/tools/llm-providers.md.

Testing

uv run pytest -m "not integration" (3512 passed) + ruff + import-linter green. Exercised end-to-end against a dev org through the MCP server running this branch's code:

#	Scenario	Expected	Observed	Verdict
1	get_llm_providers	Both union members, secrets redacted, pagination	4 system + 2 byom; token / clientSecret / customHeaders all __REDACTED__; pagination block present	✅ pass
2	validate_llm_provider_access	Green probe with visibility flags and read-only caveat	ok: true, both visibility flags true, note states green proves read access only	✅ pass
3	get_llm_provider_dependencies (real provider)	Dependents page with total	dependencies: [], total_count: 0	✅ pass
4	get_llm_provider_dependencies (bogus id)	Structured not-found + discovery hint	RESOURCE_NOT_FOUND, kind: not_found, "Use 'get_llm_providers'…" appended	✅ pass (expected rejection)
5	get_available_ai_models / get_default_llm_provider without provider-management rights	Structured permission problem	PERMISSION_DENIED, kind: permission_denied, correlation_id carried	✅ pass (expected rejection — the documented permission asymmetry)
6	Blank organization_uuid	Rejected before the wire	'organization_uuid' must be non-empty.	✅ pass (expected rejection)
7	Probe on nonexistent org UUID	Classified failure, no org-existence leak	PERMISSION_DENIED rather than not-found	✅ pass (expected rejection)

Rows 4–7 are the probe error-mapping acceptance criterion exercised live: each returns the classified problem instead of the opaque error this issue set out to eliminate.

[adriannoes]

Summary

Reviewed with local checkout and tests on tip 7cf1aa7d. This ships the #411 read/probe surface cleanly: five tools with SDK + MCP + CLI parity, readOnlyHint, registry/parity updates, and the shared pipefy_sdk.graphql_problem classifier ready for #415. CI is green on the tip (including the format + skill-ref allowlist fixup).

What worked well

• Full parity for the five discovery reads, with org UUID vs numeric ID called out at the tool boundary.
• Probe semantics are explicit and locked in tests: green means list/read access only; empty system list can mean feature off; partial GraphQL errors stay visible on the result.
• Live smoke matrix in the PR body covers the acceptance paths that matter, including expected rejections.

Also noted

• F3: A probe can return ok: true with a non-null problem when GraphQL returns partial data plus errors. That matches the locked tests and the PR contract. The success path still reads like a clean gate (MCP "confirmed", CLI exit 0), and the docs do not say a clean gate also needs problem absent. Worth a small docs (and maybe message) polish before #415 write gating keys off this probe: clean gate = ok and no problem; classifier kind on CLI is probe-shaped today. No need to flip ok for this PR.