LLM provider discovery: read tools and access probe

.github/workflows/scripts/lint_skill_refs.py

@@ -15,6 +15,7 @@
     {
         "agent",
         "ai-automation",
+        "ai-provider",
         "attachment",
         "audit",
         "automation",

README.md

@@ -36,7 +36,7 @@ Open-source toolkit for **Pipefy** developers: a Model Context Protocol (MCP) se
 
 | Component | Package / path | Purpose |
 |-----------|----------------|---------|
-| **MCP server** | `pipefy-mcp-server` | Exposes **157** tools to MCP clients (Cursor, Claude Desktop, Claude Code, and others). |
+| **MCP server** | `pipefy-mcp-server` | Exposes **162** tools to MCP clients (Cursor, Claude Desktop, Claude Code, and others). |
 | **CLI** | `pipefy-cli` | Terminal commands aligned with MCP capabilities; see [`docs/parity.md`](docs/parity.md). |
 | **SDK** | `pipefy` | Vendor GraphQL client, services, and models shared by MCP and CLI. |
 | **Skills** | [`skills/`](skills/) | Markdown playbooks (Anthropic Skills format) for common Pipefy workflows. |
@@ -141,7 +141,7 @@ Deprecation and semver (post-1.0): [`docs/DEPRECATION.md`](docs/DEPRECATION.md).
 
 ## MCP server
 
-The server registers **157 tools** across eleven domains. Canonical names: `PIPEFY_TOOL_NAMES` in [`packages/mcp/src/pipefy_mcp/tools/registry.py`](packages/mcp/src/pipefy_mcp/tools/registry.py).
+The server registers **162 tools** across twelve domains. Canonical names: `PIPEFY_TOOL_NAMES` in [`packages/mcp/src/pipefy_mcp/tools/registry.py`](packages/mcp/src/pipefy_mcp/tools/registry.py).
 
 Tool descriptions and `Args:` blocks come from Python docstrings (what MCP clients show to models). Per-area reference docs cover parameters, edge cases, and cross-cutting behavior.
 
@@ -154,6 +154,7 @@ Tool descriptions and `Args:` blocks come from Python docstrings (what MCP clien
 | **Relations** | 8 | Pipe and card relations. | [docs](docs/mcp/tools/relations.md) |
 | **Reports** | 17 | Pipe and organization reports, async exports. | [docs](docs/mcp/tools/reports.md) |
 | **Automations & AI** | 23 | Automations, AI automations, AI agents, validators. | [docs](docs/mcp/tools/automations-and-ai.md) |
+| **LLM providers** | 5 | Discovery reads for the organization's LLM providers (custom + Pipefy-managed), vendor model lists, owner defaults, dependencies, and a read-access probe. | [docs](docs/mcp/tools/llm-providers.md) |
 | **iPaaS** | 4 | Lazy discovery, invocation, and app-connection setup for a pipe's iPaaS (Advanced Automations) workspace (`get_ipaas_tools`, `call_ipaas_tool`, plus the connection meta-tools). | [docs](docs/mcp/tools/ipaas.md) |
 | **Observability** | 11 | Logs, usage, credits, execution metrics, job exports. | [docs](docs/mcp/tools/observability.md) |
 | **Members, email & webhooks** | 11 | Membership, inbox email, webhooks. | [docs](docs/mcp/tools/members-email-webhooks.md) |

docs/mcp/README.md

@@ -12,6 +12,7 @@ Material in this tree describes **`pipefy-mcp-server`**: the MCP process, tool b
 | [`tools/relations.md`](tools/relations.md) | Pipe and card relations |
 | [`tools/reports.md`](tools/reports.md) | Pipe and organization reports, async exports |
 | [`tools/automations-and-ai.md`](tools/automations-and-ai.md) | Traditional automations, AI automations, AI agents, validators |
+| [`tools/llm-providers.md`](tools/llm-providers.md) | LLM provider discovery: custom + system providers, vendor models, defaults, dependencies, access probe |
 | [`tools/observability.md`](tools/observability.md) | Logs, usage, credits, execution metrics, job exports |
 | [`tools/members-email-webhooks.md`](tools/members-email-webhooks.md) | Membership, inbox email, webhooks |
 | [`tools/organization.md`](tools/organization.md) | Organization metadata |

docs/mcp/tools/automations-and-ai.md

@@ -182,7 +182,7 @@ On **create**, if the caller omits `condition`, the MCP layer supplies `DEFAULT_
 Optional inside `actionParams.aiBehaviorParams`:
 
 - **`capabilitiesAttributes`** — list of capability entries, each exactly `{ "capabilityType": "<type>", "enabled": true|false }`. Both keys are required and no other keys are accepted; legacy shapes (bare string lists, `{ "type": ... }`) are rejected. Common `capabilityType` values: `advanced_ocr` (product name IDP / Intelligent Document Processing), `math_operations` (Calculations & Analysis), `web_search`, `web_scraping`, `max_effort`. `capabilityType` is not checked against a fixed set — any value passes through and the API validates the enum on write. Validation checks shape only, not plan/feature entitlement — a capability may still require organization-level enablement to take effect.
-- **`providerId`** / **`systemProviderId`** — select the behavior's LLM provider; set at most one (reads resolve a single active provider). Discover IDs via the organization's AI settings in the Pipefy UI.
+- **`providerId`** / **`systemProviderId`** — select the behavior's LLM provider; set at most one (reads resolve a single active provider). Discover IDs with `get_llm_providers` (see [LLM providers](llm-providers.md)): use `providerId` for a custom (`byom`) provider and `systemProviderId` for a Pipefy-managed (`system`) one. IDs are also visible in the organization's AI settings in the Pipefy UI.
 
 ### Optional `eventParams` (trigger filters)
 

docs/mcp/tools/llm-providers.md

@@ -0,0 +1,41 @@
+# LLM providers
+
+Read-only discovery of the LLM providers an organization can use: custom (BYOM) providers, Pipefy-managed system providers, vendor model lists, owner defaults, provider dependencies, and a read-access probe. **5 tools.**
+
+These are the discovery counterpart to the `providerId` / `systemProviderId` fields on AI agent behaviors (see [Automations & AI](automations-and-ai.md)): use `get_llm_providers` to find the IDs a behavior accepts.
+
+---
+
+| Tool | Read-only | Role |
+|------|-----------|------|
+| `get_llm_providers` | Yes | Lists custom and system providers in one union, paginated (`first` default 50, `after`) with `only_active` (filters custom providers only). Each node carries `type` (`byom` = custom, `system` = Pipefy-managed) plus `configuration`. |
+| `get_available_ai_models` | Yes | Lists the model names a provider vendor exposes. `provider_name` is one of `openai`, `azure_openai`, `amazon_bedrock`, `custom`, `google_vertex_ai`, `oracle_oci`, `anthropic` (the API validates membership). |
+| `get_default_llm_provider` | Yes | Resolves the default provider for an owner: `owner_type` is `organization` (default), `assistant`, or `behavior`. The returned `type` says whether the default is custom or system. |
+| `get_llm_provider_dependencies` | Yes | Lists the owners (`ownerId` / `ownerType`) that depend on a provider — the blockers to check before deactivating or removing one. Paginated, with `total_count`. |
+| `validate_llm_provider_access` | Yes | Probes whether the current credential can read the organization's providers, classifying failures into structured problems (permission denied / not found / invalid arguments) instead of opaque errors. |
+
+## Identifiers: UUID vs numeric ID
+
+The organization identifier is **not uniform** across these queries — this follows the Pipefy GraphQL API, not a toolkit choice:
+
+- `get_llm_providers` and `get_llm_provider_dependencies` take the organization **UUID** (`organization_uuid`).
+- `get_default_llm_provider` with `owner_type="organization"` takes the **numeric organization ID** as `owner_id`.
+- `get_available_ai_models` takes no organization argument at all (vendor-level lookup).
+
+`get_organization` returns both `id` and `uuid`, so one call resolves whichever form a tool needs.
+
+## Configuration redaction
+
+Provider `configuration` is included in read output, and the API **redacts secret values server-side**: placeholder markers come back instead of secrets (e.g. an access token reads as a redaction placeholder). Reading configuration never exposes credentials.
+
+## Probe semantics
+
+- A green `validate_llm_provider_access` proves **read access only** — never write entitlement. Provider mutations may still be denied.
+- An **empty system-provider list** can mean Pipefy-managed system models are not enabled for the organization, rather than a permission problem. The probe reports `system_providers_visible` / `custom_providers_visible` separately and says so in its `note`.
+- Permissions are asymmetric across the read surface: the provider **list** needs a weaker permission than the models / default / dependencies reads, so a green probe does not guarantee those three succeed for the same credential.
+- The scoping of the stronger permission also differs: `get_available_ai_models` and `get_default_llm_provider` authorize against the **credential's own organization context**, while `get_llm_provider_dependencies` authorizes against the **target organization** passed as `organization_uuid` — the same token can be denied on the former and green on the latter.
+- Probing an **unknown organization UUID** surfaces as permission denied rather than not-found (the API does not reveal whether the organization exists).
+
+## Error classification
+
+Failures on this surface are classified by a shared SDK-level module (`pipefy_sdk.graphql_problem`) into structured problems — `permission_denied`, `not_found`, `invalid_arguments`, `feature_not_enabled`, or `runtime` — carried on `error.details.kind` alongside the GraphQL `extensions.code` and `correlation_id`. The same classifier backs the CLI (`pipefy ai-provider ...`), so both surfaces report the same problem kinds.

docs/parity.md

@@ -2,7 +2,7 @@
 
 This matrix is the source of truth for **MCP tool ↔ `pipefy` CLI** coverage. Update it whenever MCP tools or CLI commands are added, renamed, or removed.
 
-**Registry source:** `PIPEFY_TOOL_NAMES` in `packages/mcp/src/pipefy_mcp/tools/registry.py` (must stay in sync with this table: **157** tools).
+**Registry source:** `PIPEFY_TOOL_NAMES` in `packages/mcp/src/pipefy_mcp/tools/registry.py` (must stay in sync with this table: **162** tools).
 
 **Later CLI coverage:** areas such as attachments, field conditions, email, audit export, traditional automations, exports/usage, introspection, and raw GraphQL appear as **shipped** below when the matching Typer commands exist in `packages/cli`.
 
@@ -102,16 +102,20 @@ For **database records**, `find_records` result nodes may use **`fields`** while
 | `get_automation_logs_by_repo` | `pipefy automation logs --repo` | shipped | — |
 | `get_automations` | `pipefy automation list` | shipped | (optional `--organization` / `--pipe`). |
 | `get_automations_usage` | `pipefy usage automations` (also `pipefy automation usage`) | shipped | (`--organization`, `--from`, `--to` ISO range). |
+| `get_available_ai_models` | `pipefy ai-provider models` | shipped | LLM provider discovery; vendor model list (`--provider-name`). |
 | `get_card` | `pipefy card get` | shipped | Supports `--include-fields`. |
 | `get_card_inbox_emails` | `pipefy email inbox list` | shipped | (`--card`). |
 | `get_card_relations` | `pipefy relation card list` | shipped | (raw ``get_card_relations`` payload). |
 | `get_cards` | `pipefy card list` | shipped | ``list`` maps to ``get_cards`` (``--pipe``, ``--title``, ``--search`` JSON / ``CardSearch``, ``--include-fields``, ``--first`` / ``--after``). Use ``pipefy card find`` for ``find_cards`` (single field equality). |
+| `get_default_llm_provider` | `pipefy ai-provider default get` | shipped | LLM provider discovery; owner default (`--owner-id`, `--owner-type`, default `organization`). |
 | `get_email_templates` | `pipefy email template list` | shipped | (`--repo`). |
 | `get_ipaas_connection_auth_url` | — | deferred | iPaaS (Advanced Automations) OAuth consent URL for connecting an app; MCP-first surface. |
 | `get_field_condition` | `pipefy field-condition get` | shipped | — |
 | `get_field_conditions` | `pipefy field-condition list` | shipped | (`--phase`). |
 | `get_ipaas_tools` | — | deferred | iPaaS (Advanced Automations) tool discovery; MCP-first surface, CLI twin considered alongside invocation's. |
 | `get_labels` | `pipefy label list` | shipped | — |
+| `get_llm_provider_dependencies` | `pipefy ai-provider dependencies` | shipped | LLM provider discovery; owners depending on a provider (`--provider-id`, `--org-uuid`). |
+| `get_llm_providers` | `pipefy ai-provider list` | shipped | LLM provider discovery; custom + system providers in one union (`--org-uuid`, `--only-active`, `--first` / `--after`). |
 | `get_organization` | `pipefy org get` | shipped | Organization-level read; out of v0.1 parity per spec. |
 | `get_organization_report` | `pipefy report-org get` | shipped | Organization reports. |
 | `get_organization_report_export` | (poll via `pipefy report-org export --format json`) | shipped | Organization reports + export-shaped. |
@@ -182,6 +186,7 @@ For **database records**, `find_records` result nodes may use **`fields`** while
 | `upload_attachment_to_table_record` | `pipefy attachment upload --record` | shipped | (same supporting flags as card). |
 | `validate_ai_agent_behaviors` | `pipefy agent validate-behaviors` | shipped | AI Agents validation tooling. |
 | `validate_ai_automation_prompt` | `pipefy ai-automation validate-prompt` | shipped | AI Automations validation tooling. |
+| `validate_llm_provider_access` | `pipefy ai-provider validate-access` | shipped | LLM provider read-access probe; green proves read access only, never write entitlement. |
 
 ## Row count check
 
@@ -194,6 +199,6 @@ for n in m.body:
             print(len(v.args[0].elts))"
 ```
 
-Expect **157** tool names in `PIPEFY_TOOL_NAMES` and **157** data rows in the parity table (excluding the header rows).
+Expect **162** tool names in `PIPEFY_TOOL_NAMES` and **162** data rows in the parity table (excluding the header rows).
 
 When adding or removing an MCP tool, update **this file** and `PIPEFY_TOOL_NAMES` in the same change set.

packages/cli/src/pipefy_cli/commands/_common.py

@@ -335,6 +335,7 @@ def run_cli_command(
     *,
     exit_code_2_on_value_error: bool = True,
     format_transport_query_error: Callable[[TransportQueryError], str] | None = None,
+    exit_1_on_unsuccessful: bool = False,
 ) -> None:
     """Run an async coroutine factory with a configured client and render the result.
 
@@ -345,6 +346,9 @@ def run_cli_command(
         exit_code_2_on_value_error: Map ``ValueError`` to process exit code 2 (stderr).
         format_transport_query_error: Optional override for GraphQL transport errors
             (defaults to a single-line formatter).
+        exit_1_on_unsuccessful: After rendering, exit 1 when the result dict has
+            ``success`` falsy. For probe-style commands whose failure is data,
+            not an exception, but should still be scriptable via exit code.
     """
     pipefy_settings, auth = settings_and_auth_from_ctx(ctx)
     transport_fmt = format_transport_query_error or _format_transport_query_error
@@ -374,3 +378,9 @@ async def _run() -> _R:
         render_json(data)
     else:
         render_rich(data)
+    if (
+        exit_1_on_unsuccessful
+        and isinstance(data, dict)
+        and not data.get("success", True)
+    ):
+        raise typer.Exit(1)