Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ For shell scripting, CI/CD pipelines, and agentic workflows, install the CLI ext
nipyapi ci get_status --process_group_id <id>

See ``docs/cli.rst`` for the complete CLI reference and ``docs/ci.rst`` for CI/CD pipeline examples.
See the companion repository `nipyapi-actions <https://github.com/Chaffelson/nipyapi-actions>`_ for GitHub Actions and GitLab CI examples and support.
See the companion repository `nipyapi-actions <https://github.com/Chaffelson/nipyapi-actions>`_ for GitHub Actions, GitLab CI, and Azure Pipelines examples and support.

**Built-in Docker Profiles:**

Expand Down
71 changes: 68 additions & 3 deletions docs/ci.rst
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Available Operations
ensure_registry
---------------

Create or update a Git Flow Registry Client (GitHub or GitLab).
Create or update a Git Flow Registry Client (GitHub, GitLab, or Azure DevOps).

.. code-block:: console

Expand All @@ -93,14 +93,50 @@ Parameter Description Env
``--token`` Personal Access Token ``GH_REGISTRY_TOKEN`` or ``GL_REGISTRY_TOKEN``
``--repo`` Repository in owner/repo format ``NIFI_REGISTRY_REPO``
``--client_name`` Registry client name ``NIFI_REGISTRY_CLIENT_NAME``
``--provider`` Git provider (github/gitlab) ``NIFI_REGISTRY_PROVIDER``
``--provider`` Git provider (github/gitlab/azuredevops) ``NIFI_REGISTRY_PROVIDER``
``--api_url`` API URL override ``NIFI_REGISTRY_API_URL``
``--default_branch`` Default branch (default: main) ``NIFI_REGISTRY_BRANCH``
``--repository_path`` Path within repository ``NIFI_REPOSITORY_PATH``
============================= ============================================ ================================

**Returns:** ``registry_client_id``, ``registry_client_name``

**Azure DevOps:**

Azure DevOps authenticates with a **service principal** (OAuth2 client credentials), not a
personal access token. Provide the service-principal credentials and repository coordinates
as arguments or via ``NIFI_ADO_*`` environment variables:

.. code-block:: console

nipyapi ci ensure_registry \
--provider azuredevops \
--organization myorg \
--project myproject \
--repository nifi-flows \
--tenant_id $NIFI_ADO_TENANT_ID \
--client_id $NIFI_ADO_CLIENT_ID \
--client_secret $NIFI_ADO_CLIENT_SECRET \
--default_branch main

============================= ============================================ ================================
Parameter Description Environment Variable
============================= ============================================ ================================
``--provider`` Set to ``azuredevops`` ``NIFI_REGISTRY_PROVIDER``
``--organization`` Azure DevOps organization ``NIFI_ADO_ORG``
``--project`` Azure DevOps project ``NIFI_ADO_PROJECT``
``--repository`` Azure DevOps repository name ``NIFI_ADO_REPO``
``--tenant_id`` Entra (Azure AD) tenant id ``NIFI_ADO_TENANT_ID``
``--client_id`` Service principal client id ``NIFI_ADO_CLIENT_ID``
``--client_secret`` Service principal client secret ``NIFI_ADO_CLIENT_SECRET``
``--default_branch`` Default branch (default: main) ``NIFI_REGISTRY_BRANCH``
``--repository_path`` Path within repository ``NIFI_REPOSITORY_PATH``
============================= ============================================ ================================

Provisioning an Azure DevOps registry client also creates the supporting management
controller services (a web client service and an OAuth2 token provider) that NiFi
needs to authenticate to Azure DevOps.

deploy_flow
-----------

Expand Down Expand Up @@ -589,10 +625,16 @@ Variable Description
``NIFI_REGISTRY_REPO`` Repository in owner/repo format
``NIFI_REGISTRY_CLIENT_ID`` Registry client ID
``NIFI_REGISTRY_CLIENT_NAME`` Registry client name
``NIFI_REGISTRY_PROVIDER`` Git provider (github/gitlab)
``NIFI_REGISTRY_PROVIDER`` Git provider (github/gitlab/azuredevops)
``NIFI_REGISTRY_API_URL`` API URL override
``NIFI_REGISTRY_BRANCH`` Default branch
``NIFI_REPOSITORY_PATH`` Path within repository
``NIFI_ADO_ORG`` Azure DevOps organization
``NIFI_ADO_PROJECT`` Azure DevOps project
``NIFI_ADO_REPO`` Azure DevOps repository name
``NIFI_ADO_TENANT_ID`` Azure DevOps (Entra) tenant id
``NIFI_ADO_CLIENT_ID`` Azure DevOps service principal client id
``NIFI_ADO_CLIENT_SECRET`` Azure DevOps service principal secret
============================= ============================================

Flow Operations
Expand Down Expand Up @@ -712,6 +754,29 @@ For GitLab CI, include the fragments template:
- !reference [.nipyapi, deploy-flow]
- !reference [.nipyapi, start-flow]

Azure Pipelines
---------------

For Azure Pipelines, the CLI auto-detects the environment (via ``TF_BUILD``) and emits
``##vso[task.setvariable]`` logging commands, so operation outputs become pipeline
variables referenceable in later steps. See the `nipyapi-actions
<https://github.com/Chaffelson/nipyapi-actions>`_ repository for a complete pipeline
and reusable step template.

.. code-block:: yaml

variables:
- group: nipyapi-ado # NIFI_API_ENDPOINT, NIFI_ADO_* credentials, etc.

steps:
- script: pip install "nipyapi[cli]"
displayName: Install nipyapi
- script: |
nipyapi ci ensure_registry --provider azuredevops
nipyapi ci deploy_flow --bucket connectors --flow postgresql
nipyapi ci start_flow --process_group_id $(process-group-id)
displayName: Deploy NiFi flow

Python Usage
============

Expand Down
18 changes: 15 additions & 3 deletions docs/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -265,14 +265,26 @@ The CLI automatically formats output based on the execution environment:
nipyapi ci deploy_flow ... > deploy.env
# Output: PROCESS_GROUP_ID=abc-123

**Azure Pipelines**
Auto-detected when ``TF_BUILD`` env var is set. Emits ``##vso[task.setvariable]``
logging commands so each output becomes a pipeline variable (``isOutput=true``),
referenceable in later steps/jobs.

.. code-block:: console

# In an Azure Pipelines step
nipyapi ci deploy_flow ...
# Output: ##vso[task.setvariable variable=process_group_id;isOutput=true]abc-123

**Manual Override**
Force a specific format using ``NIFI_OUTPUT_FORMAT``:

.. code-block:: console

export NIFI_OUTPUT_FORMAT=json # JSON (default)
export NIFI_OUTPUT_FORMAT=github # GitHub Actions format
export NIFI_OUTPUT_FORMAT=dotenv # GitLab dotenv format
export NIFI_OUTPUT_FORMAT=json # JSON (default)
export NIFI_OUTPUT_FORMAT=github # GitHub Actions format
export NIFI_OUTPUT_FORMAT=dotenv # GitLab dotenv format
export NIFI_OUTPUT_FORMAT=azurepipelines # Azure Pipelines format

Log Level Control
=================
Expand Down
22 changes: 21 additions & 1 deletion docs/history.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,32 @@ History
1.7.0 (2026-07-18)
-------------------

| Upgrade to NiFi 2.10.0 support with regenerated API clients
| NiFi 2.10.0 support and Azure DevOps Flow Registry Client integration

**Supported Versions**

- Bumped the supported and tested NiFi/Registry version from 2.7.2 to 2.10.0 and regenerated the NiFi and Registry API clients against the 2.10.0 OpenAPI specification. New Connectors API operations introduced ``operationId`` collisions in the generated clients; the affected hand-written call sites were updated accordingly (for example ``get_flow1``, ``create_asset1``, ``get_assets1``, and the renumbered ``update_run_status`` variants).

**Registry Client**

- **ensure_azuredevops_registry()**: New ``nipyapi.versioning`` function to create or update an Azure DevOps Flow Registry Client. Unlike GitHub and GitLab (which use a personal access token), Azure DevOps authenticates with a service principal via OAuth2 client credentials, so the client is provisioned with the supporting management controller services.

**Canvas Module**

- **create_management_controller()** / **ensure_management_controller()**: New functions for controller-level (management) Controller Services — the kind referenced by Flow Registry Clients, which live at the Controller Settings level rather than inside a Process Group. ``ensure_management_controller`` is idempotent by name and validates supplied properties against the service's property descriptors.

**CI Module**

- **ensure_registry()**: Now accepts ``provider=azuredevops`` in addition to ``github`` and ``gitlab``. Azure DevOps organization, project, repository, and service-principal credentials are read from arguments or the corresponding ``NIFI_ADO_*`` environment variables.

**CLI**

- **Azure Pipelines output**: The CLI now auto-detects Azure Pipelines (via the ``TF_BUILD`` environment variable) and emits ``##vso[task.setvariable]`` logging commands so operation results round-trip as pipeline output variables. Can be forced with ``NIFI_OUTPUT_FORMAT=azurepipelines``.

**Documentation**

- Added Azure DevOps / Azure Pipelines coverage to the CI operations guide (``docs/ci.rst``), the CLI output-format reference (``docs/cli.rst``), and the project README. See the companion ``nipyapi-actions`` repository for a complete Azure Pipelines example.

1.6.0 (2026-06-21)
-------------------

Expand Down
94 changes: 94 additions & 0 deletions nipyapi/canvas.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@
"delete_connection",
"get_component_connections",
"create_controller",
"create_management_controller",
"ensure_management_controller",
"list_all_controllers",
"delete_controller",
"update_controller",
Expand Down Expand Up @@ -2149,6 +2151,98 @@ def create_controller(parent_pg, controller, name=None):
)


def create_management_controller(controller, name=None):
"""
Creates a new controller-level (management) Controller Service of the given
type, with the given Name.

Management controller services live at the Controller level (Controller
Settings), not inside a Process Group, and are the kind referenced by Flow
Registry Clients, Reporting Tasks, and similar. Unlike
:func:`create_controller`, no parent Process Group is required; the service
is created via the Controller API.

Args:
controller (:class:`~nipyapi.nifi.models.DocumentedTypeDTO`): Type of
Controller Service to create, found via the list_all_controller_types
method
name (str, optional): Name for the new Controller Service. If not
provided, defaults to the short type name (e.g.,
"StandardOauth2AccessTokenProvider").

Returns:
:class:`~nipyapi.nifi.models.ControllerServiceEntity`: The created controller service

"""
assert isinstance(controller, nipyapi.nifi.DocumentedTypeDTO)
assert name is None or isinstance(name, str)
# Default name to short type name if not provided (consistent with create_controller)
if name is None:
controller_name = controller.type.split(".")[-1]
else:
controller_name = name
with nipyapi.utils.rest_exceptions():
# NiFi 2.x creates a controller-level Controller Service via ControllerApi
return nipyapi.nifi.ControllerApi().create_controller_service(
body=nipyapi.nifi.ControllerServiceEntity(
revision={"version": 0},
component=nipyapi.nifi.ControllerServiceDTO(
bundle=controller.bundle,
type=controller.type,
name=controller_name,
),
),
)


def ensure_management_controller(name, controller_type, properties=None, enable=True):
"""
Ensure a controller-level (management) Controller Service exists, is
configured, and (optionally) enabled. Idempotent by name.

Composes the existing controller-service functions: finds an existing
service by name, creates it via :func:`create_management_controller` if
absent, applies ``properties`` via :func:`update_controller`, and enables it
via :func:`schedule_controller`.

Args:
name (str): Controller Service name (used for idempotent lookup).
controller_type (str): Fully-qualified controller service class name,
e.g. ``org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider``.
properties (dict, optional): Properties to set on the service.
enable (bool): Whether to enable the service (default True).

Returns:
:class:`~nipyapi.nifi.models.ControllerServiceEntity`: The ensured controller service

Raises:
ValueError: If the controller service type is not available on this NiFi.

"""
assert isinstance(name, str)
assert isinstance(controller_type, str)
assert properties is None or isinstance(properties, dict)
controller = get_controller(name, identifier_type="name", bool_response=True, greedy=False)
if not controller:
cs_type = get_controller_type(controller_type, identifier_type="name", greedy=False)
if cs_type is None:
raise ValueError(
f"Controller service type '{controller_type}' is not available "
"on this NiFi instance"
)
controller = create_management_controller(cs_type, name=name)
if properties:
# Validate the property keys against the service descriptors (same guard
# as elsewhere), so a typo raises instead of silently creating a dynamic
# property. auto_disable lets us re-apply to an existing, already-enabled
# service (idempotent re-invocation) instead of raising.
config = prepare_controller_config(controller, properties)
controller = update_controller(controller, update=config, auto_disable=True)
if enable:
controller = schedule_controller(controller, scheduled=True, refresh=True)
return controller


def list_all_controllers( # pylint: disable=too-many-arguments,too-many-positional-arguments
pg_id="root",
descendants=True,
Expand Down
Loading