Deletion of ARO cluster is not working due to insufficient permissions - using ServicePrincipal

[im-konge]

This is autogenerated. Please review and update as needed.

Describe the bug

Command Name
az aro delete

Errors:

The command failed with an unexpected error. Here is the traceback:
'GraphError' object has no attribute 'message'
Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/role/_msgrpah/_graph_client.py", line 52, in _send
    r = send_raw_request(self._cli_ctx, method, url, resource=self._resource, uri_parameters=param,
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/core/util.py", line 993, in send_raw_request
    raise HTTPError(reason, r)
azure.cli.core.azclierror.HTTPError: Forbidden({"error":{"code":"Authorization_RequestDenied","message":"Insufficient privileges to complete the operation.","innerError":{"date":"2022-10-05T13:41:17","request-id":"f46c8a3c-8bc3-44f0-b485-f72bfe950021","client-request-id":"f46c8a3c-8bc3-44f0-b485-f72bfe950021"}}})

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/aro/custom.py", line 168, in aro_delete
    rp_client_sp_id = aad.get_service_principal_id(resolve_rp_client_id())
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/aro/_aad.py", line 31, in get_service_principal_id
    sps = self.client.service_principal_list(f"appId eq '{app_id}'")
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/role/_msgrpah/_graph_client.py", line 179, in service_principal_list
    result = self._send("GET", "/servicePrincipals" + _filter_to_query(filter))
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/role/_msgrpah/_graph_client.py", line 55, in _send
    raise GraphError(ex.response.json()['error']['message'], ex.response) from ex
azure.cli.command_modules.role._msgrpah._graph_client.GraphError: Insufficient privileges to complete the operation.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 231, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/usr/local/Cellar/azure-cli/2.39.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/aro/custom.py", line 172, in aro_delete
    logger.info(e.message)
AttributeError: 'GraphError' object has no attribute 'message'

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Put any pre-requisite steps here...
• az aro delete --name {} --resource-group {} -y

Expected Behavior

Environment Summary

macOS-12.6-x86_64-i386-64bit, Darwin 21.6.0
Python 3.10.6
Installer: HOMEBREW

azure-cli 2.39.0 *

Additional Context

[im-konge]

Hi,
before some kind of upgrade of az-cli, our automation with ServicePrincipal was working without a problem.
Currently, we are able to create ARO clusters using the ServicePrincipal, but we are getting exceptions when we are trying to delete it.
The thing is that we already added Owner role to our ServicePrincipal, which was sufficient before.
What else should be set (what permissions) for successfully deleting the cluster?

Also, isn't a bit strange that we are able to create clusters, but not delete them?

This is more question than bug report.

Thank you for your feedback. This has been routed to the support team for assistance.

[yonzhan]

route to CXP team

[novotnyJiri]

I encountered the exact same problem. What is the status of this issue?

[jsafarik]

Any progress on this issue?

[jsntcy]

@jiasli, could you please take a look?

[hikingcodercamps]

As mentioned by a commenter on the related #27146 this error is coming from AAD. I have seen this same behavior before when comparing a run of this command with my personal account which matched an automation service principal against the Azure subscription. It always worked for me but failed for the automation service principal because the automation service principal was heavily locked down and had no permissions to read or list anything in AAD.

The documentation at https://learn.microsoft.com/en-us/azure/openshift/tutorial-delete-cluster does not cover any permissions needed for the service principal executing the command against AAD but based on the error message performing
rp_client_sp_id = aad.get_service_principal_id(resolve_rp_client_id())
sps = self.client.service_principal_list(f"appId eq '{app_id}'")
result = self._send("GET", "/servicePrincipals" + _filter_to_query(filter))
there are actions taken in AAD which would require some type of permissions outside of the Azure Subscription.

If there is an assumption for minimum permissions needed for az cli I cannot find it documented anywhere. What are the minimum permissions needed outside of the Azure subscription and within the AAD tenant itself to perform az aro delete?

Also for anyone unsure if they are hitting this the clear signal is that you have all the right inputs but nothing shows up in the Azure Subscription Activity Log. Why? Because it fails at AAD before it ever hits the Azure Subscription with the cluster.

I suspect the calling of az aro delete needs to look up the Azure Red Hat OpenShift RP enterprise app registration as it will have a different app id in each customer's tenant and it is needed to execute the actual deletion as it was used in provisioning the cluster. Calling az aro delete is simply the trigger for "Azure Red Hat OpenShift RP" to do the work.

[im-konge]

Is there any progress on this issue? It's opened for quite a long time and I'm a bit blocked on this -> as I cannot upgrade to latest version of the CLI, but I need to delete the clusters using service principal in our automation.

I also see that the other issues that are related to this one are not resolved. Is it matter of permissions that need to be documented somewhere? Is it a bug in the CLI itself? Is there anything we can do to fix that?

Thanks for the answers.