Skip to content

{Storage} Remove DATA_STORAGE and DATA_COSMOS_TABLE client type reference#31583

Merged
calvinhzy merged 27 commits into
Azure:devfrom
calvinhzy:storage-azure-stack-cleanup
Aug 21, 2025
Merged

{Storage} Remove DATA_STORAGE and DATA_COSMOS_TABLE client type reference#31583
calvinhzy merged 27 commits into
Azure:devfrom
calvinhzy:storage-azure-stack-cleanup

Conversation

@calvinhzy
Copy link
Copy Markdown
Member

@calvinhzy calvinhzy commented May 30, 2025

Related command

Description

Remove DATA_STORAGE and DATA_COSMOS_TABLE usages, which have been deprecated.
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature


This checklist is used to make sure that common guidelines for a pull request are followed.

@calvinhzy calvinhzy self-assigned this May 30, 2025
@azure-client-tools-bot-prd
Copy link
Copy Markdown

azure-client-tools-bot-prd Bot commented May 30, 2025

️✔️AzureCLI-FullTest
️✔️acr
️✔️latest
️✔️3.12
️✔️3.9
️✔️acs
️✔️latest
️✔️3.12
️✔️3.9
️✔️advisor
️✔️latest
️✔️3.12
️✔️3.9
️✔️ams
️✔️latest
️✔️3.12
️✔️3.9
️✔️apim
️✔️latest
️✔️3.12
️✔️3.9
️✔️appconfig
️✔️latest
️✔️3.12
️✔️3.9
️✔️appservice
️✔️latest
️✔️3.12
️✔️3.9
️✔️aro
️✔️latest
️✔️3.12
️✔️3.9
️✔️backup
️✔️latest
️✔️3.12
️✔️3.9
️✔️batch
️✔️latest
️✔️3.12
️✔️3.9
️✔️batchai
️✔️latest
️✔️3.12
️✔️3.9
️✔️billing
️✔️latest
️✔️3.12
️✔️3.9
️✔️botservice
️✔️latest
️✔️3.12
️✔️3.9
️✔️cdn
️✔️latest
️✔️3.12
️✔️3.9
️✔️cloud
️✔️latest
️✔️3.12
️✔️3.9
️✔️cognitiveservices
️✔️latest
️✔️3.12
️✔️3.9
️✔️compute_recommender
️✔️latest
️✔️3.12
️✔️3.9
️✔️computefleet
️✔️latest
️✔️3.12
️✔️3.9
️✔️config
️✔️latest
️✔️3.12
️✔️3.9
️✔️configure
️✔️latest
️✔️3.12
️✔️3.9
️✔️consumption
️✔️latest
️✔️3.12
️✔️3.9
️✔️container
️✔️latest
️✔️3.12
️✔️3.9
️✔️containerapp
️✔️latest
️✔️3.12
️✔️3.9
️✔️core
️✔️latest
️✔️3.12
️✔️3.9
️✔️cosmosdb
️✔️latest
️✔️3.12
️✔️3.9
️✔️databoxedge
️✔️latest
️✔️3.12
️✔️3.9
️✔️dls
️✔️latest
️✔️3.12
️✔️3.9
️✔️dms
️✔️latest
️✔️3.12
️✔️3.9
️✔️eventgrid
️✔️latest
️✔️3.12
️✔️3.9
️✔️eventhubs
️✔️latest
️✔️3.12
️✔️3.9
️✔️feedback
️✔️latest
️✔️3.12
️✔️3.9
️✔️find
️✔️latest
️✔️3.12
️✔️3.9
️✔️hdinsight
️✔️latest
️✔️3.12
️✔️3.9
️✔️identity
️✔️latest
️✔️3.12
️✔️3.9
️✔️iot
️✔️latest
️✔️3.12
️✔️3.9
️✔️keyvault
️✔️latest
️✔️3.12
️✔️3.9
️✔️lab
️✔️latest
️✔️3.12
️✔️3.9
️✔️managedservices
️✔️latest
️✔️3.12
️✔️3.9
️✔️maps
️✔️latest
️✔️3.12
️✔️3.9
️✔️marketplaceordering
️✔️latest
️✔️3.12
️✔️3.9
️✔️monitor
️✔️latest
️✔️3.12
️✔️3.9
️✔️mysql
️✔️latest
️✔️3.12
️✔️3.9
️✔️netappfiles
️✔️latest
️✔️3.12
️✔️3.9
️✔️network
️✔️latest
️✔️3.12
️✔️3.9
️✔️policyinsights
️✔️latest
️✔️3.12
️✔️3.9
️✔️privatedns
️✔️latest
️✔️3.12
️✔️3.9
️✔️profile
️✔️latest
️✔️3.12
️✔️3.9
️✔️rdbms
️✔️latest
️✔️3.12
️✔️3.9
️✔️redis
️✔️latest
️✔️3.12
️✔️3.9
️✔️relay
️✔️latest
️✔️3.12
️✔️3.9
️✔️resource
️✔️latest
️✔️3.12
️✔️3.9
️✔️role
️✔️latest
️✔️3.12
️✔️3.9
️✔️search
️✔️latest
️✔️3.12
️✔️3.9
️✔️security
️✔️latest
️✔️3.12
️✔️3.9
️✔️servicebus
️✔️latest
️✔️3.12
️✔️3.9
️✔️serviceconnector
️✔️latest
️✔️3.12
️✔️3.9
️✔️servicefabric
️✔️latest
️✔️3.12
️✔️3.9
️✔️signalr
️✔️latest
️✔️3.12
️✔️3.9
️✔️sql
️✔️latest
️✔️3.12
️✔️3.9
️✔️sqlvm
️✔️latest
️✔️3.12
️✔️3.9
️✔️storage
️✔️latest
️✔️3.12
️✔️3.9
️✔️synapse
️✔️latest
️✔️3.12
️✔️3.9
️✔️telemetry
️✔️latest
️✔️3.12
️✔️3.9
️✔️util
️✔️latest
️✔️3.12
️✔️3.9
️✔️vm
️✔️latest
️✔️3.12
️✔️3.9

@azure-client-tools-bot-prd
Copy link
Copy Markdown

azure-client-tools-bot-prd Bot commented May 30, 2025

⚠️AzureCLI-BreakingChangeTest
⚠️storage
rule cmd_name rule_message suggest_message
⚠️ 1006 - ParaAdd storage blob incremental-copy cancel cmd storage blob incremental-copy cancel added parameter account_url
⚠️ 1010 - ParaPropUpdate storage blob incremental-copy cancel cmd storage blob incremental-copy cancel update parameter lease_id: updated property name from lease_id to lease
⚠️ 1006 - ParaAdd storage blob incremental-copy start cmd storage blob incremental-copy start added parameter account_url
⚠️ 1009 - ParaPropRemove storage blob incremental-copy start cmd storage blob incremental-copy start update parameter destination_if_modified_since: removed property type=custom_type
⚠️ 1009 - ParaPropRemove storage blob incremental-copy start cmd storage blob incremental-copy start update parameter destination_if_unmodified_since: removed property type=custom_type
⚠️ 1010 - ParaPropUpdate storage blob incremental-copy start cmd storage blob incremental-copy start update parameter destination_lease_id: updated property name from destination_lease_id to destination_lease
⚠️ 1010 - ParaPropUpdate storage blob incremental-copy start cmd storage blob incremental-copy start update parameter source_lease_id: updated property name from source_lease_id to source_lease
⚠️ 1006 - ParaAdd storage blob sync cmd storage blob sync added parameter account_url
⚠️ 1006 - ParaAdd storage blob sync cmd storage blob sync added parameter extra_options
⚠️ 1009 - ParaPropRemove storage blob sync cmd storage blob sync update parameter extra_options: removed property nargs=*
⚠️ 1010 - ParaPropUpdate storage blob sync cmd storage blob sync update parameter extra_options: updated property name from extra_options to account_name
⚠️ 1010 - ParaPropUpdate storage blob sync cmd storage blob sync update parameter extra_options: updated property options from [] to ['--account-name']
⚠️ 1006 - ParaAdd storage copy cmd storage copy added parameter account_url
⚠️ 1006 - ParaAdd storage copy cmd storage copy added parameter extra_options
⚠️ 1008 - ParaPropAdd storage copy cmd storage copy update parameter extra_options: added property options_deprecate_info=[{'target': '--destination-account-name', 'redirect': '--account-name'}]
⚠️ 1009 - ParaPropRemove storage copy cmd storage copy update parameter extra_options: removed property nargs=*
⚠️ 1010 - ParaPropUpdate storage copy cmd storage copy update parameter extra_options: updated property name from extra_options to account_name
⚠️ 1010 - ParaPropUpdate storage copy cmd storage copy update parameter extra_options: updated property options from [] to ['--account-name', '--destination-account-name']
⚠️ 1006 - ParaAdd storage cors add cmd storage cors add added parameter account_url
⚠️ 1006 - ParaAdd storage cors clear cmd storage cors clear added parameter account_url
⚠️ 1006 - ParaAdd storage cors list cmd storage cors list added parameter account_url
⚠️ 1006 - ParaAdd storage fs directory download cmd storage fs directory download added parameter account_url
⚠️ 1006 - ParaAdd storage fs directory upload cmd storage fs directory upload added parameter account_url
⚠️ 1006 - ParaAdd storage logging off cmd storage logging off added parameter account_url
⚠️ 1006 - ParaAdd storage logging show cmd storage logging show added parameter account_url
⚠️ 1006 - ParaAdd storage logging update cmd storage logging update added parameter account_url
⚠️ 1006 - ParaAdd storage metrics show cmd storage metrics show added parameter account_url
⚠️ 1006 - ParaAdd storage metrics update cmd storage metrics update added parameter account_url
⚠️ 1006 - ParaAdd storage remove cmd storage remove added parameter account_url

@yonzhan
Copy link
Copy Markdown
Collaborator

yonzhan commented May 30, 2025

Thank you for your contribution! We will review the pull request and get back to you soon.

@github-actions
Copy link
Copy Markdown

The git hooks are available for azure-cli and azure-cli-extensions repos. They could help you run required checks before creating the PR.

Please sync the latest code with latest dev branch (for azure-cli) or main branch (for azure-cli-extensions).
After that please run the following commands to enable git hooks:

pip install azdev --upgrade
azdev setup -c <your azure-cli repo path> -r <your azure-cli-extensions repo path>

@calvinhzy calvinhzy changed the title Storage azure stack cleanup {Storage} Remove DATA_STORAGE client type reference Jun 15, 2025
@calvinhzy
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 3 pipeline(s).

@calvinhzy
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 3 pipeline(s).

@calvinhzy calvinhzy changed the title {Storage} Remove DATA_STORAGE client type reference {Storage} Remove DATA_STORAGE and DATA_COSMOS_TABLE client type reference Jul 28, 2025
@calvinhzy calvinhzy marked this pull request as ready for review July 28, 2025 08:28
Copilot AI review requested due to automatic review settings July 28, 2025 08:28

This comment was marked as outdated.

@calvinhzy calvinhzy requested a review from Copilot August 7, 2025 08:57
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR removes deprecated DATA_STORAGE and DATA_COSMOS_TABLE client type references from the Azure CLI storage module, upgrading to more modern storage SDKs. The changes focus on modernizing the storage command module by removing legacy Track1 SDK dependencies and consolidating on Track2 implementations.

  • Updates dependency versions for azure-multiapi-storage from 1.4.1 to 1.5.0 across all platform requirement files
  • Removes deprecated DATA_STORAGE and DATA_COSMOS_TABLE resource type usage in favor of more specific resource types
  • Implements incremental blob copy operations using AAZ (Azure AutoRest for CLI) framework instead of legacy SDKs

Reviewed Changes

Copilot reviewed 41 out of 43 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
setup.py & requirements files Updates azure-multiapi-storage dependency from 1.4.1 to 1.5.0
vm/custom.py Modernizes blob client usage from BlockBlobService to BlobClient
storage/util.py Removes deprecated Track1 SDK utility functions
storage/tests Updates test expectations and removes deprecated API version constraints
storage/operations Adds new AAZ-based incremental copy implementation
storage/commands.py Simplifies command registrations by removing deprecated client factories
storage/_validators.py Updates validators to use Track2 SDK patterns exclusively
Comments suppressed due to low confidence (1)

src/azure-cli/azure/cli/command_modules/vm/custom.py:48

  • The condition isinstance(blob.name, str) will always be true in Python 3, making the else clause unreachable. The original code checked for unicode type which doesn't exist in Python 3. Consider removing this condition entirely or clarifying the intended logic.
from .aaz.latest.vm.disk import AttachDetachDataDisk

self.assertEqual(actual, expected)
actual = str(services_type_v2()(input))
expected = "bqtf"
self.assertEqual(set(actual), set(expected))
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is comparing sets instead of strings, which may not properly validate the exact output format. The original test compared strings directly. This change could mask ordering issues or allow incorrect character combinations.

Suggested change
self.assertEqual(set(actual), set(expected))
self.assertEqual(actual, expected)

Copilot uses AI. Check for mistakes.

_Start.PageBlobCopyIncremental.on_202 = on_202

_Start(command_args=cmd_args)
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function creates an instance _Start but doesn't handle potential exceptions that could occur during execution. Consider adding appropriate error handling around this call.

Suggested change
_Start(command_args=cmd_args)
try:
_Start(command_args=cmd_args)
except (HttpResponseError, AzureResponseError) as ex:
logger.error("Failed to start incremental copy: %s", ex)
raise CLIError(f"Failed to start incremental copy: {ex}")
except Exception as ex:
logger.error("An unexpected error occurred during incremental copy: %s", ex)
raise CLIError(f"An unexpected error occurred during incremental copy: {ex}")

Copilot uses AI. Check for mistakes.
Comment on lines +72 to 75
if not sas_token and client.credential and hasattr(client.credential, "account_key"):
sas_token = _generate_sas_token(cmd, client.account_name, client.credential.account_key, service=service,
resource_types='co',
permissions='rdl')
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code uses hasattr to check for account_key which is fragile. Consider using a more explicit type check or try-except pattern to handle different credential types more robustly.

Suggested change
if not sas_token and client.credential and hasattr(client.credential, "account_key"):
sas_token = _generate_sas_token(cmd, client.account_name, client.credential.account_key, service=service,
resource_types='co',
permissions='rdl')
if not sas_token and client.credential:
try:
account_key = client.credential.account_key
except AttributeError:
account_key = None
if account_key:
sas_token = _generate_sas_token(cmd, client.account_name, account_key, service=service,
resource_types='co',
permissions='rdl')

Copilot uses AI. Check for mistakes.
ns['source_share'] = source_share
# get sas token for source
if not source_sas and not is_oauth:
if not source_sas and not is_oauth and source_key != 'fake_key':
Copy link

Copilot AI Aug 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hardcoded string 'fake_key' comparison is a code smell. Consider using a constant or a more explicit way to identify test/fake credentials.

Suggested change
if not source_sas and not is_oauth and source_key != 'fake_key':
if not source_sas and not is_oauth and source_key != FAKE_STORAGE_ACCOUNT_KEY:

Copilot uses AI. Check for mistakes.
@calvinhzy calvinhzy merged commit 3cc5311 into Azure:dev Aug 21, 2025
48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Auto-Assign Auto assign by bot Storage az storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants