Skip to content

[typespec-python] simplify eng#4399

Draft
msyyc wants to merge 40 commits into
mainfrom
python-update-eng
Draft

[typespec-python] simplify eng#4399
msyyc wants to merge 40 commits into
mainfrom
python-update-eng

Conversation

@msyyc
Copy link
Copy Markdown
Member

@msyyc msyyc commented May 8, 2026 •
edited
Loading

Summary

Reduce duplication between typespec-python (a thin wrapper) and upstream @typespec/http-client-python so we don't keep maintaining two copies of the same eng/test scaffolding by hand.

Approach

  • eng/scripts/sync.ts — new allow-list mirror that copies a curated set of shared files from a sibling checkout (<repo-root>/core/packages/http-client-python/) into this package. Supports --check for CI drift detection.
  • eng/scripts/ci/regenerate-common.ts — shared helpers / spec tables / prepareBaselineOfGeneratedCode, byte-identical with the upstream copy. Synced.
  • eng/scripts/ci/regenerate.ts — kept per-repo: wrapper does single-phase in-process regen; upstream does two-phase. Imports common helpers, owns only paths/argv/orchestration.
  • dev_requirements.txt — synced as a section (everything before the # additional dependency needed for development marker is replaced from upstream; wrapper-only deps like setuptools are preserved).
  • Synced shared CI runners: run_apiview.py, run_mypy.py, run_pylint.py, run_pyright.py, run_sphinx_build.py, util.py, plus mypy.ini, pylintrc, pyrightconfig.json.
  • Synced shared test assets / drivers: tests/data/, tests/mock_api/, tests/requirements/, tests/conftest.py, tests/install_packages.py, tests/tox.ini, and the generator/ (pygen) tree.
  • eng/scripts/setup/* is intentionally not synced — the wrapper owns its own setup pipeline (install.py, prepare.py, venvtools.py, …).
  • tests/mock_api folder which contains test files is not kept. Instead, eng will download from upstream in runtime by "pnpm run sync" so that we don't need maintain the test cases in 2 repos.
  • .gitignore updated so synced directories (mirrored on demand by pnpm sync) start empty in fresh checkouts.

Result

Most engineering files now have a single upstream source of truth; wrapper-specific divergence is limited to the orchestrator (regenerate.ts), the setup scripts, and the small wrapper-only tail of dev_requirements.txt. CI keeps everything honest via pnpm sync --check.

@microsoft-github-policy-service microsoft-github-policy-service Bot added the emitter:python Issues for @azure-tools/typespec-python emitter label May 8, 2026
@azure-sdk
Copy link
Copy Markdown
Collaborator

azure-sdk commented May 8, 2026 •
edited
Loading

All changed packages have been documented.

  • ✅ @azure-tools/typespec-python
Show changes

@azure-tools/typespec-python - internal ✏️

Simplify eng system for typespec-python

@github-actions

This comment was marked as off-topic.

Sign in to view
@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 8, 2026 •
edited
Loading

Open in StackBlitz

npm i https://pkg.pr.new/@azure-tools/typespec-python@4399

commit: a01cf7a

@azure-sdk
Copy link
Copy Markdown
Collaborator

azure-sdk commented May 8, 2026 •
edited
Loading

You can try these changes here

🛝 Playground 🌐 Website

@Azure Azure deleted a comment from github-actions Bot May 8, 2026
msyyc and others added 2 commits May 9, 2026 11:30
@msyyc
Revert "update setup-python"
aff58bf 
This reverts commit 86253df.
@msyyc
Restore Azure's working install_packages.py and exclude from sync
bde5c8d 
Upstream http-client-python's tests/install_packages.py invokes 'uv pip
wheel', which is not a real uv subcommand in any released uv version
(verified against uv 0.5.0 and 0.11.x source: the PipCommand enum has
no Wheel variant). Syncing it into the wrapper broke the 'Pre-build
wheels' CI step.

Restore the working version from origin/main (which uses 'uv build
--wheel ...' per package with proper CalledProcessError handling) and
remove the file from sync.ts INCLUDES so it stops being overwritten.
A comment in sync.ts explains why and notes to re-include once the
upstream script is fixed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@msyyc msyyc force-pushed the python-update-eng branch from 4086527 to bde5c8d Compare May 9, 2026 03:35
@Azure Azure deleted a comment from github-actions Bot May 9, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026 •
edited
Loading

⚡ Benchmark Results

⚠️ 4 metric(s) regressed above the +5% threshold:

Metric Baseline Current Change
loader 🟢 130.8ms 🟢 144.3ms +10.4% 🔴
resolver 🟢 15.9ms 🟢 20.1ms +26.7% 🔴
 ↳ emit/@typespec/openapi3 🟢 131.4ms 🟢 138.3ms +5.3% 🔴
 ↳ emit/@typespec/openapi3/write 🟢 13.4ms 🟢 14.6ms +9.2% 🔴
Full details – comparing bd475d6 vs baseline b166149
Metric Baseline Current Change
total 🔴 734.5ms 🔴 762.8ms +3.9%
loader 🟢 130.8ms 🟢 144.3ms +10.4% 🔴
resolver 🟢 15.9ms 🟢 20.1ms +26.7% 🔴
checker 🟢 176.4ms 🟢 179.5ms +1.7%
validation 🟢 41.2ms 🟢 42.6ms +3.2%
 ↳ validation/@azure-tools/typespec-azure-core 🟢 5.9ms 🟢 5.9ms +0.8%
 ↳ validation/@typespec/http 🟢 5.7ms 🟢 5.3ms -8.0%
 ↳ validation/@typespec/rest 🟢 0.6ms 🟢 0.8ms +33.5%
 ↳ validation/@typespec/versioning 🔴 27.2ms 🔴 28.2ms +3.7%
 ↳ validation/compiler 🟢 1.4ms 🟢 1.8ms +27.8%
linter 🟢 125.8ms 🟢 121.7ms -3.3%
 ↳ linter/@azure-tools/typespec-azure-core/auth-required 🟢 0.0ms 🟢 0.0ms +4.5%
 ↳ linter/@azure-tools/typespec-azure-core/bad-record-type 🟢 0.2ms 🟢 0.2ms -7.8%
 ↳ linter/@azure-tools/typespec-azure-core/byos 🟢 5.7ms 🟢 5.7ms +0.4%
 ↳ linter/@azure-tools/typespec-azure-core/casing-style 🟢 0.5ms 🟢 0.6ms +14.9%
 ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance 🟢 0.1ms 🟢 0.1ms +10.0%
 ↳ linter/@azure-tools/typespec-azure-core/documentation-required 🟢 0.7ms 🟢 0.9ms +22.3%
 ↳ linter/@azure-tools/typespec-azure-core/friendly-name 🟢 0.6ms 🟢 0.6ms -5.3%
 ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required 🟢 0.1ms 🟢 0.2ms +4.1%
 ↳ linter/@azure-tools/typespec-azure-core/known-encoding 🟢 0.2ms 🟢 0.2ms +8.9%
 ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required 🟢 0.3ms 🟢 0.3ms +2.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch 🟢 0.2ms 🟢 0.2ms +9.0%
 ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union 🟢 0.3ms 🟢 0.3ms -21.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-enum 🟢 0.0ms 🟢 0.0ms +18.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes 🟢 0.1ms 🟢 0.1ms -2.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops 🟢 0.1ms 🟢 0.1ms +3.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-format 🟢 0.5ms 🟢 0.5ms -2.2%
 ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric 🟢 0.3ms 🟢 0.4ms +34.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-header-explode 🟡 18.3ms 🟡 16.8ms -8.2% 🟢
 ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage 🟢 1.0ms 🟢 1.0ms +3.7%
 ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator 🟢 0.1ms 🟢 0.1ms +17.2%
 ↳ linter/@azure-tools/typespec-azure-core/no-nullable 🟢 0.2ms 🟢 0.2ms +4.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime 🟢 1.1ms 🟢 1.2ms +6.3%
 ↳ linter/@azure-tools/typespec-azure-core/no-openapi 🟢 1.7ms 🟢 1.8ms +4.0%
 ↳ linter/@azure-tools/typespec-azure-core/no-private-usage 🟢 1.6ms 🟢 1.9ms +13.4%
 ↳ linter/@azure-tools/typespec-azure-core/no-query-explode 🟡 17.7ms 🟡 16.9ms -4.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-response-body 🔴 21.4ms 🔴 20.7ms -3.1%
 ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces 🟢 0.0ms 🟢 0.0ms +9.7%
 ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch 🟢 4.8ms 🟢 4.6ms -3.5%
 ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params 🟢 0.2ms 🟢 0.2ms +7.6%
 ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator 🟢 0.0ms 🟢 0.0ms +6.8%
 ↳ linter/@azure-tools/typespec-azure-core/no-unknown 🟢 0.2ms 🟢 0.2ms -13.4%
 ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union 🟢 0.3ms 🟢 0.3ms +4.8%
 ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version 🟢 0.2ms 🟢 0.2ms +17.8%
 ↳ linter/@azure-tools/typespec-azure-core/request-body-problem 🟢 0.2ms 🟢 0.3ms +27.4%
 ↳ linter/@azure-tools/typespec-azure-core/require-versioned 🟢 0.0ms 🟢 0.0ms +26.1%
 ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem 🔴 20.8ms 🟡 19.5ms -6.3% 🟢
 ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body 🟢 0.3ms 🟢 0.3ms -1.0%
 ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model 🟢 0.2ms 🟢 0.2ms +9.7%
 ↳ linter/@azure-tools/typespec-azure-core/use-standard-names 🟢 4.7ms 🟢 4.6ms -2.7%
 ↳ linter/@azure-tools/typespec-azure-core/use-standard-operations 🟢 0.1ms 🟢 0.1ms +2.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version 🟢 4.5ms 🟢 3.8ms -15.7%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key 🟢 0.1ms 🟢 0.1ms -7.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage 🟢 0.1ms 🟢 0.1ms -9.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes 🟢 5.6ms 🟢 4.9ms -12.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts 🟢 5.1ms 🟢 4.3ms -15.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record 🟢 0.3ms 🟢 0.4ms +2.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes 🟢 0.5ms 🟢 0.4ms -12.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes 🟢 0.0ms 🟢 0.0ms +2.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment 🟢 0.2ms 🟢 0.2ms -4.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property 🟢 0.1ms 🟢 0.1ms -8.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator 🟢 0.0ms 🟢 0.0ms +12.4%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb 🟢 0.1ms 🟢 0.1ms -9.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property 🟢 0.1ms 🟢 0.1ms -10.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format 🟢 0.0ms 🟢 0.0ms +3.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars 🟢 0.2ms 🟢 0.2ms -2.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern 🟢 0.0ms 🟢 0.0ms +3.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation 🟢 0.2ms 🟢 0.2ms -3.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response 🟢 5.6ms 🟢 4.5ms -18.6% 🟢
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch 🟢 0.3ms 🟢 0.3ms +1.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars 🟢 0.2ms 🟢 0.2ms -11.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state 🟢 0.1ms 🟢 0.1ms -6.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels 🟢 0.1ms 🟢 0.1ms -10.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties 🟢 0.1ms 🟢 0.1ms -6.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation 🟢 0.0ms 🟢 0.0ms -13.0%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header 🟡 13.3ms 🟡 12.7ms -4.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint 🟢 0.0ms 🟢 0.0ms +19.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers 🟢 0.3ms 🟢 0.3ms -2.6%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model 🟢 0.1ms 🟢 0.1ms +0.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation 🟢 0.2ms 🟢 0.2ms +7.8%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body 🔴 20.5ms 🟡 19.2ms -6.3% 🟢
 ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope 🟢 0.1ms 🟢 0.1ms -7.3%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name 🟢 0.1ms 🟢 0.1ms +0.2%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop 🟢 2.4ms 🟢 2.8ms +12.5%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type 🟢 0.4ms 🟢 0.4ms +1.1%
 ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression 🟢 0.0ms 🟢 0.0ms +11.0%
 ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict 🟢 1.0ms 🟢 1.0ms +1.9%
 ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix 🟢 0.2ms 🟢 0.2ms +15.1%
emit 🟡 237.5ms 🟡 248.9ms +4.8%
 ↳ emit/@azure-tools/typespec-autorest 🟢 149.0ms 🟢 155.6ms +4.4%
 ↳ emit/@typespec/openapi3 🟢 131.4ms 🟢 138.3ms +5.3% 🔴
 ↳ emit/@typespec/openapi3/compute 🟢 118.1ms 🟢 123.0ms +4.1%
 ↳ emit/@typespec/openapi3/write 🟢 13.4ms 🟢 14.6ms +9.2% 🔴

Averaged across 3 specs (azure-arm-resource-manager, azure-core-dataplane, azure-full).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)

msyyc and others added 7 commits May 9, 2026 13:40
@msyyc
Update sync.ts/gitignore comments to reflect tox.ini and generator/ b…
90e08a8 
…eing wrapper-owned

Following the recent commits that:
- removed 	ests/tox.ini and generator/ from sync.ts INCLUDES (now
  wrapper-owned, committed)
- removed generator/ from .gitignore (now tracked)
- committed tests/requirements/* and tests/conftest.py as synced snapshots

…the prose comments around the INCLUDES list and the gitignore header
no longer matched what's actually being synced. Refresh them so a new
reader can tell at a glance:
  * which files are upstream-truth (kept aligned via pnpm sync)
  * which files are wrapper-owned (and *why* — including the upstream
    install_packages.py bug tracked in microsoft/typespec#10636)
  * which synced paths are gitignored vs committed snapshots

No behavior change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@msyyc
Exclude tests/mock_api/unbranded/test_unbranded.py from sync
ab17822 
This upstream test is unbranded-only and doesn't apply to the Azure
wrapper. Add an EXCLUDED_FILES allow-list to sync.ts that:
  - skips the file during the copy phase, and
  - prunes it from the destination if a previous sync (or a manual
    edit) left it behind.

In --check mode the file is reported as drift so CI catches accidental
re-introduction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@msyyc
Merge branch 'main' into python-update-eng
a64737e
@msyyc
add debug configuration for typespec-python
72425b4
@msyyc
Merge branch 'main' into python-update-eng
8e300c1
@tadelesh
@msyyc
Backmerge release/may-2026 (tcgc@0.68.2 hotfix) (#4489)
ec68084 
Backmerge of hotfix release tcgc@0.69.0 from `release/may-2026` into
`main`.

This ensures the version bump and fixes are reflected in main.

⚠️ **Rebase merge** this PR (do not squash).

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: tadelesh <1726438+tadelesh@users.noreply.github.com>
Co-authored-by: tadelesh <chenjieshi@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@msyyc
update ignore
5c22b43
@microsoft-github-policy-service microsoft-github-policy-service Bot added the lib:tcgc Issues for @azure-tools/typespec-client-generator-core library label May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bterlson bterlson Awaiting requested review from bterlson bterlson will be requested when the pull request is marked ready for review bterlson is a code owner

@markcowl markcowl Awaiting requested review from markcowl markcowl will be requested when the pull request is marked ready for review markcowl is a code owner

@timotheeguerin timotheeguerin Awaiting requested review from timotheeguerin timotheeguerin will be requested when the pull request is marked ready for review timotheeguerin is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

emitter:python Issues for @azure-tools/typespec-python emitter eng lib:tcgc Issues for @azure-tools/typespec-client-generator-core library

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@msyyc @azure-sdk @tadelesh