shared-compute implementation to support multiple GPU SKUs

[mgmachado]

Description

Adds multi-tenant shared GPU Batch pools so many environments can share scarce GPU quota instead of fragmenting it across one pool per environment, plus the v2.1.0 hastelib routing + per-job SAS changes that make it work.

• Shared pools (IaC): a parameterized batchPool.bicep (fixed/autoscale × dedicated/low-priority × optional VNet) and a new standalone shared-pools.bicep + shared-pools.bicepparam that provision an H100 + T4 pool pair per group (dev, demo, …), autoscaling on low-priority nodes and scaling to zero when idle.
• Data isolation: each Batch job mints a short-lived user-delegation SAS scoped to its own storage container; the pool holds no standing storage access (ACR-pull-only identity). A tenant's task cannot read another tenant's data — enforced by Azure at the credential boundary, not by app code.
• Capacity-aware routing: hastelib now takes ordered candidate pool lists and binds a pool at submit time (select_pool) — preferred tier first, spillover to a tier with a free node.
• Reusability: IaC defaults are genericized (HASTE_RESOURCE_PREFIX → neutral haste, BYO account/ACR), so other partners can deploy without editing the templates.

All new behavior is behind per-environment flags that default to the existing single-pool, pool-identity path, so existing environments are unaffected until they explicitly opt in.

Full design: spec/features/batch-compute-expansion/.

Fixes #

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that changes existing behavior)
• Documentation update
• Infrastructure / CI change

Checklist

• I have read CONTRIBUTING.md
• My changes follow the project's coding standards (PEP 8 for Python, ESLint rules for JS/TS)
• I have added or updated tests that cover my changes
• Python tests pass locally (cd hastelib && hatch run test:pytest) and the UI lints clean (cd ui && npm run lint) — new unit tests pass (9/9); full suite to run in CI (conda env)
• I have updated the relevant documentation (README, docs/, inline comments)
• I have added an entry to CHANGELOG.md if this is a user-facing change

Testing

• Unit (new): hastelib/tests/core/runners/test_azure_batch_routing.py — 9 tests covering select_pool (single / spillover-to-idle / preferred fallback), the SAS-vs-identity blob-credential toggle, and config candidate-list parsing. All pass.
• Isolation PoC (real Azure Batch): proved SAS-based ResourceFile/OutputFile I/O works with no identity_reference; a cross-tenant SAS was denied (BlobAccessDenied); the ACR-pull-only shared identity authenticated and pulled the training image.
• IaC: batchPool.bicep + shared-pools.bicep compile clean; what-if shows no drift against the live pools.
• Not yet run: end-to-end on two live environments — gated on a low-priority GPU quota increase (tracked in rollout.md).

Additional context

• No behavior change for existing envs. Opt-in per env via AZURE_BATCH_USE_SAS=true, AZURE_BATCH_MANAGE_POOLS=false, and AZURE_BATCH_{TRAINING,INFERENCE,IMAGERYPREP}_POOL_IDS; all default to legacy.
• Storage Blob Delegator is granted to the api/queues identity (in functionApp.bicep) so it can mint the SAS.
• azure-batch pinned to ==14.2.0 — the 15.x track-2 rewrite restructures the models this code uses; migration tracked separately.
• Shared-dev pools are already provisioned (0 nodes, scale-to-zero) and idle-cost-free while waiting on the quota bump.
• Spec is template-conformant: README, plan, impact-analysis, user-stories, design, test-plan, rollout.

[Copilot]

Pull request overview

Adds multi-tenant shared Azure Batch GPU pool support (H100/T4 tiers) with capacity-aware pool routing and per-job user-delegation SAS-based blob I/O, plus IaC/templates and documentation/spec updates to enable shared-pool provisioning and tenant isolation.

Changes:

• Introduces shared-pools IaC (infra/shared-pools.bicep + params) and expands batchPool.bicep to support fixed/autoscale, dedicated/low-priority, and optional VNet injection.
• Updates hastelib Azure Batch runner to select pools at submit time and optionally use per-job SAS instead of pool identity for blob transfers; processors pass candidate pool lists; adds unit tests.
• Documents the feature via a full spec (spec/features/batch-compute-expansion/*), configuration/deployment docs, and changelog entries; bumps hastegeo wheel references.

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
spec/features/batch-compute-expansion/user-stories.md	Defines user stories + acceptance criteria for shared pools, routing, isolation, and rollout.
spec/features/batch-compute-expansion/test-plan.md	Test strategy + scenarios for routing/SAS and infra validation.
spec/features/batch-compute-expansion/rollout.md	Phased rollout plan with flags, monitoring, and rollback.
spec/features/batch-compute-expansion/README.md	Feature overview, motivation, constraints, and success criteria.
spec/features/batch-compute-expansion/plan.md	Execution plan with phases/milestones and ownership.
spec/features/batch-compute-expansion/networking.md	Network design for blob↔batch reach via VNet injection + storage VNet rules.
spec/features/batch-compute-expansion/impact-analysis.md	Risk, dependency, and rollback analysis for the change.
spec/features/batch-compute-expansion/design.md	Technical design for pool topology, SAS isolation model, and routing logic.
setup/README.md	Updates default/example resource prefix guidance.
infra/shared-pools.json	Generated ARM template for shared-pools deployment.
infra/shared-pools.bicepparam	Parameter file reading env vars for shared-pools deployment.
infra/shared-pools.bicep	New standalone deployment creating shared H100+T4 pools via module calls.
infra/modules/storage.bicep	Adds optional shared batch-subnet VNet rule for tenant storage allowlisting.
infra/modules/functions.bicep	Adds app settings for pool candidate lists and SAS/manage-pools flags.
infra/modules/functionApp.bicep	Grants Function App identity Storage Blob Delegator role assignment.
infra/modules/batchPool.bicep	Parameterizes pool scale mode/node type/min nodes and optional subnet injection.
infra/main.bicepparam	Changes default prefix and wires new batch routing/SAS params + shared subnet id.
infra/main.bicep	Wires shared subnet allowlisting into storage and routes new batch settings into functions module.
hastelib/tests/core/runners/test_azure_batch_routing.py	Adds unit tests for select_pool and SAS-vs-identity behavior.
hastelib/tests/core/runners/init.py	Adds package marker for runners tests.
hastelib/src/hastegeo/core/runners/local.py	Extends LocalRunner ctor signature for interface parity (candidate pools).
hastelib/src/hastegeo/core/runners/azure_batch.py	Implements submit-time pool selection and per-job SAS URL augmentation + manage_pools gating.
hastelib/src/hastegeo/core/processors/train.py	Passes training candidate pool list into runner creation.
hastelib/src/hastegeo/core/processors/inference.py	Passes inference candidate pool list into runner creation.
hastelib/src/hastegeo/core/processors/imagery.py	Passes imageryprep candidate pool list into runner creation.
hastelib/src/hastegeo/core/processors/embedding.py	Passes candidate pool list into runner creation.
hastelib/src/hastegeo/core/processors/artifacts.py	Passes imageryprep candidate pool list into runner creation.
hastelib/src/hastegeo/core/config.py	Adds parsing for candidate pool-id lists and SAS/manage-pools flags.
hastelib/src/hastegeo/about.py	Bumps hastegeo version.
hastelib/pyproject.toml	Pins azure-batch==14.2.0 to avoid track-2 API breakage.
docs/deployment.md	Adds production note pointing to shared multi-tenant GPU pools docs.
docs/configuration.md	Documents shared multi-tenant GPU pools, flags, and networking model.
docker/imageryprep/requirements.txt	Updates hastegeo wheel pin version.
CHANGELOG.md	Adds unreleased entry describing shared pools, routing, SAS isolation, and IaC changes.
api/hastefuncqueues/requirements.txt	Updates hastegeo wheel pin version.
api/hastefuncapi/requirements.txt	Updates hastegeo wheel pin version.

[jQuinRivero]

lgtm