Skip to content

feat(azure): Network gateway, Virtual WAN and Bastion family (7 resource types) - #127

Merged
naxty merged 6 commits into
mainfrom
naxty/wave1-netgw
Aug 31, 2026
Merged

feat(azure): Network gateway, Virtual WAN and Bastion family (7 resource types)#127
naxty merged 6 commits into
mainfrom
naxty/wave1-netgw

Conversation

@naxty

@naxty naxty commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Adds the Microsoft.Network gateway / Virtual WAN / Bastion family — 7 resource
types, all on ARM API version 2023-05-01 via
armnetwork/v4. This is the plugin's first hybrid connectivity: before this
there was no VPN, no Virtual WAN and no Bastion, only the standalone
LocalNetworkGateway.

Base is naxty/wave1-plumbing (PR #125), which carries the seven clients these
resources bind to. No shared file was touched — pkg/client/client.go,
common.go, armid.go, go.mod, README.md, .github/** and
pkg/registry/registry.go are all untouched; the resources self-register via
init().

Resource types

Formae type ARM type Client Shape
AZURE::Network::VirtualWan Microsoft.Network/virtualWans VirtualWansClient LRO CUD, free, seconds
AZURE::Network::VirtualHub Microsoft.Network/virtualHubs VirtualHubsClient LRO CUD, billed, ~10 min each way
AZURE::Network::VpnSite Microsoft.Network/vpnSites VPNSitesClient LRO CUD, free, seconds
AZURE::Network::VpnGateway Microsoft.Network/vpnGateways VPNGatewaysClient LRO CUD, billed, ~30 min each way
AZURE::Network::BastionHost Microsoft.Network/bastionHosts BastionHostsClient LRO CUD, billed, ~10 min each way
AZURE::Network::VirtualNetworkGateway Microsoft.Network/virtualNetworkGateways VirtualNetworkGatewaysClient LRO CUD, billed, 30-45 min each way
AZURE::Network::VirtualNetworkGatewayConnection Microsoft.Network/connections VirtualNetworkGatewayConnectionsClient LRO CUD, free, minutes

Every one copies the NetworkLocalNetworkGateway template verbatim: a fooAPI
interface over the SDK client, buildPropertiesFromResult, armIDParts,
normalizeAzureLocation + canonicalizeEnum on the read path,
isDeleteSuccessError for idempotent delete, encodeLROStart /
statusLRO / statusDeleteLRO for resume tokens, and the pending-poller branch
that pins the expected ARM path while the operation is in flight. Update is a
re-PUT everywhere (UpdateTags cannot reach any of the interesting fields).

Fixture runtime — please read before adding these to the matrix

These are the slowest resources in the plugin. FORMAE_TEST_TIMEOUT defaults to
5 minutes, which is not close to enough for four of the seven.

Fixture Chain Wall clock, one CRUD lifecycle
virtual-wan rg -> WAN < 2 min, free
vpn-site rg -> WAN -> site < 2 min, free
virtual-hub rg -> WAN -> hub ~20-25 min, billed
bastion-host rg -> vnet -> AzureBastionSubnet + Standard static PIP -> host ~20-25 min, billed
vpn-gateway rg -> WAN -> hub -> gateway (+ free site for the connection) ~60 min, billed
virtual-network-gateway rg -> vnet -> GatewaySubnet + PIP -> gateway ~60-90 min, billed
virtual-network-gateway-connection the above plus LocalNetworkGateway + connection ~60-90 min, billed

Recommendation for .github/conformance-matrix.txt (orchestrator's call, I did
not edit the file):

  • Add unconditionally: virtual-wan, vpn-site — free and fast.
  • Add with a TIMEOUT override: virtual-hub, bastion-host, in the style
    the Front Door entries already use.
  • Exclude with a cost comment: vpn-gateway, virtual-network-gateway,
    virtual-network-gateway-connection. Suggested wording, matching the
    cdn-* and certificate exclusions already in the file:
# vpn-gateway, virtual-network-gateway and virtual-network-gateway-connection
# excluded: these are the slowest and most expensive resources in the plugin.
# A VirtualNetworkGateway takes 30-45 min to create and the same to delete; a
# Virtual WAN VPN gateway ~30 min each way, on top of a ~10 min hub. One CRUD
# lifecycle is 60-90 min of billed gateway time per fixture. CRUD is validated
# via test-integration (mocked) plus the marshaller assertions in each
# <resource>_integration_test.go. Run live locally with a raised timeout:
#   FORMAE_TEST_TIMEOUT=90m make conformance-test-crud TEST=virtual-network-gateway
#   FORMAE_TEST_TIMEOUT=90m make conformance-test-crud TEST=virtual-network-gateway-connection
#   FORMAE_TEST_TIMEOUT=90m make conformance-test-crud TEST=vpn-gateway

The conformance-tests job's timeout-minutes: 120 would also need raising if
virtual-network-gateway-connection were ever added.

Correctness notes worth reviewing

Native-ID pinning. The highest-consequence path here. Every write returns
expectedNativeID on the in-progress branch, built from
config.SubscriptionId + resource group + name against the real ARM provider
path, and each resource has a PendingCreateReportsRealNativeID test asserting
the pending native ID equals the ID ARM actually assigns. A mismatch on a
45-minute gateway create orphans a billed resource for the whole window.

Load-bearing subnet names. Azure only accepts GatewaySubnet for a
VirtualNetworkGateway and AzureBastionSubnet for a BastionHost, and
rejects anything else after minutes of provisioning. Both resources check the
subnet ID's last segment before the request goes out and fail with a specific
message; both are documented on the subnetId field in the schema; both
fixtures create the subnet with the exact required name. lastARMSegment is a
small shared helper defined in bastionhost.go.

Service-populated values dropped on read. Nested hasProviderDefault does
not work, so anything Azure assigns inside a nested object is dropped rather
than compared:

  • bgpSettings.bgpPeeringAddress / .bgpPeeringAddresses on both gateways —
    Azure allocates these out of the hub prefix / GatewaySubnet.
  • sku.capacity on VirtualNetworkGateway.
  • ipConfigurations[].privateIPAddress, and the per-child id / etag /
    type / provisioningState on every nested list (vpnSiteLinks,
    connections, ipConfigurations, vpnClientRootCertificates).
  • VpnGateway.connections[].vpnLinkConnections (Azure seeds one per remote
    link), connectionStatus, byte counters.
  • VirtualWan's virtualHubs / vpnSites back-references — the hub and site
    own their side of the reference.
  • VirtualHub's routingState, virtualRouterAsn, virtualRouterIPs and the
    azureFirewall / vpnGateway / p2sVpnGateway / bgpConnections pointers.
  • BastionHost.dnsName.
  • On VirtualNetworkGatewayConnection, ARM types virtualNetworkGateway1,
    virtualNetworkGateway2 and localNetworkGateway2 as whole gateway
    objects
    , not SubResources, and inflates them on read. The plugin sends only
    their id and reads back only their id.

Each is covered by a Read_drops_service_state subtest.

Secrets. VirtualNetworkGatewayConnection.sharedKey and
.authorizationKey, VpnGateway.connections[].sharedKey and
VirtualNetworkGateway.vpnClientConfiguration.radiusServerSecret are all
writeOnly = true and typed (formae.Value|String). ARM does return the
connection's shared key on Get; the read path never emits it, and the tests
assert the literal key text is absent from the properties JSON.

Renamed fields. VirtualWan.virtualWanTier maps to ARM's
properties.type; renamed because the flattened property map already carries
the resource's own type. Cross-resource references use the repo's existing
fooId convention (virtualWanId, virtualHubId, subnetId,
publicIpAddressId, virtualNetworkGateway1Id, ...), matching
firewallPolicyId / publicIPAddressId on ApplicationGateway.

createOnly calls. gatewayType, vpnType, vpnGatewayGeneration,
connectionType, all three connection peer fields, virtualWanId,
virtualHubId, addressPrefix, VirtualHub.sku, BastionHost.sku,
VpnGateway.bgpSettings (ASN is immutable once set),
VpnGateway.isRoutingPreferenceInternet, and both gateways'
ipConfigurations (ARM cannot swap a front end in place). SKU on
VirtualNetworkGateway is not createOnly — resize within a family is a real
in-place update, exercised by the unit test.

Deliberately left out

  • VpnGateway.natRules and VirtualHub route tables / hub connections.
    Separate ARM child types with their own clients; out of scope for this batch.
  • vpnClientConfiguration in the virtual-network-gateway fixture.
    Point-to-site needs real root-certificate data or a RADIUS secret
    (write-only), neither of which the conformance harness round-trips. The full
    marshalling path is asserted in TestVirtualNetworkGateway_CRUD.
  • connections[].sharedKey in the vpn-gateway fixture. Write-only nested
    inside an array — the harness cannot strip those, the same limitation the
    application-gateway fixture documents for sslCertificates[].data. Azure
    generates a key when it is omitted. The top-level
    VirtualNetworkGatewayConnection.sharedKey is exercised, since top-level
    write-only fields are handled (as acr-webhook shows).
  • VirtualWan.allowVnetToVnetTraffic and
    VirtualHub.allowBranchToBranchTraffic in the fixtures.
    Both are in the
    schema, but Virtual WAN does not honour them in the way ARM echoes them back,
    so declaring them would compare against a value the service ignores.
  • ExpressRoute circuits. connectionType = "ExpressRoute" and peerId are
    modelled and validated, but no ExpressRouteCircuit resource type exists yet
    and creating one requires a provider order, so there is no fixture.
  • Developer-SKU Bastion (networkAcls, virtualNetwork) — not in the
    armnetwork/v4 SKU enum.

Verification

Run in the worktree on the final commit:

$ gofmt -l pkg schema
(no output)

$ go build ./...
Go build: Success

$ go test ./pkg/...
Go test: 1161 passed in 6 packages

$ make lint
golangci-lint run
0 issues.

$ make verify-schema
Status: PASSED
- Total modules: 156
- Total resource types: 155
- Duplicate files found: 0
- Duplicate types found: 0
VERIFICATION PASSED - Schema is valid

155 resource types = the 148 on the base branch plus these 7. All 14 fixture
files also evaluate cleanly under pkl eval --project-dir testdata.

Conformance was not run: no command in this branch touched real Azure.

naxty added 3 commits August 28, 2026 16:43
Adds the Virtual WAN container plus its two directly-attached children on
Microsoft.Network API version 2023-05-01:

- AZURE::Network::VirtualWan  - Microsoft.Network/virtualWans
- AZURE::Network::VirtualHub  - Microsoft.Network/virtualHubs
- AZURE::Network::VpnSite     - Microsoft.Network/vpnSites

All three are LRO on create, update and delete and follow the existing
LocalNetworkGateway shape: re-PUT on update, idempotent delete, resume-token
Status, and the pending-poller branch pinning the ARM path so an in-flight
create cannot orphan the resource.

VirtualHub and VpnSite take the WAN by resolvable ARM ID, so a single forma
expresses the 1 -> 2 and 1 -> 4 orderings without an explicit dependency.

Conformance fixtures: virtual-wan, virtual-hub, vpn-site (each with -update).
- AZURE::Network::VpnGateway  - Microsoft.Network/vpnGateways  (2023-05-01)
- AZURE::Network::BastionHost - Microsoft.Network/bastionHosts (2023-05-01)

Both are LRO on create, update and delete and follow the existing
LocalNetworkGateway shape, including the pending-poller branch that pins the ARM
path while the operation runs - the gateway takes ~30 minutes to create and the
Bastion host ~10, so a mis-pinned native ID would orphan a billed resource.

VpnGateway is the Virtual-WAN-scoped gateway: it takes its hub by resolvable ARM
ID, models bgpSettings as ASN plus peer weight only (Azure allocates the peering
addresses out of the hub prefix and they are dropped on read), and models
connections with the caller-owned fields only - Azure seeds one vpnLinkConnections
entry per remote link and reports status and byte counters, all discarded.
sharedKey is write-only and never surfaced.

BastionHost validates up front that its subnet is named exactly
AzureBastionSubnet, turning a ten-minute ARM rejection into an immediate error.

Conformance fixtures: vpn-gateway (rg -> WAN -> hub -> gateway, plus a free VPN
site for the connection) and bastion-host (rg -> vnet -> AzureBastionSubnet +
Standard static public IP -> host), each with -update.
…Connection

- AZURE::Network::VirtualNetworkGateway           - Microsoft.Network/virtualNetworkGateways (2023-05-01)
- AZURE::Network::VirtualNetworkGatewayConnection - Microsoft.Network/connections            (2023-05-01)

The classic vnet-scoped gateway family, completing hybrid connectivity alongside
the existing LocalNetworkGateway. Both are LRO on create, update and delete and
copy the LocalNetworkGateway shape verbatim, including the pending-poller branch
that pins the ARM path while the operation runs - a gateway create takes 30-45
minutes, so a mis-pinned native ID would orphan a billed resource for that whole
window.

VirtualNetworkGateway validates up front that its subnet is named exactly
GatewaySubnet and that activeActive comes with two ipConfigurations. bgpSettings
models ASN plus peer weight only: Azure allocates the peering addresses out of
the GatewaySubnet and they are dropped on read, as is sku.capacity.
vpnClientConfiguration covers the point-to-site surface, with radiusServerSecret
write-only.

VirtualNetworkGatewayConnection sends the two gateway peers as ARM models
carrying only their id (ARM types them as whole gateway objects) and reads back
only the id, so the inflated bodies ARM returns never reach state. sharedKey and
authorizationKey are write-only - ARM does return the shared key on Get and the
plugin never surfaces it. connectionType is checked against the peer field it
requires, and ipsecPolicies is capped at Azure's one-entry limit.

Conformance fixtures: virtual-network-gateway (rg -> vnet -> GatewaySubnet +
public IP -> gateway) and virtual-network-gateway-connection (the full
site-to-site chain, gateway + LocalNetworkGateway + connection with a custom
IPsec/IKE policy), each with -update.
@naxty
naxty force-pushed the naxty/wave1-plumbing branch from 9d658c6 to 61383de Compare August 28, 2026 14:44
@naxty
naxty force-pushed the naxty/wave1-netgw branch from 8802dd3 to 00002b9 Compare August 28, 2026 14:44
@naxty
naxty changed the base branch from naxty/wave1-plumbing to main August 28, 2026 15:29
@naxty naxty closed this Aug 28, 2026
@naxty naxty reopened this Aug 28, 2026
naxty added 3 commits August 31, 2026 10:15
…tures

The bastion-host conformance fixture omitted
ipConfigurations[0].privateIpAllocationMethod. Azure populates it
unconditionally, and hasProviderDefault is only honoured on top-level fields, so
the value ARM returned read as drift in every comparison phase:

  Property ipConfigurations[0].privateIpAllocationMethod is not expected and not
  a provider default (after create / extract / sync / update)

The host itself provisioned fine - this was purely a fixture omission, not a
provisioning or marshalling bug.

Declaring the value is the established fix rather than dropping the field:
dns-resolver-inbound-endpoint models the same nested field, spells it out in its
fixture, and passes conformance today. Keeping it in the schema also leaves it
settable, which Static allocation needs.
The virtual-hub conformance fixture failed on Destroy. Every earlier phase
passed, so the bug was in the delete path. The provider error never reached the
workflow log; the Azure activity log for the failing run has it:

  {"status":"Failed","error":{"code":"ResourceOperationFailure","message":"The
  resource operation completed with terminal provisioning state 'Failed'.",
  "details":[{"code":"InvalidOperation","message":"The specified operation
  'DeleteVirtualHub' is not supported. Deletion is not supported when
  RoutingStatus on Hub is 'Provisioning'. Retry when state is not
  Provisioning."}]}}

A virtual hub has two independent states. `provisioningState` covers the ARM
resource and is what the create/update LRO reports on; `routingState` covers the
hub router behind it, and the service keeps programming that for another ten to
twelve minutes after the LRO says Succeeded. ARM refuses DeleteVirtualHub for
the whole of that window, and the fixture's destroy landed squarely inside it.
Measured on a bare hub in eastus with no gateway attached: create accepted at
T+0, provisioningState Succeeded at T+5m15s, routingState Provisioned only at
T+16m45s, and a delete issued at that point completed in 14m46s.

Delete now reads the hub first and parks the operation when it cannot succeed:
routingState Provisioning, provisioningState Deleting (a delete is already in
flight, this plugin's or the resource group's), or provisioningState Updating (a
write would collide). A parked delete carries the virtual-hub-only
lroOpVirtualHubAwaitRouting operation type and no resume token, and Status
re-evaluates it on the same rules until ARM will take the DELETE, at which point
it hands back a normal lroOpDelete request ID and the operation joins the usual
poller path. Delete and the parked branch share one function, so the two cannot
drift apart. Delete failures now also carry StatusMessage, so the next provider
error of this class shows up without a trip to the activity log.

Local conformance run against subscription ddfe7ebf, one clean run, all phases:

  Resource                    Create Verify Extract Sync Update Replace Destroy OOB Del Duration
  WAVE1-NETGW::virtual-hub    [+]    [+]    [+]     [+]  [+]    [~]     [+]     [+]     68m 50s
  1 passed, 0 failed, 0 skipped (68m 50s)

The Destroy step took 27m10s of the 30 minute FORMAE_TEST_TIMEOUT the
virtual-hub arm of scripts/ci/run-conformance-phase.sh grants it: roughly 11
minutes parked waiting for the router, then 15 minutes of delete. That is a pass
with little headroom, and raising that arm to 45 is worth doing separately.
Splits virtual-hub out of the shared virtual-hub|bastion-host arm and raises its
per-command timeout from 30 to 45 minutes.

virtual-hub's Destroy measured 27m10s against the 30 minute budget - three
minutes of headroom, which will flake. The wait is irreducible: ARM refuses
DeleteVirtualHub while routingState is still Provisioning, which runs ~11 min
past the point the create LRO reports Succeeded, and the delete itself takes
~15 min. The plugin now parks the delete until ARM will accept it rather than
failing, so the time is spent waiting instead of erroring.

The go-test cap stays at 90: the full lifecycle measured 69 min.

bastion-host keeps 30/90 - its lifecycle measured ~57 min with no single phase
close to the limit.
@naxty
naxty merged commit 97af55b into main Aug 31, 2026
12 of 13 checks passed
naxty added a commit that referenced this pull request Aug 31, 2026
The 25 resource types from PRs #126, #127 and #128 landed on main with no
ongoing test coverage and no documentation. Their pull-request conformance runs
proved them once, but nothing guarded them afterwards: none of their fixtures
were in the curated matrix, so neither main nor nightly exercised any of them.

Matrix: adds the 15 fixtures that passed CRUD *and* discovery on their own PR,
which is the bar this file asks for - static-site, virtual-wan, vpn-site,
bastion-host and all eleven cosmos-*. 113 active entries becomes 128.

Ten fixtures stay out, each with its reason recorded inline:

- app-service-plan, web-app, function-app, web-app-slot - the subscription has a
  dedicated-VM quota of 0 and an App Service plan of any sku runs on dedicated
  VMs. Verified against both B1 and F1, refused identically.
- app-service-certificate, custom-hostname-binding - need a real verified custom
  domain.
- virtual-hub - passes locally end to end (68m50s) but its CRUD phase measures
  ~59 min, longer than the OIDC client assertion stays valid, so the run dies on
  AADSTS700024 in the final phase. A credential-lifetime limit, not a plugin bug.
- vpn-gateway, virtual-network-gateway, virtual-network-gateway-connection -
  60-90 min lifecycles, past the job's timeout-minutes: 120.

virtual-hub also joins conformance-pr-skip.txt. It was excluded from the matrix
but not from PR scope, so a later PR touching its fixture would have spent an
hour to fail on the same expired assertion.

README: the resource table was 148 rows against 173 registered types. Adds the 25
missing rows, re-sorted; the table and the schemas now agree exactly.
@naxty
naxty deleted the naxty/wave1-netgw branch September 4, 2026 06:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant