fix(upgrade): keep the cluster serviceable across the Antelope -> Caracal rolling window

[Eandalf-Bigstack]

What type of PR is this?

• bug

What this PR does / why we need it

Everything here came out of the first real v3.1.10 -> v3.1.20 rolling upgrade (OpenStack Antelope -> Caracal), run end to end on cube4510. The roll wedged three separate times, each on a different service, and each time on the same underlying shape: a rolling upgrade runs a mixed-version cluster for its whole duration, and it drains every node by live migration. Anything that breaks port binding, volume attach, or the subnet API during that window does not degrade the roll — it stops it, because the drain cannot move a single VM.

• The subnet API stays up across the mixed window. Caracal's 2023.2/expand/93f394357a27_remove_in_use_on_subnets.py drops subnets.in_use, and upstream put it in the expand branch with an expand_drop_exceptions() opt-out, so there is no phased form that leaves the column standing. The moment the first node migrates the shared schema, every still-Antelope neutron-server answers 500 to any subnet query — (1054, "Unknown column 'subnets.in_use' in 'SELECT'") — because Antelope's models_v2.HasInUse declares it as a real column. That is 2 of 3 servers behind the VIP, and with them the live migration the drain depends on. Re-adding it as a generated column answers both dialects; Caracal already stopped reading it, so the value only has to read false. migrate_neutron_db_post() drops it again once os_neutron_version_uniform reports no Antelope server left
• The OVN northbound is no longer lost on the master's reboot. /etc/ovn was not in CONFIG_MIGRATE, so an upgraded node boots with an empty northbound. Survivable in isolation — a backup syncs from the promoted master — except rolling_upgrade rolls the master first, so the node holding the only live copy is the one that reboots into the empty one. Pacemaker re-promotes it, the backups sync the empty DB from it, and the last good copy is gone cluster-wide. Every port bind then fails with RowNotFound: Cannot find Logical_Switch with name=neutron-<network-id>
• The OVN sync runs after pacemaker promotes ovndb, and only claims success when it succeeded. It was called from config_neutron's Commit() at bootstrap stage 14; ovndb_servers is promoted at stage 19. Measured on cube4510: the sync ran at 19:08:58 and the northbound ovsdb-server did not start until 19:17 — it synced against a database that was not listening and silently did nothing, then marked itself done. The marker lives under /etc/appliance/state, which is CONFIG_MIGRATE'd, so that one early failure rode onto the next partition and no later boot ever retried. Moved to CommitLast(), bounded at every level, and the marker is now only written on success
• The Dell EMC SC Series backend still loads on Caracal. Caracal ships SCFCDriver with SUPPORTED = False, so cinder-volume refuses to initialize it without enable_unsupported_driver. Antelope's copy has no SUPPORTED attribute at all, so this only bites on the hop. Two halves: the built-in model carries the opt-in, and migrate_cinder_ext_storage_unsupported() rewrites the backends an upgraded cluster brought forward (see the note below — we are deliberately shipping a driver upstream labels unsupported)
• Keycloak stops discarding its own build option on every pod start. cube-cos-login.spec builds the image with kc.sh build --transaction-xa-enabled=false because MariaDB refuses XA whenever wsrep is on. That build is correct and the shipped image really does bake it — but Keycloak compares baked build config against env plus defaults, and this was the one baked option not mirrored in the environment. So it saw baked=false vs default=xa, decided the optimized image was stale, and re-augmented back to the default on every start
• No index is ever named after an unresolved field reference. Logstash renders a missing field as the literal %{name}, and es-index feeds straight into logs-%{es-index}, so an event reaching the end of the filter without [program] is indexed into a real index called logs-<date>-%{program}, braces and all

Which issue(s) this PR fixes

• Refs [Task] Language - Python upgrade from 3.10 to 3.11 #625
• Refs [Epic] Upgrade OpenStack to Caracal (2024.1) in v3.1.20 #594

Special notes for your reviewer

We are knowingly shipping a driver upstream labels unsupported. Caracal marks SCFCDriver with SUPPORTED = False — upstream sets that flag on drivers whose third-party CI has stopped reporting, and 12 drivers carry it in Caracal including three other Dell EMC ones. The SC Series is a supported external storage model for us, so we accept the CI non-compliance and ship the opt-in with the model rather than asking operators to add it themselves and discover the requirement through a failed attach. Of the five external models we ship, only this one is affected — PowerStore, both Fujitsu drivers and NFS are still supported upstream. The practical consequence to be aware of: upstream is no longer gating changes to this driver on hardware CI, so regressions in it will reach us unannounced.

Why the fix is in two commits and not one. A built-in model only reaches backends created or re-applied after the upgrade. /etc/cinder/backends is CONFIG_MIGRATE'd, so an upgraded cluster carries its pre-Caracal backend config forward verbatim and the model alone would leave the driver dead on exactly the clusters that matter. migrate_cinder_ext_storage_unsupported() is placed in SetStorageBackend() after CINDER_BACKEND_DIR is settled and immediately before that function wipes cinder.d/ext_storage_*.conf and re-copies from it — which is what makes one commit both durable and effective now: backends/ is the source of truth so the rewrite survives, and the copy carries the opt-in into the running config on this commit rather than the next.

Why subnets.in_use is re-added rather than the migration deferred. Deferring only inverts which servers are broken, and shrinks the healthy pool as the roll proceeds instead of growing it. The accepted trade with the generated column is that Antelope reads the flag as "not in use", so read/write_lock_register stop guarding concurrent subnet deletes for the length of the window — the same trade the Yoga shim made for port forwardings. That Yoga shim is retired in the same commit: the supported path is stepwise, 3.1.0 (Yoga) -> 3.1.10 (Antelope) -> 3.1.20 (Caracal) with no jumping, so a Caracal build can never meet a Yoga server. Exactly one shim is carried at a time, and the comment now says so, so the next hop swaps rather than accumulates.

The /etc/ovn migrate and the OVN sync are belt and braces, deliberately. The northbound is derived state and neutron's MySQL is the source of truth, so the sync alone is defensible — but with only the sync, one failed sync leaves no OVN data anywhere. The migrate removes the single point of total loss. It is safe across an OVN version bump: ovn-ctl's upgrade_db converts in place (NB 7.0.0 -> 7.3.0 and SB 20.27.0 -> 20.33.0 verified byte-identical in ovn-nbctl/ovn-sbctl show), and a failed convert creates an empty database — degrading to exactly the behaviour without the line. The /var/lib/ovn entry it replaces migrated nothing live: neither ovn23.03 nor ovn24.03 owns anything there, 3.1.10 nodes have no such directory, and rpm -qf reports the files as owned by no package.

The logstash guard fixes nothing that is broken today, and that is the point. a56f9a24 already fixed the cause by defaulting [program], and it holds — but only on a build that carries it. This is a guard against the class: any future field that goes missing upstream of the index name would mint a new literal-braced index every day, silently and unbounded, findable only by someone who thinks to search for braces. It is explicitly not a substitute for the fallbacks.

Additional documentation

Verified on cube4510 (cube451/cube452/cube453, control-converged, HA) across a full v3.1.10 -> v3.1.20 rolling upgrade, unless stated otherwise.

For the requirement "the subnet API stays up while the cluster is mixed-version",

[cube451 ~]# # schema migrated, no shim -- per-node neutron subnet list
cube451 (caracal)   200
cube452 (antelope)  500   (1054, "Unknown column 'subnets.in_use' in 'SELECT'")
cube453 (antelope)  500
[cube451 ~]# # drain of cube452 under that condition
VMs evacuated: 0 of 13

With the generated column in place all three nodes and the VIP return 200 and the drain runs. migrate_neutron_db_post() then drops the column once no Antelope server is left — confirmed by os_neutron_version_uniform reporting uniform 24.x after the third node finished.

For the requirement "an upgraded node does not lose the OVN northbound",

[cube451 ~]# ovn-nbctl --db=... list Logical_Switch | grep -c ^_uuid
0
[cube451 ~]# openstack network list -f value | wc -l
17
[cube451 ~]# # every port bind, cluster-wide:
RowNotFound: Cannot find Logical_Switch with name=neutron-<network-id>

That is the state the old ordering produced. Running the fixed migrate_neutron_ovn_sync rebuilt all 17 logical switches and set the marker; cube452 then picked them up from the promoted master by ovsdb replication, and the drain that had moved 0 of 13 VMs completed. The ordering itself is the control: the sync ran at 19:08:58 and the northbound ovsdb-server did not start until 19:17.

For the requirement "the sync cannot wedge a roll and cannot skip itself forever",

[cube451 ~]# # bounds now in place
northbound probe   24 * (5s connect + 5s sleep)  = 4 min
neutron-ovn-db-sync-util                timeout 600
outer HexUtilSystemF                           900

HexUtilSystemF only arms alarm() when its timeout argument is non-zero, so the old call — passed 0 — had no bound at all while running inside a hex_config commit, where blocking blocks the node's whole bootstrap and its slot in the roll. Every failure path now logs and returns without the marker, so the next boot retries.

For the requirement "Compellent-backed instances live migrate on Caracal",

[cube451 ~]# # before
cinder-volume: "Unsupported drivers are disabled"
dell-emc-sc-fc@dell-emc-sc-fc   down
attach: Driver initialize connection failed
        (error: 'SCFCDriver' object has no attribute '_client')
[cube451 ~]# # after
cinder-volume: "Driver post RPC initialization completed successfully"
dell-emc-sc-fc@dell-emc-sc-fc   enabled  up

Note the failure is indirect: the backend never completes do_setup, so the first symptom is not a driver error but every attach failing, with the service reporting itself down while systemd still shows it active.

For the requirement "an upgraded cluster's carried-forward backends get the opt-in",

[cube451 ~]# # against the real backend config off cube4510
key lands immediately after volume_driver, inside [dell-emc-sc-fc]   yes
second pass                                                          no-op
ceph backend                                                         skipped

The key is inserted after volume_driver rather than appended: these files hold one section each today, but appending would land outside the section the day one does not. iSCSI is matched too — same upstream flag, same failure — though only the FC model ships built in. Applied by hand to all three nodes mid-roll, after which Compellent-backed instances live migrate again.

For the requirement "keycloak keeps the option its image was built with",

[cube451 ~]# # before, on every pod start
Changes detected in configuration. Updating the server image.
  - transaction-xa-enabled=false > transaction-xa-enabled=xa
[cube451 ~]# # symptoms it produced
keycloak admin API           401 to a valid admin token (no realm_access claim)
cube-cos-api (cube453)       crash-loop, "failed to init oidc auth in keycloak(401)"
cluster check                ApiService NG  [ api(1 control api down) ]
[cube451 ~]# # after
override line                gone
XA/Galera errors across all three pods   0
GET /auth/admin/realms       200
hex_sdk health_api_check     rc=0

This is why it looked like a regression while the source was untouched: 06893e78 in cube-cos-ui is still on the default branch and the deployed image really was built with the flag. Nothing reverted — the option was being overridden on every single pod start.

For the requirement "no index is named after an unresolved field reference",

[cube451 ~]# logstash --config.test_and_exit -f <rendered config>
Configuration OK
[cube451 ~]# # the substitution
"20260905-%{program}"  ->  "20260905-unknown"
well-formed names                     untouched

On the historical damage: cube4510 ran 3.1.10, which predates a56f9a24, and accumulated seven literal-braced indices totalling 2.3GB — 3.5M documents on 09-05 alone. They stopped the moment it upgraded to 3.1.20: over a 70s window the malformed index gained 0 documents while logs-20260905-keystone_access, the very source that had been feeding it, gained 370. All seven were reindexed into correctly named indices (0 failures, sampled IDs confirmed present in the correct targets with correct routing) and removed.

Worth recording, because it is the reason no repair tooling is shipped here: the malformed names are cosmetic, not a health risk. Tested directly rather than assumed —

[cube453 ~]# # a braced-name index, 2 shards 1 replica
test-%{foo}  green   all 4 shards STARTED
[cube453 ~]# # forced peer relocation
_cluster/reroute move shard 0 cube453 -> cube451   completed, still green

The one shard that did wedge during the roll (logs-20260831-%{program}, REALLOCATED_REPLICA stuck at 0%) was a rolling-reboot recovery casualty that happened to land on that index, not a consequence of its name. So an upgraded cluster does not need a cleanup pipeline to stay healthy, and none is added.

End state after the full roll:

[cube451 ~]# hex_cli -v -c cluster check
27 services ok, 0 NG
[cube451 ~]# # all three nodes
CUBE_3.1.20_20260904-0153_51fd955
[cube451 ~]# # VMs, post-rebalance
cube451 7    cube452 7    cube453 6

[SekiXu]

The shim keeps subnet reads working across the mixed window, but I think it breaks subnet
creation on the still-Antelope servers.

Antelope declares the column with a python-side default (neutron/db/models_v2.py:54,
unmaintained/2023.1):

in_use = sa.Column(sa.Boolean(), nullable=False,
                   server_default=sql.false(), default=False)

default=False means SQLAlchemy names in_use in the INSERT for every new subnet, and MariaDB
refuses a value supplied for a generated column. Measured on 10.11.19 against the schema this
ALTER produces:

• Caracal-style INSERT (does not name in_use) -> OK, reads back 0
• full-model SELECT, FOR UPDATE, LOCK IN SHARE MODE -> all OK, so the read half does work,
  and you are right that neither lock register writes the column
• Antelope-style INSERT naming in_use = 0:
  • sql_mode=TRADITIONAL -> ERROR 1906 (HY000)
  • sql_mode='' -> warning only, insert succeeds

oslo.db's mysql_sql_mode defaults to TRADITIONAL and nothing in the tree overrides it (only
the commented #mysql_sql_mode = TRADITIONAL in the *.conf.sample files), so the strict path is
the one that applies.

The good news is that this one does not need a generated column. The portforwardings shim did,
because the values had to be derived from the new columns. Here the value only has to read
false, and Caracal's HasInUse in 2024.1 has no in_use attribute at all, so a plain column
answers both dialects:

ALTER TABLE subnets ADD COLUMN in_use tinyint(1) NOT NULL DEFAULT 0

Antelope's INSERT names it and is accepted, Caracal's does not and takes the default, and
migrate_neutron_db_post() drops it unchanged.

The rest of this one looks right to me. CONFIG_REQUIRES(neutron_last, pacemaker_last) really
does order CommitLast after the ovndb promotion, $HEX_SDK shared_id resolves
(core/main/proj_functions:170, sourced unconditionally by the dispatcher), and writing the OVN
sync marker only on success is the fix that stops one early failure riding onto the next
partition.

[Eandalf-Bigstack]

@SekiXu thank you — this was a real catch, and a precise one. You were right that the generated column only fixes half the problem, and right about the mechanism: the default=False on Antelope's HasInUse is a python-side default, so SQLAlchemy names in_use in the INSERT for every subnet, and MariaDB refuses a supplied value for a generated column. Fixed in 3824776 — the column is now a plain tinyint(1) NOT NULL DEFAULT 0, exactly as you suggested.

I reproduced all of it on jim-1cc (MariaDB 10.11.18) before changing anything, first against a CREATE TABLE LIKE copy of the live neutron.subnets and then against the real table.

[cc1 ~]# # what each release's ORM actually emits (SQLAlchemy 1.4.51, the shipped one)
antelope  INSERT INTO subnets (id, network_id, in_use) VALUES ('s1', 'n1', false)
caracal   INSERT INTO subnets (id, network_id)         VALUES ('s1', 'n1')
[cc1 ~]# # generated column, Antelope-style INSERT
sql_mode=TRADITIONAL          ERROR 1906   row not inserted
sql_mode=STRICT_TRANS_TABLES  ERROR 1906   row not inserted
sql_mode=''                   Warning 1906 ... has been ignored   (row inserted)
[cc1 ~]# # plain column, both dialects, every sql_mode
antelope INSERT   OK   in_use reads back 0
caracal  INSERT   OK   in_use reads back 0
[cc1 ~]# # the read half was never the problem -- both shapes pass
full-model SELECT / FOR UPDATE / LOCK IN SHARE MODE   ok
[cc1 ~]# # information_schema: the plain ALTER vs Antelope's own BOOL NOT NULL DEFAULT false
in_use  tinyint(1)  NO  0          <- identical

Three things worth adding to what you found:

It does not depend on the oslo.db setting at all. You are right that nothing in the tree overrides mysql_sql_mode, so TRADITIONAL applies — but this cluster's own @@global.sql_mode is already STRICT_TRANS_TABLES, and ERROR 1906 fires under that too. So the strict path applies before oslo.db sets anything on the session, and the only configuration that would have hidden this is sql_mode='', where it degrades to the warning above and the row inserts. That is presumably why a loose-mode scratch test would have looked fine.

The column shape is byte-identical to the original. ussuri/expand/d8bdf05313f4_add_in_use_to_subnet.py created it as sa.Column('in_use', sa.Boolean(), server_default=sa.sql.false(), nullable=False), which MariaDB materialises as tinyint(1) NOT NULL DEFAULT 0 — the same row in information_schema as your ALTER produces. So the shim now restores the exact schema Antelope's ORM was written against, rather than an approximation of it.

Your remark that neither lock register writes the column also retires the trade-off I claimed. I had written that the shim costs Antelope's concurrent-subnet-delete guarding; checking both releases, that was simply wrong. In 22.2.1 and 24.2.2 alike, read_lock_register and write_lock_register only take a row lock (SELECT … FOR UPDATE / LOCK IN SHARE MODE) and never read or write the flag, so the guarding keeps working unchanged on both sides for the whole window. Worth noting for the next hop that 22.0.x did UPDATE … SET in_use=True, which a generated column rejects with the same ERROR 1906 — 3.1.10 ships 22.2.1, past that change, so we are clear, but the write path is not hypothetical in general. The "Why subnets.in_use is re-added rather than the migration deferred" paragraph in the description still describes the generated column and that non-existent trade — treat this comment as superseding it.

The general rule I have taken from this, and recorded in the handbook: a generated column can only shim a dropped column if the old ORM never names it in a write, so the thing to check is a python-side default= on the old model, not just explicit assignments.

Thanks also for verifying the other half — CONFIG_REQUIRES(neutron_last, pacemaker_last), $HEX_SDK shared_id, and the success-only OVN sync marker. That was the part I was least able to prove from the cube4510 run alone.

[SekiXu]

3824776a is exactly right. I checked the reasoning against the upstream tags rather than
taking it on trust:

• 22.2.1 models_v2.py:55 — in_use = sa.Column(sa.Boolean(), nullable=False, server_default=sql.false(), default=False). The python-side default=False is there.
• 24.2.2 models_v2.py:33-52 — HasInUse's body is the two classmethods and nothing
  else, no sa.Column at all, so the new side never names it.
• Both tags' lock registers are .enable_eagerloads(False).with_for_update() /
  .with_for_update(read=True) and nothing more.
• HasInUse is mixed into exactly one model in both tags (models_v2.py:202 / :196,
  Subnet), and 93f394357a27's upgrade() is the single op.drop_column, so the blast
  radius is one table.

Approving. One thing left, and it is the description rather than the code — the comment at
sdk_migrate.sh:283 reads correctly now and the handbook entry is clean, so the body is the
only surface still describing the generated column: the summary bullet, the "Why
subnets.in_use is re-added" paragraph with the locking trade-off you already retired, and
the evidence section ("With the generated column in place all three nodes and the VIP return
200"). Your comment supersedes it for anyone reading the thread, but the AC is written in the
body, so I would rather it not need the thread.

No re-run needed for that last one: on the read path the plain column is a strict superset of
the generated one, and that is all the 200/drain result rests on, so the evidence transfers
and only the wording has to stop naming the shape.

One note for whoever re-tests. migrate_neutron_db() returns early on
$STATE_DIR/neutron_db_migrated, and the guard at sdk_migrate.sh:297 counts the column
without checking its shape — so a node that already ran the old shim keeps the generated
column under the fixed build. cube4510 and jim-1cc are both in that state: drop the column
and remove the state file first, or the re-test silently exercises the old shape. Not asking
for a code change to defend an unreleased intermediate commit.