Skip to content

Add fabric port isolate reason isolated link count in the STATE_DB for fabric ports - #4704

Merged
rlhui merged 3 commits into
sonic-net:masterfrom
saksarav-nokia:saksarav-nokia-isolate
Aug 9, 2026
Merged

rlhui merged 3 commits into
sonic-net:masterfrom
saksarav-nokia:saksarav-nokia-isolate

Conversation

@saksarav-nokia

@saksarav-nokia saksarav-nokia commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What I did

Added ISOLATE_REASON on each FABRIC_PORT_TABLE entry in STATE_DB so operators and CLI can see why a link is isolated (config, CRC/FEC auto-isolate, permanent, link-down counter reset, admin unisolate, etc.).
Added computeFabricIsolateReason() to derive the reason from isolation
state and error poll counters during fabric debug monitoring.
Added FABRIC_CAPACITY_DATA.isolated_links alongside existing
capacity fields for chassis-wide isolated link telemetry.
Also simplify (unnecessary) fabric counter parsing loops and fix minor log/comment
typos in FabricPortsOrch.
The corresponding sonic-utilities PR sonic-net/sonic-utilities#4641
sonic-mgmt PR : sonic-net/sonic-mgmt#25637

Why I did it
To make the debugging easier in the field when the crc/fec errors are seen and ports gets isolated
How I verified it
Induced the crc and fec errors in COUNTERS_DB and verified the fields are updated correctly in STATE_DB. Also updated the sonic-mgmt test and verified the tests are passing.
Details if related

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

@vmittal-msft @ysmanman @kenneth-arista , Please review.
This is a minor enhancement to the fabric isolation and fabric monitor feature for better debugging. With these changes, the "show fabric monitor capacity" and 'show fabric isolation' commands tell us what is exact reason for the isolated ports Instead of checking multiple cli commands.

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

/azpw run

@mssonicbld

Copy link
Copy Markdown
Collaborator

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1147768:

✅Stage Test:

  • Job vstest: retried.

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

/azpw run

@saksarav-nokia
saksarav-nokia marked this pull request as ready for review June 29, 2026 15:45
@mssonicbld

Copy link
Copy Markdown
Collaborator

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1147768:

✅Stage Test:

  • Job vstest: retried.

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

/azpw run

@mssonicbld

Copy link
Copy Markdown
Collaborator

⚠️ Notice: /azpw run only runs failed jobs now. If you want to trigger a whole pipline run, please rebase your branch or close and reopen the PR.
💡 Tip: You can also use /azpw retry to retry failed jobs directly.

Retrying failed(or canceled) jobs...

Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
@saksarav-nokia
saksarav-nokia force-pushed the saksarav-nokia-isolate branch from 64adc46 to 7412a8c Compare July 6, 2026 13:55
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@abdosi

abdosi commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@kenneth-arista and @vmittal-msft to sign-off

Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@abdosi

abdosi commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@kenneth-arista : please sign off and @vmittal-msft : also please review/sign-off.

kenneth-arista
kenneth-arista previously approved these changes Jul 22, 2026

@kenneth-arista kenneth-arista left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. @jfeng-arista also took a look

@vmittal-msft vmittal-msft left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the enhancement — the intent (operator-friendly isolation debugging) is well served and the change is nicely scoped. All CI (incl. vstest) is green. A few points below; only the first is worth resolving before merge, the rest are minor.

What looks good

  • computeFabricIsolateReason() reuses the same threshold vars (isolationPollsCfg/fecIsolatedPolls) that drive the actual isolate trigger, so the reported reason stays consistent with the isolation decision within a poll.
  • The removed inner for (cnt ...) loops in updateFabricDebugCounters()/updateFabricRate() were genuinely dead — cnt was never used and the body only matched field names. Good simplification, and it de-spams the per-field log too.
  • Race-tolerant comparators for the ephemeral link_event_counters_reset / admin_unisolate reasons are a nice touch.
  • Removing the unused DVSDatabase import and the duplicate configKey reassignment are both safe (configKey is still defined earlier in the same scope).

Please align before merge — reason vocabulary vs sonic-utilities #4641

The companion CLI renders show fabric isolation by passing the STATE_DB string straight through, but its fixtures/expected output use reason strings this PR never emits:

  • swss writes config → utilities examples show user_config
  • utilities references symbol_errors → swss has no symbol-error isolation path (the monitor only tracks CRC + FEC-uncorrectable)

So the operator-facing examples won't match what's actually written. The set swss can produce is:
none, config, permanent, crc_errors, fec_uncorrectable, "crc_errors & fec_uncorrectable", auto, unknown (+ ephemeral link_event_counters_reset, admin_unisolate). Could we make that the single source of truth and align both PRs on one vocabulary?

Minor / non-blocking

  1. & and spaces in a DB value"crc_errors & fec_uncorrectable" works, but a machine-parseable form (e.g. crc_errors,fec_uncorrectable) is more conventional for a multi-valued STATE_DB field and easier for the CLI to split/render.
  2. config vs permanent precedence — a port that is both config-isolated and permanently isolated reports config (checked first). Is that intended? permanent is arguably the more important state to surface.
  3. Redundant input — by the time the reason is computed, permIsolate is already forced to 1 whenever origPermIsolated == 1, so the permIsolate || origPermIsolated check's second operand never changes the outcome. Similarly the final unknown is unreachable (isolated == 1 always implies cfg/auto/perm). Both are harmless/defensive — just flagging they're effectively dead.
  4. Untested branchesauto (auto-isolated while current counters are below the trigger, i.e. the recovery window) and unknown aren't covered by the new tests. Worth a short comment documenting when a port shows auto vs crc_errors, since operators may be surprised to see it flip.
  5. StylecomputeFabricIsolateReason touches no member state; making it static/a free function would signal it has no side effects.
  6. Lingering reason when monitoring is offadmin_unisolate / link_event_counters_reset only refresh on the next debug-counter poll, so with monState=disable they persist. Cosmetic.

Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

@vmittal-msft , i have addressed your comments. Please take a look at it and sign off.
Also can we cherry-pick this PR to msft-2601?

@mssonicbld

Copy link
Copy Markdown
Collaborator

The label Approved for msft-202601 Branch can only be added by a member of the @sonic-net/release-manager-202601 team. Removing the label. Please contact one of the release managers: @rlhui to approve the cherry pick.

@jfeng-arista

Copy link
Copy Markdown
Contributor

Looks good to me. @jfeng-arista also took a look

thank you for tagging me. about the review.

the production code and test code look good to me as well.

@vmittal-msft vmittal-msft left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed db3ab60c ("Address the review comments") — thanks for the thorough turnaround. The one item I'd asked to resolve before merge is fully addressed, so this is good to go from my side.

Blocker resolved — reason vocabulary now aligned with sonic-utilities #4641

swss keeps config (and has no symbol-error isolation path), and the companion CLI PR was updated in lockstep: user_config and symbol_errors are gone, and both sides now share one vocabulary — none, config, permanent, crc_errors, fec_uncorrectable, crc_errors,fec_uncorrectable, auto, unknown plus the ephemeral link_event_counters_reset / admin_unisolate. The fabricstat header comment enumerates exactly this set and the mock fixtures match. 👍

Also addressed

  • Multi-value reason is now machine-parseable crc_errors,fec_uncorrectable (comma, no spaces).
  • permanent is now evaluated before config, with a dedicated test.
  • The new comment documents auto vs crc_errors and the ephemeral-persistence behavior when monState=disable; FEC-only and combined CRC+FEC cases now have tests.

Non-blocking nits (fine to leave)

  1. computeFabricIsolateReason() touches no member state — could be static/free to signal that, but harmless as-is.
  2. The final unknown branch is effectively unreachable (whenever isolated==1, one of cfg/auto/perm is set); the added SWSS_LOG_WARN is a fine defensive guard.
  3. ISOLATE_REASON isn't seeded in updateFabricPortState(), so a freshly discovered port has no reason field until the first debug-counter poll. The CLI tolerates the missing field, so it's cosmetic — defaulting to none at creation would round it out.

All 18 checks are green on this commit (incl. Test vstest / TestAsan vstest), so the new reason tests pass. LGTM.

@vmittal-msft

Copy link
Copy Markdown
Contributor

@rlhui @yxieca please help merge

@mssonicbld

Copy link
Copy Markdown
Collaborator

This PR has backport request label(s) for branch(es): msft-202601, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202601: <test result>) in the Test result section as well in your PR description.

---Powered by SONiC BuildBot

@rlhui
rlhui merged commit 8cd833e into sonic-net:master Aug 9, 2026
19 checks passed
@mssonicbld

Copy link
Copy Markdown
Collaborator

The change is not in msft-202601 yet. @saksarav-nokia, please manually create the cherry pick PR for branch msft-202601.
You can ping the release branch owner(github account: rlhui) to approve your cherry pick PR.
If this change is already in msft-202601, please comment "already in msft-202601". Thanks!

---Powered by SONiC BuildBot

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

This PR has backport request label(s) for branch(es): msft-202601, but is missing required test information. Please make sure you tick the tested branch(es) in the Tested branch section and provide test evidence (e.g., 202601: ) in the Test result section as well in your PR description.

---Powered by SONiC BuildBot

Azure/sonic-swss.msft#264

@saksarav-nokia

Copy link
Copy Markdown
Contributor Author

PR for msft-201601

Azure/sonic-swss.msft#264

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

7 participants