[action] [PR:27904] Ignore recurring PCIe AER Correctable errors on NVMe boot SSD in loganalyzer - #1412
Merged
Conversation
…nalyzer
### Description of PR
Summary:
Adds a `loganalyzer` ignore-regex entry so that recurring PCIe AER (Advanced Error Reporting) **Correctable** errors reported against the onboard NVMe boot SSD kernel device no longer fail loganalyzer-gated test runs.
Fixes # (N/A - triage finding, see Nokia-ION/nokia-th6#36)
### Type of change
- [x] Testbed and Framework(new/improvement)
- [ ] Bug fix
- [ ] New Test case
- [ ] Skipped for non-supported platforms
- [ ] Test case improvement
### Back port request
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [ ] 202605
### Approach
#### What is the motivation for this PR?
While triaging syslog on a Nokia-7220-Th6p testbed device, we found recurring kernel log lines like:
```
kernel: [ 8340.896498] nvme 0000:04:00.0: device [1bc0:5236] error status/mask=00000041/00000000
```
These are `severity=Correctable` PCIe AER Physical-Layer errors (RxErr/BadTLP) on the onboard NVMe M.2 boot SSD's PCIe link. They are self-recovered by the PCIe link with **zero data loss**; `smartctl` on the affected drive shows overall health PASSED, 0 media/data integrity errors, and 0 error-log entries, both during and after the observed episode.
An existing `loganalyzer_common_ignore.txt` rule already ignores the equivalent message when the kernel component name is `pcieport`:
```
r, ".* WARNING kernel:.*pcieport.*device.*error.*status/mask=.*"
```
However this does not match when the reporting component is `nvme` (as seen on this platform), so loganalyzer-gated tests would fail/flag on this benign, hardware-corrected condition.
#### How did you do it?
Added a new ignore regex mirroring the existing `pcieport` rule, scoped to the `nvme` component:
```
r, ".* WARNING kernel:.*nvme.*device.*error.*status/mask=.*"
```
#### How did you verify/test it?
Loaded `ansible/roles/test/files/tools/loganalyzer/loganalyzer.py` locally and confirmed with `line_matches()`:
- Sample `nvme ... device [...] error status/mask=00000041/00000000`, `=00000001/00000000`, and `=000000c1/00000000` lines are now correctly ignored (not flagged).
- An unrelated genuine kernel error (`kernel: Oops: general protection fault`) is still correctly flagged, confirming the new rule is narrowly scoped and does not suppress real kernel errors.
#### Any platform specific information?
Observed on Nokia-IXR7220-H6-O256 (Th6p), affects the onboard M.2 NVMe boot SSD PCIe link (`0000:04:00.0`). The regex is not vendor/platform-specific — it targets the generic Linux kernel AER log format for `nvme`-named PCIe endpoints, so it will apply to any platform emitting this message format.
#### Supported testbed topology if it's a new test case?
N/A - this is a loganalyzer ignore-list update, not a new test case.
### Documentation
N/A
Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
mssonicbld
requested review from
Sai Kiran (opcoder0) and
Ying Xie (yxieca)
as code owners
September 14, 2026 22:52
Collaborator
Author
|
Original PR: sonic-net/sonic-mgmt#27904 |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Collaborator
Author
|
/azp run |
1 similar comment
Collaborator
Author
|
/azp run |
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. |
12 tasks
|
Azure Pipelines: 1 pipeline(s) were filtered out due to trigger conditions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
Adds a
loganalyzerignore-regex entry so that recurring PCIe AER (Advanced Error Reporting) Correctable errors reported against the onboard NVMe boot SSD kernel device no longer fail loganalyzer-gated test runs.Fixes # (N/A - triage finding, see Nokia-ION/nokia-th6#36)
Type of change
Back port request
Approach
What is the motivation for this PR?
While triaging syslog on a Nokia-7220-Th6p testbed device, we found recurring kernel log lines like:
These are
severity=CorrectablePCIe AER Physical-Layer errors (RxErr/BadTLP) on the onboard NVMe M.2 boot SSD's PCIe link. They are self-recovered by the PCIe link with zero data loss;smartctlon the affected drive shows overall health PASSED, 0 media/data integrity errors, and 0 error-log entries, both during and after the observed episode.An existing
loganalyzer_common_ignore.txtrule already ignores the equivalent message when the kernel component name ispcieport:However this does not match when the reporting component is
nvme(as seen on this platform), so loganalyzer-gated tests would fail/flag on this benign, hardware-corrected condition.How did you do it?
Added a new ignore regex mirroring the existing
pcieportrule, scoped to thenvmecomponent:How did you verify/test it?
Loaded
ansible/roles/test/files/tools/loganalyzer/loganalyzer.pylocally and confirmed withline_matches():nvme ... device [...] error status/mask=00000041/00000000,=00000001/00000000, and=000000c1/00000000lines are now correctly ignored (not flagged).kernel: Oops: general protection fault) is still correctly flagged, confirming the new rule is narrowly scoped and does not suppress real kernel errors.Any platform specific information?
Observed on Nokia-IXR7220-H6-O256 (Th6p), affects the onboard M.2 NVMe boot SSD PCIe link (
0000:04:00.0). The regex is not vendor/platform-specific — it targets the generic Linux kernel AER log format fornvme-named PCIe endpoints, so it will apply to any platform emitting this message format.Supported testbed topology if it's a new test case?
N/A - this is a loganalyzer ignore-list update, not a new test case.
Documentation
N/A
Signed-off-by: Sonic Build Admin sonicbld@microsoft.com