Skip to content

[action] [PR:27904] Ignore recurring PCIe AER Correctable errors on NVMe boot SSD in loganalyzer - #1412

Merged
mssonicbld merged 1 commit into
Azure:202512from
mssonicbld:cherry/msft-202512/27904
Sep 14, 2026
Merged

mssonicbld merged 1 commit into
Azure:202512from
mssonicbld:cherry/msft-202512/27904

Conversation

@mssonicbld

Copy link
Copy Markdown
Collaborator

Description of PR

Summary:
Adds a loganalyzer ignore-regex entry so that recurring PCIe AER (Advanced Error Reporting) Correctable errors reported against the onboard NVMe boot SSD kernel device no longer fail loganalyzer-gated test runs.

Fixes # (N/A - triage finding, see Nokia-ION/nokia-th6#36)

Type of change

  • Testbed and Framework(new/improvement)
  • Bug fix
  • New Test case
    • Skipped for non-supported platforms
  • Test case improvement

Back port request

  • 202311
  • 202405
  • 202411
  • 202505
  • 202511
  • 202512
  • 202605

Approach

What is the motivation for this PR?

While triaging syslog on a Nokia-7220-Th6p testbed device, we found recurring kernel log lines like:

kernel: [ 8340.896498] nvme 0000:04:00.0:   device [1bc0:5236] error status/mask=00000041/00000000

These are severity=Correctable PCIe AER Physical-Layer errors (RxErr/BadTLP) on the onboard NVMe M.2 boot SSD's PCIe link. They are self-recovered by the PCIe link with zero data loss; smartctl on the affected drive shows overall health PASSED, 0 media/data integrity errors, and 0 error-log entries, both during and after the observed episode.

An existing loganalyzer_common_ignore.txt rule already ignores the equivalent message when the kernel component name is pcieport:

r, ".* WARNING kernel:.*pcieport.*device.*error.*status/mask=.*"

However this does not match when the reporting component is nvme (as seen on this platform), so loganalyzer-gated tests would fail/flag on this benign, hardware-corrected condition.

How did you do it?

Added a new ignore regex mirroring the existing pcieport rule, scoped to the nvme component:

r, ".* WARNING kernel:.*nvme.*device.*error.*status/mask=.*"

How did you verify/test it?

Loaded ansible/roles/test/files/tools/loganalyzer/loganalyzer.py locally and confirmed with line_matches():

  • Sample nvme ... device [...] error status/mask=00000041/00000000, =00000001/00000000, and =000000c1/00000000 lines are now correctly ignored (not flagged).
  • An unrelated genuine kernel error (kernel: Oops: general protection fault) is still correctly flagged, confirming the new rule is narrowly scoped and does not suppress real kernel errors.

Any platform specific information?

Observed on Nokia-IXR7220-H6-O256 (Th6p), affects the onboard M.2 NVMe boot SSD PCIe link (0000:04:00.0). The regex is not vendor/platform-specific — it targets the generic Linux kernel AER log format for nvme-named PCIe endpoints, so it will apply to any platform emitting this message format.

Supported testbed topology if it's a new test case?

N/A - this is a loganalyzer ignore-list update, not a new test case.

Documentation

N/A

Signed-off-by: Sonic Build Admin sonicbld@microsoft.com

…nalyzer

### Description of PR
Summary:
Adds a `loganalyzer` ignore-regex entry so that recurring PCIe AER (Advanced Error Reporting) **Correctable** errors reported against the onboard NVMe boot SSD kernel device no longer fail loganalyzer-gated test runs.

Fixes # (N/A - triage finding, see Nokia-ION/nokia-th6#36)

### Type of change
- [x] Testbed and Framework(new/improvement)
- [ ] Bug fix
- [ ] New Test case
    - [ ] Skipped for non-supported platforms
- [ ] Test case improvement

### Back port request
- [ ] 202311
- [ ] 202405
- [ ] 202411
- [ ] 202505
- [ ] 202511
- [ ] 202512
- [ ] 202605

### Approach
#### What is the motivation for this PR?
While triaging syslog on a Nokia-7220-Th6p testbed device, we found recurring kernel log lines like:
```
kernel: [ 8340.896498] nvme 0000:04:00.0:   device [1bc0:5236] error status/mask=00000041/00000000
```
These are `severity=Correctable` PCIe AER Physical-Layer errors (RxErr/BadTLP) on the onboard NVMe M.2 boot SSD's PCIe link. They are self-recovered by the PCIe link with **zero data loss**; `smartctl` on the affected drive shows overall health PASSED, 0 media/data integrity errors, and 0 error-log entries, both during and after the observed episode.

An existing `loganalyzer_common_ignore.txt` rule already ignores the equivalent message when the kernel component name is `pcieport`:
```
r, ".* WARNING kernel:.*pcieport.*device.*error.*status/mask=.*"
```
However this does not match when the reporting component is `nvme` (as seen on this platform), so loganalyzer-gated tests would fail/flag on this benign, hardware-corrected condition.

#### How did you do it?
Added a new ignore regex mirroring the existing `pcieport` rule, scoped to the `nvme` component:
```
r, ".* WARNING kernel:.*nvme.*device.*error.*status/mask=.*"
```

#### How did you verify/test it?
Loaded `ansible/roles/test/files/tools/loganalyzer/loganalyzer.py` locally and confirmed with `line_matches()`:
- Sample `nvme ... device [...] error status/mask=00000041/00000000`, `=00000001/00000000`, and `=000000c1/00000000` lines are now correctly ignored (not flagged).
- An unrelated genuine kernel error (`kernel: Oops: general protection fault`) is still correctly flagged, confirming the new rule is narrowly scoped and does not suppress real kernel errors.

#### Any platform specific information?
Observed on Nokia-IXR7220-H6-O256 (Th6p), affects the onboard M.2 NVMe boot SSD PCIe link (`0000:04:00.0`). The regex is not vendor/platform-specific — it targets the generic Linux kernel AER log format for `nvme`-named PCIe endpoints, so it will apply to any platform emitting this message format.

#### Supported testbed topology if it's a new test case?
N/A - this is a loganalyzer ignore-list update, not a new test case.

### Documentation
N/A

Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

Original PR: sonic-net/sonic-mgmt#27904

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run

1 similar comment
@mssonicbld

Copy link
Copy Markdown
Collaborator Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
1 pipeline(s) were filtered out due to trigger conditions.

@mssonicbld
mssonicbld merged commit 2f55667 into Azure:202512 Sep 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant