CI Health Report

[johnramsden]

Issue for weekly automated reporting of CI test health

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CEPH-1764.

This message was autogenerated

[github-actions]

CI Health Report

Last 30 days — generated 2026-04-22T18:18:21Z

Job Failure Rates

Workflow	Job	Runs	Failures	Rate	Trend
Tests	Test quincy upgrades	101	32	31.7%	↑ +29.8%
Main Documentation Checks	documentation-checks / Spell check / Check spelling in the documentation	52	9	17.3%	↑ +19.0%
Tests	Multi node testing	135	17	12.6%	↑ +18.3%
Tests	Run Unit tests	145	15	10.3%	↓ -12.3%
Main Documentation Checks	documentation-checks / Link check / Check links in the documentation	52	5	9.6%	↑ +6.9%
Tests	Test reef upgrades	135	12	8.9%	↑ +2.5%
Tests	Single node with encryption	135	11	8.1%	↑ +11.8%
Commits	Signed-off-by (DCO)	87	7	8.0%	↑ +3.2%
Tests	API tests	69	5	7.2%	—
Tests	Build microceph snap	145	10	6.9%	↑ +9.7%
Tests	Run static checks	98	6	6.1%	↓ -44.8%
Tests	Test MicroCeph Cephfs Remote Replication features.	135	8	5.9%	↓ -8.7%
Tests	Test MicroCeph RBD Remote Replication features.	135	8	5.9%	↓ -8.7%
Tests	DSL functional tests	77	3	3.9%	↓ -27.1%
Commits	Canonical CLA signed	87	3	3.4%	↓ -4.2%
Tests	Adopt test with cephadm	135	4	3.0%	↓ -2.6%
Tests	Test maintenance mode	135	4	3.0%	↑ +4.3%
Tests	Regression test for sequential join mon host refresh (issue	90	2	2.2%	↓ -40.0%
Tests	Test WAL/DB device usage	135	3	2.2%	↓ -3.7%
Tests	Availability zone crush rule testing	100	1	1.0%	↑ +1.1%
Tests	Test MicroCeph NFS feature (multinode)	135	1	0.7%	↓ -2.4%
CI Health Report	test	3	0	0.0%	—
Main Documentation Checks	OpenAPI Spec Check	52	0	0.0%	→ +0.0%
Main Documentation Checks	documentation-checks / Inclusive language check / Check for inclusive language in the documentation	52	0	0.0%	→ +0.0%
Sphinx python dependency build checks	build	100	0	0.0%	→ +0.0%
Tests	Multi node testing with custom microceph IP	135	0	0.0%	→ +0.0%
Tests	Verify pristineness and wiping	135	0	0.0%	→ +0.0%
Tests	Messenger v2 tests	135	0	0.0%	→ +0.0%
Tests	Test MicroCeph Cluster features.	135	0	0.0%	→ +0.0%
Tests	Test with loopback file OSDs	135	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature	135	0	0.0%	→ +0.0%
Markdown style checks	markdown-lint	43	0	0.0%	→ +0.0%
Close inactive issues	close-issues	30	0	0.0%	→ +0.0%
Upgrade a s/stable cluster to s/candidate	Test squid/stable to squid/candidate upgrades	7	0	0.0%	—
Copilot code review	Prepare	5	0	0.0%	—
Copilot code review	Agent	5	0	0.0%	—
Copilot code review	Upload results	5	0	0.0%	—
Copilot code review	Cleanup artifacts	5	0	0.0%	—
Tests	Regression test - backwardCompatPubnet (#703)	1	0	0.0%	—
CI Scripts Tests	test-scripts	1	0	0.0%	—

Trend: the 30-day window is divided into equal time buckets (daily for ≤ 14 days, weekly for ≤ 90 days, ~monthly beyond that). The failure rate in the first half of those buckets is compared to the second half: ↑ = getting worse, ↓ = improving, → = stable (< 1 pp change). — = fewer than 2 runs per bucket on average; not enough data.

Top 5 Most Failing Jobs

1. Tests / Test quincy upgrades — 32 failures (31.7%)
2. Tests / Multi node testing — 17 failures (12.6%)
3. Tests / Run Unit tests — 15 failures (10.3%)
4. Tests / Test reef upgrades — 12 failures (8.9%)
5. Tests / Single node with encryption — 11 failures (8.1%)

Summary

• Total job runs: 3437
• Total failures: 166
• Overall failure rate: 4.8%

[github-actions]

CI Health Report

Last 30 days — generated 2026-04-27T09:42:48Z

Job Failure Rates

Workflow	Job	Runs	Failures	Rate	Trend
Tests	Test quincy upgrades	77	31	40.3%	↑ +58.8%
Main Documentation Checks	documentation-checks / Spell check / Check spelling in the documentation	44	10	22.7%	↑ +14.2%
Tests	Multi node testing	123	18	14.6%	↓ -1.3%
Main Documentation Checks	documentation-checks / Link check / Check links in the documentation	44	6	13.6%	↑ +6.7%
Tests	Run Unit tests	133	14	10.5%	↓ -20.3%
Commits	Signed-off-by (DCO)	72	7	9.7%	↓ -13.4%
Tests	Single node with encryption	123	11	8.9%	↓ -7.8%
Tests	Test reef upgrades	123	10	8.1%	↓ -5.9%
Tests	Build microceph snap	133	10	7.5%	↓ -10.6%
Tests	Test MicroCeph RBD Remote Replication features.	123	9	7.3%	↓ -4.0%
Tests	Test MicroCeph Cephfs Remote Replication features.	123	8	6.5%	↓ -5.4%
Tests	API tests	81	5	6.2%	↓ -45.5%
Tests	Run static checks	108	5	4.6%	↓ -10.1%
Commits	Canonical CLA signed	72	3	4.2%	↓ -2.5%
Tests	Test maintenance mode	123	4	3.3%	↓ -7.7%
Tests	Adopt test with cephadm	123	3	2.4%	→ +0.9%
Tests	Test WAL/DB device usage	123	3	2.4%	→ +0.9%
Tests	Regression test for sequential join mon host refresh (issue	101	2	2.0%	↓ -6.7%
Tests	DSL functional tests	82	1	1.2%	↑ +1.4%
Tests	Availability zone crush rule testing	110	1	0.9%	↓ -2.6%
Tests	Test MicroCeph NFS feature (multinode)	123	1	0.8%	↓ -1.9%
Sphinx python dependency build checks	build	84	0	0.0%	→ +0.0%
Main Documentation Checks	OpenAPI Spec Check	44	0	0.0%	→ +0.0%
Main Documentation Checks	documentation-checks / Inclusive language check / Check for inclusive language in the documentation	44	0	0.0%	→ +0.0%
Markdown style checks	markdown-lint	36	0	0.0%	→ +0.0%
Tests	Multi node testing with custom microceph IP	122	0	0.0%	→ +0.0%
Tests	Test with loopback file OSDs	123	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature	123	0	0.0%	→ +0.0%
Tests	Verify pristineness and wiping	122	0	0.0%	→ +0.0%
Tests	Test MicroCeph Cluster features.	123	0	0.0%	→ +0.0%
Tests	Messenger v2 tests	123	0	0.0%	→ +0.0%
Close inactive issues	close-issues	30	0	0.0%	→ +0.0%
CI Health Report	test	4	0	0.0%	—
CI Health Report	report	1	0	0.0%	—
Upgrade a s/stable cluster to s/candidate	Test squid/stable to squid/candidate upgrades	4	0	0.0%	—
Copilot code review	Prepare	4	0	0.0%	—
Copilot code review	Agent	4	0	0.0%	—
Copilot code review	Upload results	4	0	0.0%	—
Copilot code review	Cleanup artifacts	4	0	0.0%	—
Tests	Regression test - backwardCompatPubnet (#703)	1	0	0.0%	—

Trend: the 30-day window is divided into equal time buckets (daily for ≤ 14 days, weekly for ≤ 90 days, ~monthly beyond that). The failure rate in the first half of those buckets is compared to the second half: ↑ = getting worse, ↓ = improving, → = stable (< 1 pp change). — = fewer than 2 runs per bucket on average; not enough data.

Top 5 Most Failing Jobs

1. Tests / Test quincy upgrades — 31 failures (40.3%)
2. Tests / Multi node testing — 18 failures (14.6%)
3. Tests / Run Unit tests — 14 failures (10.5%)
4. Tests / Single node with encryption — 11 failures (8.9%)
5. Main Documentation Checks / documentation-checks / Spell check / Check spelling in the documentation — 10 failures (22.7%)

Summary

• Total job runs: 3164
• Total failures: 162
• Overall failure rate: 5.1%

[github-actions]

CI Health Report

Last 30 days — generated 2026-05-04T09:41:57Z

Job Failure Rates

Workflow	Job	Runs	Failures	Rate	Trend
Tests	Test quincy upgrades	63	26	41.3%	—
Main Documentation Checks	documentation-checks / Spell check / Check spelling in the documentation	48	9	18.8%	↓ -13.6%
Tests	Multi node testing	130	23	17.7%	↓ -4.5%
Main Documentation Checks	documentation-checks / Link check / Check links in the documentation	48	7	14.6%	↑ +10.7%
Tests	Test reef upgrades	130	14	10.8%	↑ +5.6%
Tests	Build microceph snap	144	14	9.7%	↓ -1.2%
Tests	Single node with encryption	130	12	9.2%	↓ -9.9%
Commits	Signed-off-by (DCO)	74	6	8.1%	↓ -5.1%
Tests	Run Unit tests	144	9	6.2%	↓ -2.6%
Tests	Run static checks	131	7	5.3%	↓ -1.0%
Tests	API tests	102	5	4.9%	↓ -8.2%
Tests	Adopt test with cephadm	130	6	4.6%	↑ +7.5%
Tests	Test MicroCeph RBD Remote Replication features.	130	5	3.8%	↑ +1.5%
Tests	Test maintenance mode	130	4	3.1%	↓ -4.5%
Tests	Test MicroCeph Cephfs Remote Replication features.	130	3	2.3%	↓ -3.4%
Tests	Test WAL/DB device usage	130	2	1.5%	↑ +1.3%
Commits	Canonical CLA signed	74	1	1.4%	↓ -1.9%
Tests	DSL functional tests	103	1	1.0%	↓ -1.6%
Tests	Availability zone crush rule testing	128	1	0.8%	↓ -1.1%
Sphinx python dependency build checks	build	89	0	0.0%	→ +0.0%
Close inactive issues	close-issues	30	0	0.0%	→ +0.0%
Tests	Regression test for sequential join mon host refresh (issue	118	0	0.0%	→ +0.0%
Tests	Test with loopback file OSDs	130	0	0.0%	→ +0.0%
Tests	Multi node testing with custom microceph IP	129	0	0.0%	→ +0.0%
Tests	Verify pristineness and wiping	129	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature	130	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature (multinode)	130	0	0.0%	→ +0.0%
Tests	Test MicroCeph Cluster features.	130	0	0.0%	→ +0.0%
Tests	Messenger v2 tests	130	0	0.0%	→ +0.0%
Main Documentation Checks	OpenAPI Spec Check	48	0	0.0%	→ +0.0%
Main Documentation Checks	documentation-checks / Inclusive language check / Check for inclusive language in the documentation	48	0	0.0%	→ +0.0%
Markdown style checks	markdown-lint	39	0	0.0%	→ +0.0%
CI Health Report	test	7	0	0.0%	—
Upgrade a s/stable cluster to s/candidate	Test squid/stable to squid/candidate upgrades	7	0	0.0%	—
Copilot code review	Prepare	4	0	0.0%	—
Copilot code review	Agent	4	0	0.0%	—
Copilot code review	Upload results	4	0	0.0%	—
Copilot code review	Cleanup artifacts	4	0	0.0%	—
CI Health Report	report	2	0	0.0%	—

Trend: the 30-day window is divided into equal time buckets (daily for ≤ 14 days, weekly for ≤ 90 days, ~monthly beyond that). The failure rate in the first half of those buckets is compared to the second half: ↑ = getting worse, ↓ = improving, → = stable (< 1 pp change). — = fewer than 2 runs per bucket on average; not enough data.

Top 5 Most Failing Jobs

1. Tests / Test quincy upgrades — 26 failures (41.3%)
2. Tests / Multi node testing — 23 failures (17.7%)
3. Tests / Build microceph snap — 14 failures (9.7%)
4. Tests / Test reef upgrades — 14 failures (10.8%)
5. Tests / Single node with encryption — 12 failures (9.2%)

Summary

• Total job runs: 3411
• Total failures: 155
• Overall failure rate: 4.5%

[github-actions]

CI Health Report

Last 30 days — generated 2026-05-11T10:11:31Z

Job Failure Rates

Workflow	Job	Runs	Failures	Rate	Trend
Tests	Test quincy upgrades	25	20	80.0%	—
Main Documentation Checks	documentation-checks / Spell check / Check spelling in the documentation	41	10	24.4%	↓ -13.8%
Main Documentation Checks	documentation-checks / Link check / Check links in the documentation	41	9	22.0%	↑ +10.4%
Tests	Multi node testing	117	18	15.4%	↑ +5.1%
Tests	Test reef upgrades	117	13	11.1%	↑ +12.5%
Tests	Adopt test with cephadm	117	11	9.4%	↑ +15.4%
Tests	Run static checks	121	11	9.1%	↑ +18.2%
Tests	Run Unit tests	123	10	8.1%	↑ +15.6%
Commits	Signed-off-by (DCO)	55	4	7.3%	↑ +9.9%
Tests	Test MicroCeph RBD Remote Replication features.	117	7	6.0%	→ +0.2%
Tests	Build microceph snap	123	6	4.9%	↑ +4.7%
Tests	Single node with encryption	117	5	4.3%	↓ -3.8%
Tests	Test MicroCeph Cephfs Remote Replication features.	117	3	2.6%	↓ -4.4%
Commits	Canonical CLA signed	55	1	1.8%	↓ -3.1%
Tests	Test WAL/DB device usage	117	2	1.7%	→ +0.6%
Tests	DSL functional tests	116	1	0.9%	↓ -1.5%
Close inactive issues	close-issues	30	0	0.0%	→ +0.0%
Sphinx python dependency build checks	build	66	0	0.0%	→ +0.0%
Tests	API tests	116	0	0.0%	→ +0.0%
Tests	Availability zone crush rule testing	117	0	0.0%	→ +0.0%
Tests	Multi node testing with custom microceph IP	116	0	0.0%	→ +0.0%
Tests	Test MicroCeph Cluster features.	117	0	0.0%	→ +0.0%
Tests	Test with loopback file OSDs	117	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature (multinode)	117	0	0.0%	→ +0.0%
Tests	Messenger v2 tests	117	0	0.0%	→ +0.0%
Tests	Regression test for sequential join mon host refresh (issue	117	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature	117	0	0.0%	→ +0.0%
Tests	Test maintenance mode	117	0	0.0%	→ +0.0%
Tests	Verify pristineness and wiping	116	0	0.0%	→ +0.0%
CI Health Report	test	12	0	0.0%	→ +0.0%
Markdown style checks	markdown-lint	36	0	0.0%	→ +0.0%
Main Documentation Checks	OpenAPI Spec Check	41	0	0.0%	→ +0.0%
Main Documentation Checks	documentation-checks / Inclusive language check / Check for inclusive language in the documentation	41	0	0.0%	→ +0.0%
Upgrade a s/stable cluster to tentacle/candidate	Test squid/stable to tentacle/candidate upgrades	7	0	0.0%	—
CI Health Report	report	3	0	0.0%	—
Upgrade a s/stable cluster to s/candidate	Test squid/stable to squid/candidate upgrades	7	0	0.0%	—
Copilot code review	Prepare	2	0	0.0%	—
Copilot code review	Agent	2	0	0.0%	—
Copilot code review	Upload results	2	0	0.0%	—
Copilot code review	Cleanup artifacts	2	0	0.0%	—

Trend: the 30-day window is divided into equal time buckets (daily for ≤ 14 days, weekly for ≤ 90 days, ~monthly beyond that). The failure rate in the first half of those buckets is compared to the second half: ↑ = getting worse, ↓ = improving, → = stable (< 1 pp change). — = fewer than 2 runs per bucket on average; not enough data.

Top 5 Most Failing Jobs

1. Tests / Test quincy upgrades — 20 failures (80.0%)
2. Tests / Multi node testing — 18 failures (15.4%)
3. Tests / Test reef upgrades — 13 failures (11.1%)
4. Tests / Run static checks — 11 failures (9.1%)
5. Tests / Adopt test with cephadm — 11 failures (9.4%)

Summary

• Total job runs: 3054
• Total failures: 131
• Overall failure rate: 4.3%

[github-actions]

CI Health Report

Last 30 days — generated 2026-05-18T10:25:47Z

Job Failure Rates

Workflow	Job	Runs	Failures	Rate	Trend
Commits	Signed-off-by (DCO)	40	13	32.5%	↑ +55.0%
Tests	Multi node testing	84	26	31.0%	↑ +38.6%
Main Documentation Checks	documentation-checks / Link check / Check links in the documentation	28	7	25.0%	↑ +12.3%
Tests	Test reef upgrades	71	13	18.3%	↑ +6.5%
Main Documentation Checks	documentation-checks / Spell check / Check spelling in the documentation	28	5	17.9%	↑ +22.8%
Tests	Adopt test with cephadm	71	10	14.1%	↑ +8.5%
Tests	Run static checks	73	9	12.3%	↑ +28.1%
Tests	Run Unit tests	75	9	12.0%	↑ +28.1%
Tests	Test MicroCeph RBD Remote Replication features.	70	4	5.7%	↑ +1.3%
Tests	Build microceph snap	79	4	5.1%	↓ -9.3%
Tests	Single node with encryption	71	1	1.4%	↑ +3.1%
Tests	DSL functional tests	71	1	1.4%	↑ +3.1%
Tests	Test WAL/DB device usage	71	1	1.4%	↓ -2.6%
Close inactive issues	close-issues	30	0	0.0%	→ +0.0%
Commits	Canonical CLA signed	40	0	0.0%	→ +0.0%
Sphinx python dependency build checks	build	48	0	0.0%	→ +0.0%
Tests	Messenger v2 tests	75	0	0.0%	→ +0.0%
Tests	Multi node testing with custom microceph IP	70	0	0.0%	→ +0.0%
Tests	Availability zone crush rule testing	71	0	0.0%	→ +0.0%
Tests	Test with loopback file OSDs	71	0	0.0%	→ +0.0%
Tests	Test MicroCeph Cluster features.	71	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature	71	0	0.0%	→ +0.0%
Tests	Test MicroCeph Cephfs Remote Replication features.	71	0	0.0%	→ +0.0%
Tests	Test maintenance mode	71	0	0.0%	→ +0.0%
Tests	Verify pristineness and wiping	70	0	0.0%	→ +0.0%
Tests	API tests	71	0	0.0%	→ +0.0%
Tests	Test MicroCeph NFS feature (multinode)	71	0	0.0%	→ +0.0%
Tests	Regression test for sequential join mon host refresh (issue	71	0	0.0%	→ +0.0%
CI Health Report	test	13	0	0.0%	→ +0.0%
CI Health Report	report	4	0	0.0%	—
Markdown style checks	markdown-lint	24	0	0.0%	→ +0.0%
Main Documentation Checks	OpenAPI Spec Check	28	0	0.0%	→ +0.0%
Main Documentation Checks	documentation-checks / Inclusive language check / Check for inclusive language in the documentation	28	0	0.0%	→ +0.0%
Upgrade a s/stable cluster to tentacle/candidate	Test squid/stable to tentacle/candidate upgrades	7	0	0.0%	—
Upgrade a s/stable cluster to s/candidate	Test squid/stable to squid/candidate upgrades	7	0	0.0%	—
Copilot code review	Prepare	2	0	0.0%	—
Copilot code review	Agent	2	0	0.0%	—
Copilot code review	Upload results	2	0	0.0%	—
Copilot code review	Cleanup artifacts	2	0	0.0%	—

Trend: the 30-day window is divided into equal time buckets (daily for ≤ 14 days, weekly for ≤ 90 days, ~monthly beyond that). The failure rate in the first half of those buckets is compared to the second half: ↑ = getting worse, ↓ = improving, → = stable (< 1 pp change). — = fewer than 2 runs per bucket on average; not enough data.

Top 5 Most Failing Jobs

1. Tests / Multi node testing — 26 failures (31.0%)
2. Commits / Signed-off-by (DCO) — 13 failures (32.5%)
3. Tests / Test reef upgrades — 13 failures (18.3%)
4. Tests / Adopt test with cephadm — 10 failures (14.1%)
5. Tests / Run static checks — 9 failures (12.3%)

Summary

• Total job runs: 1923
• Total failures: 103
• Overall failure rate: 5.4%

[github-actions]

CI Health Report

Last 30 days — generated 2026-05-25T10:29:03Z

Job Failure Rates

Workflow	Job	Runs	Failures	Rate	Trend
Tests	DSL WAL-DB consistency tests	5	2	40.0%	—
Tests	DSL WAL-DB cleanup tests	5	2	40.0%	—
Tests	Multi node testing	79	28	35.4%	↑ +45.0%
Commits	Signed-off-by (DCO)	46	16	34.8%	↑ +43.5%
Main Documentation Checks	documentation-checks / Link check / Check links in the documentation	18	5	27.8%	—
Tests	Adopt test with cephadm	66	16	24.2%	↑ +30.7%
Tests	Test reef upgrades	66	16	24.2%	↑ +14.9%
Tests	Run static checks	65	14	21.5%	↑ +18.1%
Tests	Run Unit tests	67	14	20.9%	↑ +18.7%
Tests	DSL baseline tests	5	1	20.0%	—
Tests	DSL WAL-DB validation tests	5	1	20.0%	—
Tests	DSL WAL-DB provision tests	5	1	20.0%	—
Tests	DSL WAL-DB dry-run tests	5	1	20.0%	—
Main Documentation Checks	documentation-checks / Spell check / Check spelling in the documentation	18	3	16.7%	—
Tests	Verify pristineness and wiping	66	8	12.1%	↑ +47.1%
Tests	API tests	66	6	9.1%	↑ +35.3%
Tests	Test MicroCeph RBD Remote Replication features.	65	5	7.7%	↑ +6.4%
Tests	Build microceph snap	76	5	6.6%	↓ -3.2%
Tests	Test MicroCeph Cephfs Remote Replication features.	66	4	6.1%	↑ +23.5%
Tests	DSL functional tests	59	3	5.1%	↑ +18.0%
Tests	Test maintenance mode	66	2	3.0%	↑ +11.8%
Tests	Test WAL/DB device usage	66	2	3.0%	↑ +3.8%
Tests	Single node with encryption	66	2	3.0%	↑ +3.8%
Commits	Canonical CLA signed	46	1	2.2%	↑ +4.3%
Tests	Availability zone crush rule testing	66	1	1.5%	↑ +5.9%
Tests	Test with loopback file OSDs	66	1	1.5%	↑ +5.9%
Tests	Multi node testing with custom microceph IP	66	1	1.5%	↑ +5.9%
Tests	Test MicroCeph NFS feature (multinode)	66	1	1.5%	↑ +5.9%
Tests	Test MicroCeph NFS feature	66	1	1.5%	↑ +5.9%
Tests	Test MicroCeph Cluster features.	66	1	1.5%	↑ +5.9%
Tests	Regression test for sequential join mon host refresh (issue	66	1	1.5%	↑ +5.9%
Tests	Messenger v2 tests	70	1	1.4%	↑ +4.8%
Close inactive issues	close-issues	30	0	0.0%	→ +0.0%
Sphinx python dependency build checks	build	50	0	0.0%	→ +0.0%
Checks	Run Unit tests	5	0	0.0%	—
Checks	Run static checks	5	0	0.0%	—
Tests	Test quincy upgrades	2	0	0.0%	—
CI Health Report	test	10	0	0.0%	→ +0.0%
CI Health Report	report	4	0	0.0%	—
Markdown style checks	markdown-lint	16	0	0.0%	—
Main Documentation Checks	OpenAPI Spec Check	18	0	0.0%	—
Main Documentation Checks	documentation-checks / Inclusive language check / Check for inclusive language in the documentation	18	0	0.0%	—
Upgrade a s/stable cluster to tentacle/candidate	Test squid/stable to tentacle/candidate upgrades	7	0	0.0%	—
Upgrade a s/stable cluster to s/candidate	Test squid/stable to squid/candidate upgrades	3	0	0.0%	—
Copilot code review	Prepare	2	0	0.0%	—
Copilot code review	Agent	2	0	0.0%	—
Copilot code review	Upload results	2	0	0.0%	—
Copilot code review	Cleanup artifacts	2	0	0.0%	—

Trend: the 30-day window is divided into equal time buckets (daily for ≤ 14 days, weekly for ≤ 90 days, ~monthly beyond that). The failure rate in the first half of those buckets is compared to the second half: ↑ = getting worse, ↓ = improving, → = stable (< 1 pp change). — = fewer than 2 runs per bucket on average; not enough data.

Top 5 Most Failing Jobs

1. Tests / Multi node testing — 28 failures (35.4%)
2. Commits / Signed-off-by (DCO) — 16 failures (34.8%)
3. Tests / Adopt test with cephadm — 16 failures (24.2%)
4. Tests / Test reef upgrades — 16 failures (24.2%)
5. Tests / Run Unit tests — 14 failures (20.9%)

Summary

• Total job runs: 1805
• Total failures: 166
• Overall failure rate: 9.2%