Skip to content

POC: add tool qualification verification report model - #831

Draft
AlexanderLanin wants to merge 3 commits into
eclipse-score:mainfrom
etas-contrib:tool-verification-report
Draft

AlexanderLanin wants to merge 3 commits into
eclipse-score:mainfrom
etas-contrib:tool-verification-report

Conversation

@AlexanderLanin

Copy link
Copy Markdown
Member

Why

A tool qualification report needs to show, in one readable model, which tool use cases are evaluated, which malfunctions can violate requirements, how those malfunctions affect safety, and which tests provide evidence. The POC makes this chain machine-readable and renders it as a qualification matrix and traceability table.

The structure is designed for a process reader: use cases are visible as top-level report items, while their malfunctions are shown directly beneath them. The report then connects each malfunction to violated requirements and available verification evidence.

What the process reader can follow

The model expresses this flow:

tool-usecase
  1:n
tool-malfunction
  n:1 violates
requirements
  1:n verified by
testcase

A malfunction can be fully or partially verified by multiple testcases.

Metamodel elements and links

Element Purpose Links and attributes
tool-usecase A behavior or view of the tool that is being qualified Top-level Need; its malfunctions are nested below it
tool-malfunction A possible tool failure or false result within a use case Nested under one tool-usecase; violates one or more requirements; optional free-text safety_measure; safety impact and detection attributes
Existing testcase Verification evidence for a malfunction fully_verifies or partially_verifies points to one or more malfunctions

The supported requirement targets for violates are general, stakeholder, feature, component, tool, and assumption-of-use requirements.

The nesting relationship is represented by Sphinx-Needs parent_needs. The generated UI labels this relationship as contains on the use case and contained by on the malfunction, avoiding an implementation-oriented parent label.

The qualified doc_tool Need remains the report identity and owns the post-template report generation, but the use cases are intentionally not nested under it. The report template discovers the top-level tool-usecase Needs and follows their nested malfunctions.

Example and report output

The example is based on the Docs-as-Code tool verification concerns from the S-CORE documentation, including:

  • build and CI behavior
  • safety-critical link checks
  • requirements coverage
  • testcase linkage and references
  • assumptions of use
  • generated architecture views, backlinks, and documentation

The generated report contains:

  1. a tool qualification matrix grouped by use case and malfunction
  2. a malfunction traceability table with violated requirements, safety measures, and linked testcase evidence

The POC adds additional best-guess testcase links to demonstrate that one malfunction can be covered by multiple tests while another can remain without direct testcase evidence.

Scope

This is a POC for validating the metamodel shape, nested-Need rendering, graph-driven report generation, and testcase traceability.

@AlexanderLanin AlexanderLanin changed the title feat: add tool qualification verification report model POC: add tool qualification verification report model Sep 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Documentation preview for this pull request is available at:
pr-831: https://eclipse-score.github.io/docs-as-code/pr-831/

This report evaluates the Docs-as-Code toolchain as a qualified tool for
producing and verifying safety-relevant documentation.

.. tool-usecase:: Build and CI behavior

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the author of the doc_as_code.rst this is derived from, I wonder why there is only two use cases here?

For me, each malfunction is a separate use case and their distinct description is important. Also, reference to the process requirements are important. This is all lost here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

POC scope has ended there :p

:post_template: tool_qualification_report

This report evaluates the Docs-as-Code toolchain as a qualified tool for
producing and verifying safety-relevant documentation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security evaluation is missing but required by S-CORE process.

Comment thread docs/internals/requirements/tool_qualification_example.rst
:detection_sufficient: NO
:additional_safety_measure_required: YES
:safety_measure: Pull-request review checks changes to safety-critical links before merge.
:violates: tool_req__docs_common_attr_safety_link_check

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No way that this single requirement is sufficient. I expect a list of ten or more requirements for this one. Similar for the other malfunctions.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: dont review the content. this is a POC

@attifunel
attifunel self-requested a review September 16, 2026 13:38
@AlexanderLanin
AlexanderLanin force-pushed the tool-verification-report branch from 2ca13d2 to 47270ce Compare September 17, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants