Neil Comments on Conformance docs in this Repo

[neiljthomson]

Comments on: Conformance for CIOSC-103-1

Comments are organized mostly around the GitHub file names (.md files)

Documents

read.me

Line #15 - Regulated Programs

• List of examples does not include independent (accredited agencies or services) issuing Identifiers (e.g. 100-3 assigned identifiers), identities (what ever that means) or (verifiable credentials). Does that mean that Decentralized (Self-Sovereign Identity) is not supported (vs. Gov't, large non-profit/profit orgs (e.g., banks) - which is a Sovereign Identity/Distributed Centralized model.

Line #37 - Scope of Schema 

• Conspicuous by it's absence in the scope - Devices (IoT) - devices will certainly require identities. They also need a CAS as to how their identities are verified/validated. Consideration for Devices are missing throughout the CIOSC Conformance document set in this GitHub repository

conformity-assessment.md

General comments

• I believe that this document would benefit from describing the context (and components) that Conformity Assessment fits into, possibly by pointing to a backgrounder doc (e.g. Conformance Vocabulary; NIST document ABCs of Conformity Assessment)
• That includes positioning Governance (which I'm still not clear on how that relates to the AB/CAB/CAS structure). That would also be very useful for the ToIP crowd. 

iaf-md25-requirements.md

• line #15 - AB acronym not defined. Accreditation Body?

Primer.md

General comment is the diagrams are not entirely in sync with each other (understanding that these are early drafts) - names of the parts, acronyms and their relationships are not consistent across the different diagrams. Possibly consolidate?

• Accreditation Process - Diagram. I'm unclear on the "Clients" box and its relationship to the boxes inside the dotted line. They are Clients of what? The Accreditation process? Is a Client and organization a producer of products and services (1st Party)?

The following is a partial alternative diagram to the one in the primer.md "Accreditation Process" diagram:

I was expecting to see the following parties 1st (producer of products and services), 2nd (consumer) and 3rd (independent assessment organization). I see ABs (listed as Accredited Certification Bodies), but don't see Schema Owners (SOs) or Conformance Assessment Bodies (CABs).

• Certification Process - I don't see testing (Determination), Inspection or other specific methods of assessment, only audit (audit-methods.md) (which happens to confirm continued testing and inspection?). In the vocabulary document (see below) Audit is mentioned separately from determination and inspection.
• Trust Frameworks and Standards Development Landscape - I don't see CABs, only ACs. Is this diagram about assessing CABs (as accredited evaluators of Conformance Schemas) or Providers/1st Parties (producers of products and services)
• Standards and Architecture Landscape - No entirely clear I understand the terms and relationships
  • "Accreditation Programs" - my understanding is there are specific terms used in different contexts - Accreditation applies to CABs and Conformance Certificates for Products and services
  • "Acceptance" s this issuance of a Conformance Certificate?
  • Is "Governance" in this context - managing ongoing conformance to approved organizational polices, processes and procedures, including auditing, inspections and testing?
  • Unclear why PSP PCTF is significant enough to include in the diagram - sounds like one of many work products (Assessment Worksheet)
  • "Certification Scheme" or "Conformance Assessment Scheme"?

schema-manual.md

General comments

• Schema Owner and Schema Development
  • Which organization is the Scheme Owner (SO)?
  • Which organizations can vs. must contribute to the CAS
    • I assume that the CAB assigned to the producer/product or service is responsible for approving the CAS

Specific comments

• 4.1 Applicability
  • lines 30, 31 - are there not additional documents (already planned) that will apply beyond 103-1, 103-2?
  • line 33 - Not clear on this statement "An organization can review the applicability of requirements due to the size or complexity of the organization"
    • Not clear on what the size/complexity of the organization has to do with what requirements they can choose to conform (or ignore). Link to reference that clarifies?
    • As also true in 4.2 Exclusions. The language I would expect in such a document for what is mandatory and optional (SHALL, MUST, SHOULD, ...) vs. organizations having the option to "cherry pick" what they conform to. 

audit-methods.md

No mention of specific assessment methods (e.g., test, inspection), which, pragmatically, impacts the rigor/quality of the "audit methods"/assessment process.

Given that "conformance testing" of automobiles and aircraft are specifically required to pass "crash tests" of various types, what would be the context for general conformance compliance - or are vehicles outside the scope of Conformance Compliance for software? 

Useful documents (outside the docs in GitHub)

• Conformance Vocabulary; NIST document ABCs of Conformity Assessment

Terms

A pass at understanding the related components, their purpose, acronyms, etc.

• A Conformance Assessment Body - is a 3rd party organization, which evaluates Conformance of an organization's processes and procedures (which constitute and are captured in a(?) (Conformance Assessment) Schema (CAS). CABs are validated/certified/approved/given accreditation by an Accreditation Body

[trbouma]

Thanks, Neil. I will be going through your comments in detail.