Background
While reviewing #427 (OSPS-SA-02.01), the reviewer noted that sec_assessment.HasDesignDocumentation (SA-01.01) and the new HasExternalInterfaceDocumentation (SA-02.01) now disagree on release gating and pass strictness for identical repository states, with SA-02.01 on the more defensible side.
Divergences
- Release gating — SA-02.01 gates on a published (non-draft) release via
reusable_steps.HasPublishedRelease and returns NotApplicable when there is none. SA-01.01 does not consider releases at all.
- Pass strictness — SA-02.01 never returns
Passed from automated heuristics (any positive signal → NeedsReview), because automation cannot confirm coverage of all interfaces/actors. SA-01.01 still returns Passed when a design-doc file is found.
- Confidence hygiene — SA-01.01's terminal
Failed branch (and its NeedsReview fallback) return the unset named confidence (i.e. Undetermined), the same bug that was fixed for SA-02.01. These should be explicit levels.
Proposed follow-up
- Consider having SA-01.01 downgrade a design-doc file match from
Passed to NeedsReview for consistency with SA-02.01 (SA-01 is a similar "cannot confirm completeness" heuristic).
- Set explicit confidence levels on all SA-01.01 return branches.
- Optionally reuse
reusable_steps.HasPublishedRelease if SA-01's applicability should also be release-gated (needs a baseline check — SA-01 may intentionally apply pre-release).
- More broadly, the project may want a shared convention for the "tag-only release yields NotApplicable at High confidence" pattern noted across several steps.
Refs: #427, #426.
Filed as a follow-up from PR #427 review; not blocking that PR.
Background
While reviewing #427 (OSPS-SA-02.01), the reviewer noted that
sec_assessment.HasDesignDocumentation(SA-01.01) and the newHasExternalInterfaceDocumentation(SA-02.01) now disagree on release gating and pass strictness for identical repository states, with SA-02.01 on the more defensible side.Divergences
reusable_steps.HasPublishedReleaseand returnsNotApplicablewhen there is none. SA-01.01 does not consider releases at all.Passedfrom automated heuristics (any positive signal →NeedsReview), because automation cannot confirm coverage of all interfaces/actors. SA-01.01 still returnsPassedwhen a design-doc file is found.Failedbranch (and itsNeedsReviewfallback) return the unset namedconfidence(i.e.Undetermined), the same bug that was fixed for SA-02.01. These should be explicit levels.Proposed follow-up
PassedtoNeedsReviewfor consistency with SA-02.01 (SA-01 is a similar "cannot confirm completeness" heuristic).reusable_steps.HasPublishedReleaseif SA-01's applicability should also be release-gated (needs a baseline check — SA-01 may intentionally apply pre-release).Refs: #427, #426.
Filed as a follow-up from PR #427 review; not blocking that PR.