Repository navigation
test(contracts): 저위험 문구 핀을 구조·토큰 검사로 정리 (T1) - #360
Conversation
- check_changelog_quality_contract_notes 삭제(과거 토큰 존재만 확인) - check_skill_frontmatter_minimal: description 문구 핀 → 비어 있지 않음 - check_readme_quality_contract_map: 37개 문자열 → 품질 계약 표 8행 구조와 행별 회귀 검증 참조(check_*/router-*), 채점기 한계 문구는 유지 - check_readme_investigation_assist_posture: 예시 5개 핀 → 상태 태그 존재, 과장 문구 금지와 조사 보조 포지션 순서 토큰은 유지 - epic_317 기록·bulk grid·design archive 검사는 손대지 않음(교차 리뷰 High 판정) mutation 9건: 행 삭제·참조 제거·한계 문구 삭제·태그 삭제·과장 주입·빈 description은 FAIL, 문구 다듬기 3건은 PASS. no-changelog: 정적 검사 내부의 문구 핀 정리로 스킬 동작과 사용자 표면 변화가 없다 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configuration
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41bff34b60
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if required not in description: | ||
| raise AssertionError(f"SKILL.md description missing {required!r}") | ||
| # The description's wording is free; only its presence is a contract. | ||
| if not str(frontmatter.get("description", "")).strip(): |
There was a problem hiding this comment.
When the frontmatter contains description: null (or a list/mapping), converting the value with str(...) produces a non-empty representation such as "None", so this check passes despite the description not being usable skill metadata. Check that the value is a string before applying strip() so the promised non-empty-description contract is actually enforced.
Useful? React with 👍 / 👎.
| if len(row) < 3 or not re.search(r"check_\w+|router-\d+", row[2]): | ||
| raise AssertionError(f"{label}: {required_row!r} 행에 회귀 검증 참조가 없다") |
There was a problem hiding this comment.
If a verification cell is changed to an unquoted typo such as check_definitely_does_not_exist, this regex accepts it, while check_readme_quality_verification_refs_resolve ignores it because that function only extracts backtick-quoted references. The two checks therefore pass even though the row no longer points to an existing regression check; require at least one extracted, resolvable reference rather than accepting any matching substring.
Useful? React with 👍 / 👎.
| for status_tag in ["[INSUFFICIENT]", "[UNVERIFIED]"]: | ||
| assert_contains(text, status_tag, label) |
There was a problem hiding this comment.
If [INSUFFICIENT] and [UNVERIFIED] remain anywhere else in README.md, these whole-file searches pass even when the introductory contract-review example removes all status labels and presents its conclusions as confirmed. Since the stated contract is that the public example demonstrates reserved status, inspect the example’s fenced block or another explicit example anchor rather than the entire document.
Useful? React with 👍 / 👎.
- description: 문자열 타입 확인, "기억만으로 답하지 않는다" 경계를 부정 토큰으로, 본문 집(기본 조사 계약 환각 방지)에 순서 토큰 가드 - 품질 계약 지도: 백틱 참조만 인정(resolver와 동일), 안전 행 핵심 참조를 REQUIRED_MAP_REFS로 행 구조에 바인드(중복 블록 제거), evaluator 한계 문구를 주어 문장에 순서 토큰으로, gate 우회 금지 규칙 순서 토큰 - 조사 보조 포지션: 도입 예시 블록 안에서 유보 태그 요구, [VERIFIED] 금지 mutation 10건(null description, 경계 삭제, 환각 방지 삭제, 가짜 참조, 핵심 참조 삭제, 한계 반전, 우회 금지 삭제, 예시 [VERIFIED])은 FAIL, 문구 다듬기 2건은 PASS. no-changelog: 정적 검사 내부의 가드 보강으로 스킬 동작과 사용자 표면 변화가 없다 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7a9a07025
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| refs = set(re.findall(r"`([^`]+)`", row[2])) if len(row) >= 3 else set() | ||
| if not any(ref.startswith(("check_", "router-")) for ref in refs): | ||
| raise AssertionError(f"{label}: {required_row!r} 행에 회귀 검증 참조가 없다") |
There was a problem hiding this comment.
For rows omitted from REQUIRED_MAP_REFS, this accepts any existing check or router scenario, even if it tests a different contract. For example, replacing the Legal verification core's router-16 with the unrelated but valid router-01 still lets the complete validator pass, so the README can silently lose its only scenario mapping. The new backtick-resolution logic proves that a reference exists, but not that it belongs to the row; require row-specific references for every required row.
Useful? React with 👍 / 👎.
| raise AssertionError( | ||
| f"README.md Contract review 행에 {required!r}가 없다: {contract_row!r}" | ||
| ) | ||
| assert_contains(text, "품질 계약 변경 체크리스트", label) |
There was a problem hiding this comment.
This whole-file substring remains present in the Changelog gate row and release checklist, so removing the actual ### 품질 계약 변경 체크리스트 heading and its eight-step table still passes this assertion; the later gate-bypass sentence and command block satisfy the remaining checks as well. That allows the maintainer workflow promised by the README to disappear while the full validator stays green, so parse and validate the checklist section or its table directly.
Useful? React with 👍 / 👎.
- 품질 계약 변경 체크리스트: 단계 1..8 연속과 단계별 포인터(SKILL.md, 16_router_regression.yaml, router_guardrail_outputs.yaml, validate_skill_contracts.py, CHANGELOG)를 표 구조로 검사 - description 경계: 부정어를 "기억으로 답하는" 행위에 결속(반전 차단) - 환각 방지 순서 토큰에 "만들지 않", gate 우회 금지 문단 첫 규칙 포함, Legal verification core 행 router-16 결속 - 도입 예시: 쟁점 줄마다 유보 태그, "확정" 금지 mutation 29건(기존 9, 1차 10, 2차 10) 모두 기대대로. no-changelog: 정적 검사 내부의 가드 보강으로 스킬 동작과 사용자 표면 변화가 없다 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67828d998d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # 부정어는 "기억으로 답하는" 행위 자체에 붙어야 한다("답해도 제한하지 않는다" 반전 차단). | ||
| if not re.search(r"기억\S{0,4}\s*(?:답하지|의존하지)\s*않", description): | ||
| raise AssertionError("SKILL.md description: 기억만으로 답하지 않는다는 경계가 없다") |
There was a problem hiding this comment.
When the description is harmlessly reworded to something like 한국 법령과 계약 질문에 사용하며 기억에 의존해 답하지 않는다, this regex rejects it because it only accepts 답하지 or 의존하지 immediately after a short suffix on 기억. That preserves the same no-memory boundary but now fails the contract suite, recreating the prose lock this migration is intended to remove; validate the relevant concepts and negation without requiring this exact Korean grammatical shape.
Useful? React with 👍 / 👎.
| # The description's wording is free; its presence and the "not from memory" | ||
| # honesty boundary are the contract. | ||
| description = frontmatter.get("description") | ||
| if not isinstance(description, str) or not description.strip(): | ||
| raise AssertionError("SKILL.md frontmatter description is empty") | ||
| # 부정어는 "기억으로 답하는" 행위 자체에 붙어야 한다("답해도 제한하지 않는다" 반전 차단). | ||
| if not re.search(r"기억\S{0,4}\s*(?:답하지|의존하지)\s*않", description): | ||
| raise AssertionError("SKILL.md description: 기억만으로 답하지 않는다는 경계가 없다") |
There was a problem hiding this comment.
A description containing only 기억으로 답하지 않는다 now passes this check, even though it no longer identifies Korean law, contract review, or compliance as activation triggers. Because this frontmatter description is what the skill dispatcher can use before loading the body, such an edit can prevent the automatic activation advertised by the README while all contract checks remain green; retain semantic domain/usage requirements without pinning the original sentence.
Useful? React with 👍 / 👎.
| issue_lines = [ | ||
| line | ||
| for line in example.group("body").splitlines() | ||
| if re.match(r"\s*- .+:", line) or (re.match(r"\s*\| [A-Z]", line) and "위험도" not in line) |
There was a problem hiding this comment.
When an example issue row begins with Korean text, lowercase text, or a number—for example | 손해배상 | 상 | 면책은 무효다 |—this classifier excludes it because table rows must start with [A-Z]. The example can therefore contain an unreserved definitive conclusion while separate rows retain the two required tags, and the validator still passes; identify Markdown data rows structurally rather than by the first cell's alphabet.
Useful? React with 👍 / 👎.
- 도입 예시 쟁점 줄을 문자 종류가 아니라 구조로(모든 불릿·표 데이터 행, 헤더·구분선 제외) 판정해 한글 조항명·콜론 없는 불릿 우회 차단 - description 경계 부정어 끝을 "않는다/않음/않고"로 결속, gate 우회 금지 토큰을 "우회하지 말"로, 도입 문단 공개 약속(출처 권위 라벨·verification status·최신성) 순서 토큰 - 지도 참조 resolver도 parse_markdown_table을 써서 들여쓴 행이 숨지 못하게 mutation 35건(기존 29 + 신규 6) 모두 기대대로. no-changelog: 정적 검사 내부의 가드 보강으로 스킬 동작과 사용자 표면 변화가 없다 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 220921b955
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for status_tag in ["[INSUFFICIENT]", "[UNVERIFIED]"]: | ||
| assert_contains(example.group("body"), status_tag, f"{label} 도입 예시") | ||
| assert_not_contains(example.group("body"), "[VERIFIED]", f"{label} 도입 예시") | ||
| assert_not_contains(example.group("body"), "확정", f"{label} 도입 예시") |
There was a problem hiding this comment.
A safely reserved example such as [UNVERIFIED] 현재 자료로는 확정할 수 없음 or [UNVERIFIED] 미확정 쟁점 now fails solely because it contains 확정. These wordings explicitly avoid presenting a confirmed conclusion, so match affirmative confirmation semantics rather than rejecting the substring everywhere in the example.
Useful? React with 👍 / 👎.
| if re.match(r"\s*[-|]", line) | ||
| and "위험도" not in line | ||
| and not re.match(r"^\s*\|[\s:|-]+\|\s*$", line) |
There was a problem hiding this comment.
If the example table harmlessly renames 위험도 to 리스크 or 심각도, its header row starts with |, is not a separator, and is therefore treated as an issue row that must contain a reservation tag. This recreates a prose lock on the header text; identify the Markdown header structurally (for example, from its following separator) rather than exempting only the current 위험도 spelling.
Useful? React with 👍 / 👎.
- 도입 예시 쟁점 줄: 표 헤더를 문자열("위험도")이 아니라 표 묶음의 첫 행으로
판별, 구분선 제외, 목록은 -·*·+·번호 목록 모두 포함
- gate 우회 금지 순서 토큰에 보호 대상(출처 권위 라벨, 자가 검증) 포함
mutation 40건(기존 35 + 신규 5) 모두 기대대로.
no-changelog: 정적 검사 내부의 가드 보강으로 스킬 동작과 사용자 표면 변화가 없다
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 33b035f329
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| assert_ordered_tokens( | ||
| research_contract.group(0), ["환각 방지", "만들지 않", "[INSUFFICIENT]", "[UNVERIFIED]"], "SKILL.md 환각 방지" |
There was a problem hiding this comment.
This ordered-token check accepts an explicit reversal such as 환각 방지 — 확인되지 않은 조문을 만들지 않아도 되며 [INSUFFICIENT]와 [UNVERIFIED]를 쓰지 않는다, because all four tokens still appear in order. A future edit could therefore permit fabricated legal citations and forbid the reservation labels while the contract suite remains green; bind the negation to fabrication and require the tags as fallback outcomes rather than checking token order alone.
Useful? React with 👍 / 👎.
| if is_row and not in_table: | ||
| in_table = True | ||
| continue # header row of this table run |
There was a problem hiding this comment.
The new first-row branch assumes every run of pipe-prefixed lines begins with a header. If the example's header and separator are accidentally removed, its first actual issue row is skipped unconditionally; that row can contain an unreserved definitive conclusion while later tagged rows satisfy the global tag checks, and validation still passes. Only treat a row as a header when the following row is a Markdown separator, or otherwise validate the first row as issue content.
Useful? React with 👍 / 👎.
1 participant
요약
키워드·문장 그대로를 고정하던 정적 검사 중 위험이 낮은 4개를 구조·토큰 검사로 바꾼다(T1). 스킬 동작은 바꾸지 않는다. 앞서 Grok과 Sol이 서로의 분류를 교차 검토했고, 그 결과 FAIL 기록을 지키는 에픽 기록 검사(epic_317)와 README 과장 광고 방지는 대상에서 뺐다.
check_changelog_quality_contract_notescheck_skill_frontmatter_minimalcheck_readme_quality_contract_mapcheck_readme_investigation_assist_posture독립 리뷰 (Claude 안전·정합, GPT-6.1 Sol 테스트)
5라운드를 돌았고, 두 리뷰어 모두
33b035f에서 LGTM을 냈다. 리뷰는 "문구 핀을 풀자 조용히 통과하던 회귀"를 다음과 같이 찾아 막았다.[VERIFIED]·"확정"을 금지했다. 표 헤더와 목록은 구조로 판별한다.최종 mutation은 40건이며 모두 기대대로 나왔다.
리뷰 후 남은 Low (후속, merge를 막지 않음):
검증
no-changelog: 정적 검사 내부의 문구 핀 정리로 스킬 동작과 사용자 표면 변화가 없다
🤖 Generated with Claude Code