fix(analyzer): add Korean context terms to KrRrnRecognizer (#2212) - #2258
Open
bodapatisaikrishna wants to merge 1 commit into
Open
bodapatisaikrishna wants to merge 1 commit into
bodapatisaikrishna wants to merge 1 commit into
Conversation
…acy-stack#2212) KrRrnRecognizer.CONTEXT only contained English keywords ('Korean RRN', 'RRN', etc.), so context enhancement never fired for Korean-language documents where RRNs appear. All other Korean recognizers already define native Korean context terms. Adds '주민등록번호', '주민번호', '주민등록증', '주민등록', '신분증', and '본인인증' to KrRrnRecognizer.CONTEXT and adds unit tests covering context presence and score enhancement. Fixes data-privacy-stack#2212.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
Fixes an omission in
KrRrnRecognizer.CONTEXTwhere only English terms ("Korean RRN","RRN", etc.) were defined, unlike all other Korean recognizers (KrDriverLicenseRecognizer,KrPassportRecognizer,KrFrnRecognizer,KrBrnRecognizer) which include native Korean keywords.Because Korean Resident Registration Numbers (주민등록번호) primarily appear in Korean documents and text, context enhancement was never firing for native Korean context phrases. This was especially problematic for post-October 2020 RRNs (which lack the pre-2020 checksum algorithm and return pattern score
0.5), leaving them below typical analyzer acceptance thresholds without the context boost.Changes:
KrRrnRecognizer.CONTEXT:"주민등록번호"(Resident Registration Number)"주민번호"(short form: Resident Number)"주민등록증"(Resident Registration Card / ID card)"주민등록"(Resident Registration)"신분증"(ID card / Identification)"본인인증"(Identity Verification)presidio-analyzer/tests/test_kr_rrn_recognizer.py:KrRrnRecognizer.context.LemmaContextAwareEnhancer(score boosted from 0.5 to 0.85 withsupportive_context_wordset).Issue reference
Fixes #2212
Checklist