Repository navigation
Add cpf-lgpd scanner package with CLI, docs, tests and CI - #1
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 90db5da4c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self._analyze_record(self._buffer[:boundary]) | ||
| self._buffer = self._buffer[boundary:] |
There was a problem hiding this comment.
When a newline-free record exceeds maximum_buffer, this analyzes the prefix and retains only the configured context tail without any regex overlap. A valid 11-digit CPF that straddles boundary is split into two non-matching fragments and is silently omitted; for example, with the defaults, placing a CPF at the internal cutoff of a text line longer than 64 KiB produces zero findings. Retain enough overlap for the longest CPF representation or defer parsing the boundary fragment.
Useful? React with 👍 / 👎.
| total = 0 | ||
| members = 0 |
There was a problem hiding this comment.
For nested archives, these counters are reset on every recursive _archive call, so max_archive_size and max_archive_members apply independently to each inner archive rather than to the top-level input. An outer ZIP can therefore contain many small, highly compressed inner ZIPs, each expanding up to the full limit; the scanner may process hundreds of gigabytes despite a 200 MiB configured cap, and the processing deadline is only checked after the entire top-level file returns. Track a shared extraction budget through recursion.
Useful? React with 👍 / 👎.
| print("Relatorio protegido gravado com sucesso.") | ||
| if args.csv_report: | ||
| print("Relatorio CSV protegido gravado com sucesso.") | ||
| return 1 if result.valid_cpfs else 0 |
There was a problem hiding this comment.
When max_processing_seconds expires before any CPF is encountered, scan_directory sets scan_timed_out and stops early, but this return expression still exits with status 0. Without an optional JSON report, the terminal also shows no timeout indication and zero failed files, so automation and operators can interpret a partial scan as a completed scan with no CPFs. Return an operational/incomplete status when result.scan_timed_out is true and surface that condition.
Useful? React with 👍 / 👎.
| for info in archive.infolist(): | ||
| if info.file_size > limits.max_member_size: | ||
| raise ExtractionLimitError("parte Office excede limite") | ||
| if info.filename in names or info.filename.startswith(prefixes): | ||
| yield _xml_text(archive.read(info)) + "\n" |
There was a problem hiding this comment.
DOCX and XLSX inputs only enforce max_member_size; they never apply max_archive_members or max_archive_size. A crafted workbook containing many worksheet XML entries just below the per-member limit can therefore force unbounded aggregate decompression and XML parsing, bypassing the zip-bomb limits used for ordinary archives. Count members and cumulative uncompressed bytes before reading selected Office entries.
Useful? React with 👍 / 👎.
| weights = ScoreWeights(**_known_values(ScoreWeights, values.pop("weights", {}))) | ||
| governance_values = _known_values(GovernanceMetadata, values.pop("governance", {})) |
There was a problem hiding this comment.
If a user supplies syntactically valid JSON with a wrong nested type, such as "weights": null, _known_values receives a non-mapping and raises TypeError. The CLI only catches OSError and ValueError, so this configuration error escapes as a traceback instead of the documented operational exit code 2; similar failures occur for scalar extensions and string numeric fields. Validate the JSON value types and raise ValueError before constructing the dataclasses.
Useful? React with 👍 / 👎.
Harden local and UNC path handling, assess NTFS and SMB ACLs, protect reports, add corporate preflight, and validate the 2.2.0rc1 package across Linux and Windows CI.
Compare canonical paths by file identity, set the executing user as the protected report owner, and cover the native ACL call contract.
1 participant
Motivation
Description
CPF_indevidosscript and introduced a newcpf_lgpdpackage with modules forclassification,extractors,scanner,risk,reporting,permissions,configurationand a CLI incli.py.redact_cpfsto mask CPF candidates in paths and output.docs/METODOLOGIA.md,SECURITY.md, an expandedREADME.md, and apyproject.tomlexposing thecpf-lgpdCLI entry point.tests/covering classification, scanning, extractors and CLI behavior, and added a GitHub Actions workflow.github/workflows/ci.ymlthat runs linting, unit tests and coverage.Testing
ruff check .as part of CI and the project includescoveragesettings inpyproject.toml.python -m unittest discover -v(covered bycoverage run -m unittest discover) and the repository includestests/test_classification.pyandtests/test_scanner.py.ruff, unit tests andcoverage report) were configured and ran successfully in the validation environment.Codex Task