Skip to content

Conversation

@PRAteek-singHWY
Copy link
Contributor

@PRAteek-singHWY PRAteek-singHWY commented Dec 28, 2025

⚠️ This PR depends on:


Note for reviewers

The intended change in this PR is limited to
application/web/web_main.py (CSV import validation logic).

Any additional file diffs shown by GitHub are inherited from branch history / stacking and are not part of the validation change itself.

Please let me know if you’d prefer this PR to be rebased or split into a narrower diff.


Summary

This PR improves server-side validation for the MyOpenCRE CSV import endpoint, aligning it closely with the CSV format produced by the CRE catalogue export flow.

The intent is not to introduce strict or surprising validation rules, but to ensure that:

  • every CSV produced by the exporter can be safely imported
  • malformed or ambiguous input is rejected early with clear, structured errors
  • exporter artifacts (padding rows, empty rows) do not cause failures

What this PR does

File-level validation

  • Rejects missing, empty, non-CSV, or non–UTF-8 uploads
  • Returns consistent, structured error responses

Schema / header validation

  • Requires at least one CRE* column
  • Requires standard|name and standard|id
  • Rejects rows with more columns than the header (misaligned CSVs)

Row-level validation (export-compatible)

  • Completely empty rows are ignored (exported templates include padding rows)
  • Rows without any CRE values are ignored
  • CRE entries are validated only when present (<CRE-ID>|<Name>)
  • Malformed CRE entries return row-specific validation errors

No-op import guard

  • If a file contains no importable rows after filtering, the request returns success with no changes
  • This avoids confusing partial imports or unnecessary failures

What is intentionally ignored (by design)

  • Empty rows
  • Padding rows from exported templates
  • Rows without CRE mappings
  • Extra unused columns (as long as the CSV structure itself is valid)

This mirrors how production importers typically behave and allows exported files to be round-tripped without manual cleanup.


What is out of scope for this PR

  • Frontend error presentation / formatting
  • UX changes around errors

These will be handled in follow-up PRs to keep this change focused and reviewable.


Why this approach

The importer now accepts everything the exporter produces, ignores exporter padding rows, and enforces validation only where semantic meaning exists.

This keeps the import process resilient while still preventing invalid data from entering the system.


Dependency note

This PR is stacked on top of:

feat(myopencre): add CSV upload UI and wire to existing import endpoint (#664)

It should be reviewed and merged after that PR.


Feedback welcome

If any validation behavior is too permissive or too strict, or if there are exporter edge cases I may have missed, I’d really appreciate guidance and am happy to adjust.

@PRAteek-singHWY
Copy link
Contributor Author

Hi! @northdpole Sir I spent a few hours validating this against the CSV produced by the CRE catalogue exporter and tried to keep the importer behavior as close to that format as possible.

If any of the validation rules here feel too permissive, too strict, or misaligned with intended usage, I’d really appreciate feedback and guidance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant