Skip to content

Zaaktype/Documenttype urls migration tool#6268

Merged
sergei-maertens merged 4 commits into
mainfrom
feature/6262-zaaktype-documenttype-urls-migration-tool
May 21, 2026
Merged

Zaaktype/Documenttype urls migration tool#6268
sergei-maertens merged 4 commits into
mainfrom
feature/6262-zaaktype-documenttype-urls-migration-tool

Conversation

@sergei-maertens
Copy link
Copy Markdown
Member

@sergei-maertens sergei-maertens commented May 8, 2026

Closes #6262

Changes

  • Added management command to report/migrate forms to migrate
  • Handles migration of file components, zgw apis and objects api registration backends

[skip: e2e]

Checklist

Check off the items that are completed or not relevant.

  • Impact on features

    • Checked copying a form
    • Checked import/export of a form
    • Config checks in the configuration overview admin page
    • Checked new model fields are usable in the admin
    • Problem detection in the admin email digest is handled
  • Dockerfile/scripts

    • Updated the Dockerfile with the necessary scripts from the ./bin folder
  • Commit hygiene

    • Commit messages refer to the relevant Github issue
    • Commit messages explain the "why" of change, not the how
  • Documentation

    • Added documentation which describes the changes

@sergei-maertens sergei-maertens added the needs-backport Fix must be backported to stable release branch label May 8, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented May 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.10%. Comparing base (02e8439) to head (b156b91).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6268      +/-   ##
==========================================
+ Coverage   97.07%   97.10%   +0.03%     
==========================================
  Files         866      868       +2     
  Lines       32760    33117     +357     
  Branches     2982     3045      +63     
==========================================
+ Hits        31802    32159     +357     
  Misses        646      646              
  Partials      312      312              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sergei-maertens sergei-maertens force-pushed the feature/6262-zaaktype-documenttype-urls-migration-tool branch 10 times, most recently from 3f557d2 to 44759ea Compare May 19, 2026 21:31
@sergei-maertens
Copy link
Copy Markdown
Member Author

Checked import/export of a form

I don't think there's anything to do here - it's a lot of work to support this and there are no real benefits:

  • Samen Delen forms between different organisations -> probably a bad idea anyway to keep registration backend configuration in there, it will already have broken references to the API groups anyway
  • Within the same organization -> we'll remove the legacy functionality in 4.0, so probably best to just create a new export on a migrated environment

@sergei-maertens sergei-maertens marked this pull request as ready for review May 19, 2026 21:34
@sergei-maertens sergei-maertens marked this pull request as draft May 20, 2026 07:57
@sergei-maertens sergei-maertens force-pushed the feature/6262-zaaktype-documenttype-urls-migration-tool branch from 44759ea to a8e56bb Compare May 20, 2026 08:40
@sergei-maertens sergei-maertens marked this pull request as ready for review May 20, 2026 08:41
Copy link
Copy Markdown
Contributor

@annashamray annashamray left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested locally such cases:

  • file component:
    • correct legacy url
    • incorrect legacy url
  • registration backend
    • zgw - correct legacy url

All work as expected

I've reviewed code and looked at test cases and just have a couple minor remarks, which are nice to have

Comment thread src/openforms/zgw_urls_migrator.py
Comment thread src/openforms/zgw_urls_migrator.py
Comment thread src/openforms/zgw_urls_migrator.py Outdated
Comment thread src/openforms/zgw_urls_migrator.py
Comment thread src/openforms/tests/test_zgw_urls_migrator.py
The migrator migrates legacy URL references to document types and case
types to their 'pointers': the description of the document or the
identification of the case type, combined with the catalogue they
belong to (domain + rsin).

The migrator has a number of phases:

- file upload components processing (the shittiest / least complete)
- objects api registration backends
- zgw api registration backends
- objects API groups

**File upload components**

The registration options of a component may have an override for the
document type, stored as a URL reference. #6269 added support for
indirect references (catalogue + description). The migrator only
touches components that have a legacy URL and *no modern* config yet.
It looks up the catalogue + description of the document type and stores
it, but does not validate that this catalogue matches (all of) the
registration backend configuration, as that gets insanely complex and
will cause problems. The long term idea here is to *move* that config
from the component to the registration backend options itself anyway.

**Objects API registration backends**

Similarly here, there are three document types with a legacy and
modern config. We don't touch the modern config, if present. Legacy
URL references are resolved to their catalogue and description, and
the coherence of all resolved catalogues in the whole backend config
is checked (all documents must exist within the same catalog). If a
catalog is already configured in the options, it is taken into account.
If not, the catalog is written - we do not assume anything based on
the related API group and instead make the config explicit per form.

**ZGW API registration backends**

Similar to Objects API, except here we consider the case type and
document type references/legacy URLs. We don't touch modern config here
either, and again validate the coherence:

- doc type and case type must belong to the same catalogue, and match
  any pre-existing catalogue configuration
- the doc type must be related to the specified case type. This may be
  tricky based on validity dates of these objects and on which day the
  migration tool is ran, but failures during migration would probably
  also manifest at runtime

We don't touch the embedded object types configuration yet inside the
ZGW APIs - that's also legacy, but a modern alternative is not yet
available and honestly this migrator and commit message are huge
already.

**Objects API groups**

Also at the group level legacy URLs (and modern replacements) can be
present for the three relevant document types. These are updated the
same way like the backend options - don't touch modern config, and
resolve what is there to resolve and still missing.

A management command is added that runs in dry-run mode by default. It
just makes the changes and then rolls back the DB transaction - this
way you can dry-run and run it as a diagnostic tool to inform the
functional administrators which forms are problematic and need to be
fixed manually before the migration tool can run.
I'm sure I missed some edge cases, but we'll find out about those when
running the tool in diagnostic mode for real on test environments.
@sergei-maertens sergei-maertens force-pushed the feature/6262-zaaktype-documenttype-urls-migration-tool branch from a8e56bb to b156b91 Compare May 21, 2026 15:14
@sergei-maertens sergei-maertens merged commit 73a5129 into main May 21, 2026
31 checks passed
@sergei-maertens sergei-maertens deleted the feature/6262-zaaktype-documenttype-urls-migration-tool branch May 21, 2026 15:27
sergei-maertens added a commit that referenced this pull request May 21, 2026
The migrator migrates legacy URL references to document types and case
types to their 'pointers': the description of the document or the
identification of the case type, combined with the catalogue they
belong to (domain + rsin).

The migrator has a number of phases:

- file upload components processing (the shittiest / least complete)
- objects api registration backends
- zgw api registration backends
- objects API groups

**File upload components**

The registration options of a component may have an override for the
document type, stored as a URL reference. #6269 added support for
indirect references (catalogue + description). The migrator only
touches components that have a legacy URL and *no modern* config yet.
It looks up the catalogue + description of the document type and stores
it, but does not validate that this catalogue matches (all of) the
registration backend configuration, as that gets insanely complex and
will cause problems. The long term idea here is to *move* that config
from the component to the registration backend options itself anyway.

**Objects API registration backends**

Similarly here, there are three document types with a legacy and
modern config. We don't touch the modern config, if present. Legacy
URL references are resolved to their catalogue and description, and
the coherence of all resolved catalogues in the whole backend config
is checked (all documents must exist within the same catalog). If a
catalog is already configured in the options, it is taken into account.
If not, the catalog is written - we do not assume anything based on
the related API group and instead make the config explicit per form.

**ZGW API registration backends**

Similar to Objects API, except here we consider the case type and
document type references/legacy URLs. We don't touch modern config here
either, and again validate the coherence:

- doc type and case type must belong to the same catalogue, and match
  any pre-existing catalogue configuration
- the doc type must be related to the specified case type. This may be
  tricky based on validity dates of these objects and on which day the
  migration tool is ran, but failures during migration would probably
  also manifest at runtime

We don't touch the embedded object types configuration yet inside the
ZGW APIs - that's also legacy, but a modern alternative is not yet
available and honestly this migrator and commit message are huge
already.

**Objects API groups**

Also at the group level legacy URLs (and modern replacements) can be
present for the three relevant document types. These are updated the
same way like the backend options - don't touch modern config, and
resolve what is there to resolve and still missing.

A management command is added that runs in dry-run mode by default. It
just makes the changes and then rolls back the DB transaction - this
way you can dry-run and run it as a diagnostic tool to inform the
functional administrators which forms are problematic and need to be
fixed manually before the migration tool can run.

Backport-of: #6268
@sergei-maertens
Copy link
Copy Markdown
Member Author

Backport commit 0f6af59 to stable/3.5.x in PR #6266

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-backport Fix must be backported to stable release branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement file component migration and problem detection

2 participants