Upgrade to django-privates 4.0+

.github/workflows/ci.yml

@@ -84,6 +84,7 @@ jobs:
 
           python src/manage.py compilemessages
           python src/manage.py collectstatic --noinput --link
+          mkdir private_media
           coverage run \
             --concurrency=multiprocessing \
             --parallel-mode \
@@ -113,6 +114,20 @@ jobs:
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
 
+      - name: Check that private-media tests are properly isolated
+        run: |
+          num_private_media_files=$(find private_media -type f | wc -l)
+          if [[ "$num_private_media_files" == "0" ]]; then
+            exit 0
+          else
+            echo "# Writes to private_media detected!" >> $GITHUB_STEP_SUMMARY
+            echo "$num_private_media_files were written into the private media storage, while tests should not leave any files." >> $GITHUB_STEP_SUMMARY
+            echo "" >> $GITHUB_STEP_SUMMARY
+            echo "Files:" >> $GITHUB_STEP_SUMMARY
+            find private_media -type f >> $GITHUB_STEP_SUMMARY
+            exit 1
+          fi
+
   tests-reverse:
     name: Run the Django test suite in reverse
     runs-on: ubuntu-latest
@@ -148,6 +163,7 @@ jobs:
         run: |
           python src/manage.py compilemessages
           python src/manage.py collectstatic --noinput --link
+          mkdir private_media
           src/manage.py test src \
             --force-color \
             --parallel 4 \

docs/developers/backend/file-uploads.rst

@@ -13,43 +13,62 @@ The following events happen during that process:
 
 - The user adds a file to the component:
 
-  - A ``POST`` request is made to ``/api/v1/formio/fileupload`` with the content of the file.
-  - If configured, the file is scanned for viruses (more details :ref:`here<configuration_general_virus_scan>`). In case
-    a virus is found, the file is not saved and the user receives an error alerting them that a virus was found in the file.
-  - An instance of the :class:`openforms.submissions.models.TemporaryFileUpload` model is created.
-  - The endpoint returns the url of the file ``/api/v1/submissions/files/<uuid>``, the file name and size. This information is added
-    to the Formio submission step data.
-  - The UUID of the :class:`openforms.submissions.models.TemporaryFileUpload` is added to the user session.
-  - The content of the file is saved to the disk. The file is placed in the private media directory (configured through
-    the ``PRIVATE_MEDIA_ROOT`` setting), within the ``temporary-uploads`` folder.
+  - A ``POST`` request is made to ``/api/v1/formio/fileupload`` with the content of the
+    file.
+  - If configured, the file is scanned for viruses (more details
+    :ref:`here<configuration_general_virus_scan>`). In case a virus is found, the file
+    is not saved and the user receives an error alerting them that a virus was found in
+    the file.
+  - An instance of the :class:`openforms.submissions.models.TemporaryFileUpload` model
+    is created. It is related to the submission currently in the session.
+  - The endpoint returns the url of the file ``/api/v1/submissions/files/<uuid>``, the
+    file name and size. This information is added to the Formio submission step data.
+  - The content of the file is saved to the disk. The file is placed in the private
+    media directory (configured through the ``STORAGES`` setting), within the
+    ``temporary-uploads`` folder.
 
 - The user saves the form step:
 
-  - An instance of :class:`openforms.submissions.models.SubmissionFileAttachment` is created (with a relation to the
-    :class:`openforms.submissions.models.TemporaryFileUpload`).
-  - The file gets copied to the ``submission-uploads`` folder (which is also in the private media directory).
+  - An instance of :class:`openforms.submissions.models.SubmissionFileAttachment` is
+    created (with a relation to the :class:`openforms.submissions.models.TemporaryFileUpload`).
+  - The file gets copied to the ``submission-uploads`` folder (which is also in the
+    private media directory).
 
 - The user completes the submission:
 
-  - The UUID of the :class:`openforms.submissions.models.TemporaryFileUpload` is removed from the session.
-  - The task ``cleanup_temporary_files_for`` deletes all :class:`openforms.submissions.models.TemporaryFileUpload`
-    associated with the submission that has been completed.
+  - The task ``cleanup_temporary_files_for`` deletes all
+    :class:`openforms.submissions.models.TemporaryFileUpload` associated with the
+    submission that has been completed.
 
 .. note::
 
     When instances of  :class:`openforms.submissions.models.TemporaryFileUpload` and
-    :class:`openforms.submissions.models.SubmissionFileAttachment` are deleted, the associated
-    files are removed from the file system (thanks to the :class:`openforms.utils.files.DeleteFileFieldFilesMixin` mixin).
+    :class:`openforms.submissions.models.SubmissionFileAttachment` are deleted, the
+    associated files are removed from the file system (thanks to the
+    :class:`openforms.utils.files.DeleteFileFieldFilesMixin` mixin).
 
+Access control
+--------------
+
+Users must have an active submission in the session before they can upload any
+(temporary) files. Additionally, the temporary file is related to the submission UUID
+in the session, and users can only download or delete files that belong to a submission
+present in their session data.
 
 Periodical clean up
 ===================
 
 There are Celery beat tasks that periodically clean up files:
 
-- The task ``cleanup_unclaimed_temporary_files`` cleans up any :class:`openforms.submissions.models.TemporaryFileUpload` which is not related to a
-  :class:`openforms.submissions.models.SubmissionFileAttachment`. This task runs once a day.
-- The task ``delete_submissions`` deletes any successful/incomplete/errored submission that are older than a
-  configured amount of time. This deletes the associated :class:`openforms.submissions.models.SubmissionFileAttachment`. This task runs once a day.
-- The task ``make_sensitive_data_anonymous`` clears any sensitive data from a submission. It also deletes any
-  :class:`openforms.submissions.models.SubmissionFileAttachment` related to the submission being cleaned. This task runs once a day.
+- The task ``cleanup_unclaimed_temporary_files`` cleans up any
+  :class:`openforms.submissions.models.TemporaryFileUpload` which is not related to a
+  :class:`openforms.submissions.models.SubmissionFileAttachment`. This task runs once a
+  day.
+- The task ``delete_submissions`` deletes any successful/incomplete/errored submission
+  that are older than a configured amount of time. This deletes the associated
+  :class:`openforms.submissions.models.SubmissionFileAttachment`. This task runs once a
+  day.
+- The task ``make_sensitive_data_anonymous`` clears any sensitive data from a
+  submission. It also deletes any
+  :class:`openforms.submissions.models.SubmissionFileAttachment` related to the
+  submission being cleaned. This task runs once a day.

requirements/base.txt

@@ -168,7 +168,7 @@ django-csp==4.0
     # via -r requirements/base.in
 django-csp-reports==1.10.0
     # via -r requirements/base.in
-django-digid-eherkenning==0.24.0
+django-digid-eherkenning==0.25.1
     # via -r requirements/base.in
 django-filter==25.2
     # via -r requirements/base.in
@@ -206,10 +206,11 @@ django-otp==1.7.0
     #   maykin-2fa
 django-phonenumber-field==8.4.0
     # via django-two-factor-auth
-django-privates==3.1.1
+django-privates==4.0.2
     # via
     #   -r requirements/base.in
     #   django-simple-certmanager
+    #   zgw-consumers
 django-redis==6.0.0
     # via -r requirements/base.in
 django-relativedelta==2.0.0
@@ -222,7 +223,7 @@ django-setup-configuration==0.11.0
     # via
     #   mozilla-django-oidc-db
     #   zgw-consumers
-django-simple-certmanager==2.5.0
+django-simple-certmanager==4.0.0
     # via
     #   -r requirements/base.in
     #   django-digid-eherkenning
@@ -709,7 +710,7 @@ xmltodict==1.0.3
     # via -r requirements/base.in
 zeep==4.3.2
     # via -r requirements/base.in
-zgw-consumers==1.2.0
+zgw-consumers==2.0.2
     # via -r requirements/base.in
 zipp==3.23.0
     # via importlib-metadata

requirements/ci.txt

@@ -259,7 +259,7 @@ django-csp-reports==1.10.0
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
-django-digid-eherkenning==0.24.0
+django-digid-eherkenning==0.25.1
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
@@ -326,11 +326,12 @@ django-phonenumber-field==8.4.0
     #   -c requirements/base.txt
     #   -r requirements/base.txt
     #   django-two-factor-auth
-django-privates==3.1.1
+django-privates==4.0.2
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
     #   django-simple-certmanager
+    #   zgw-consumers
 django-redis==6.0.0
     # via
     #   -c requirements/base.txt
@@ -354,7 +355,7 @@ django-setup-configuration==0.11.0
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
-django-simple-certmanager==2.5.0
+django-simple-certmanager==4.0.0
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
@@ -947,7 +948,6 @@ pyyaml==6.0.3
     #   pydantic-settings
     #   sphinxcontrib-mermaid
     #   vcrpy
-    #   zgw-consumers
 qrcode==8.2
     # via
     #   -c requirements/base.txt
@@ -992,9 +992,7 @@ requests-file==3.0.1
     #   -r requirements/base.txt
     #   zeep
 requests-mock==1.12.1
-    # via
-    #   -r requirements/test-tools.in
-    #   zgw-consumers
+    # via -r requirements/test-tools.in
 requests-oauthlib==2.0.0
     # via
     #   -c requirements/base.txt
@@ -1273,7 +1271,7 @@ zeep==4.3.2
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
-zgw-consumers==1.2.0
+zgw-consumers==2.0.2
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt

requirements/dev.txt

@@ -284,7 +284,7 @@ django-csp-reports==1.10.0
     #   -r requirements/ci.txt
 django-debug-toolbar==6.2.0
     # via -r requirements/dev.in
-django-digid-eherkenning==0.24.0
+django-digid-eherkenning==0.25.1
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
@@ -353,11 +353,12 @@ django-phonenumber-field==8.4.0
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
     #   django-two-factor-auth
-django-privates==3.1.1
+django-privates==4.0.2
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
     #   django-simple-certmanager
+    #   zgw-consumers
 django-redis==6.0.0
     # via
     #   -c requirements/ci.txt
@@ -385,7 +386,7 @@ django-setup-configuration==0.11.0
     #   -r requirements/ci.txt
 django-silk==5.4.3
     # via -r requirements/dev.in
-django-simple-certmanager==2.5.0
+django-simple-certmanager==4.0.0
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
@@ -1419,7 +1420,7 @@ zeep==4.3.2
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
-zgw-consumers==1.2.0
+zgw-consumers==2.0.2
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt

requirements/extensions.txt

@@ -250,7 +250,7 @@ django-csp-reports==1.10.0
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
-django-digid-eherkenning==0.24.0
+django-digid-eherkenning==0.25.1
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
@@ -317,11 +317,12 @@ django-phonenumber-field==8.4.0
     #   -c requirements/base.txt
     #   -r requirements/base.txt
     #   django-two-factor-auth
-django-privates==3.1.1
+django-privates==4.0.2
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
     #   django-simple-certmanager
+    #   zgw-consumers
 django-redis==6.0.0
     # via
     #   -c requirements/base.txt
@@ -345,7 +346,7 @@ django-setup-configuration==0.11.0
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
-django-simple-certmanager==2.5.0
+django-simple-certmanager==4.0.0
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
@@ -1156,7 +1157,7 @@ zeep==4.3.2
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt
-zgw-consumers==1.2.0
+zgw-consumers==2.0.2
     # via
     #   -c requirements/base.txt
     #   -r requirements/base.txt

requirements/type-checking.txt

@@ -276,7 +276,7 @@ django-csp-reports==1.10.0
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
-django-digid-eherkenning==0.24.0
+django-digid-eherkenning==0.25.1
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
@@ -343,11 +343,12 @@ django-phonenumber-field==8.4.0
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
     #   django-two-factor-auth
-django-privates==3.1.1
+django-privates==4.0.2
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
     #   django-simple-certmanager
+    #   zgw-consumers
 django-redis==6.0.0
     # via
     #   -c requirements/ci.txt
@@ -371,7 +372,7 @@ django-setup-configuration==0.11.0
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
-django-simple-certmanager==2.5.0
+django-simple-certmanager==4.0.0
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
@@ -1405,7 +1406,7 @@ zeep==4.3.2
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt
-zgw-consumers==1.2.0
+zgw-consumers==2.0.2
     # via
     #   -c requirements/ci.txt
     #   -r requirements/ci.txt

src/openapi.yaml

@@ -1,4 +1,4 @@
 openapi: 3.0.3
 info:
   title: Open Forms API
   version: 3.5.0
@@ -28,7 +28,7 @@
       description: Haal informatie op over de analytics-tools voor de frontend.
       summary: Analytics tools-configuratie
       tags:
       - analytics
       security:
       - tokenAuth: []
       - cookieAuth: []
@@ -50,7 +50,7 @@
             Content-Language:
               $ref: '#/components/headers/Content-Language'
   /api/v2/appointments/{submission_uuid}/cancel:
     post:
       operationId: appointments_cancel_create
       summary: Cancel an appointment
       parameters:
@@ -66,7 +66,7 @@
           type: string
         required: true
       tags:
       - appointments
       requestBody:
         content:
           application/json:
@@ -124,11 +124,11 @@
             Content-Language:
               $ref: '#/components/headers/Content-Language'
   /api/v2/appointments/appointments:
     post:
       operationId: appointments_appointments_create
       summary: Create an appointment
       tags:
       - appointments
       requestBody:
         content:
           application/json:
@@ -230,7 +230,7 @@
         explode: true
         style: form
       tags:
       - appointments
       responses:
         '200':
           content:
@@ -322,7 +322,7 @@
         explode: true
         style: form
       tags:
       - appointments
       responses:
         '200':
           content:
@@ -363,7 +363,7 @@
         explode: true
         style: form
       tags:
       - appointments
       responses:
         '200':
           content:
@@ -399,7 +399,7 @@
         explode: true
         style: form
       tags:
       - appointments
       responses:
         '200':
           content:
@@ -7030,9 +7030,17 @@
       - ztc
       - drc
       - brc
+      - rc
+      - kic
+      - oc
+      - ic
+      - pc
+      - ptc
+      - vrc
+      - tc
+      - bc
       - cmc
       - kc
-      - vrc
       - orc
       type: string
       description: |-
@@ -7042,9 +7050,17 @@
         * `ztc` - ZTC (Zaaktypen)
         * `drc` - DRC (Informatieobjecten)
         * `brc` - BRC (Besluiten)
-        * `cmc` - Contactmomenten API
-        * `kc` - Klanten API
+        * `rc` - Referentielijsten API
+        * `kic` - Klantinteracties API
+        * `oc` - Organisatie API
+        * `ic` - Identiteit API
+        * `pc` - Producten API
+        * `ptc` - Producttypen API
         * `vrc` - Verzoeken API
+        * `tc` - Taken API
+        * `bc` - Berichten API
+        * `cmc` - Contactmomenten API - (Deprecated)
+        * `kc` - Klanten API - (Deprecated)
         * `orc` - ORC (Overige)
     AppearanceEnum:
       enum: