Test `test_delete_question` is flaky

[styrix560]

The test sometimes fails an error like the following:

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/EvaP/EvaP/evap/staff/tests/test_views.py", line 3392, in test_delete_question
    self.assertQuerySetEqual(other_question.questionnaires.all(), [other_questionnaire])
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Lists differ: [<Questionnaire: ccBmyxSEHFbordyEnZcw>, <Questionnaire: VjfPCFfTrMXLkbvedxEg>] != [<Questionnaire: VjfPCFfTrMXLkbvedxEg>]

First differing element 0:
<Questionnaire: ccBmyxSEHFbordyEnZcw>
<Questionnaire: VjfPCFfTrMXLkbvedxEg>

First list contains 1 additional elements.
First extra element 1:
<Questionnaire: VjfPCFfTrMXLkbvedxEg>

- [<Questionnaire: ccBmyxSEHFbordyEnZcw>, <Questionnaire: VjfPCFfTrMXLkbvedxEg>]
+ [<Questionnaire: VjfPCFfTrMXLkbvedxEg>]

The used shuffle seed was 785091457

Runs:

• https://github.com/e-valuation/EvaP/actions/runs/23161647011/job/67290765485
• https://github.com/e-valuation/EvaP/actions/runs/22596857685/job/65468976117?pr=2587

[niklasmohrin]

See #2513 (comment)

[niklasmohrin]

May be fixed by #2675, but we are not sure

[niklasmohrin]

Failed today here: https://github.com/e-valuation/EvaP/actions/runs/23598704830/job/68722380622?pr=2684

[richardebeling]

Okay, what could be conditions that change between test runs that make this flaky and not reproducible locally?

• Date/Time -- should be completely irrelevant
• RNG seeding -- we could try setting a constant seed for python's random module in our test setup, maybe that helps with reproducing
  • model-bakery choices in foreign keys should be stable with our test setup and independent of the data it uses to fill out non-optional model fields, but maybe this is worth double checking.
  • We could also double-check our copy-on-write / deduplication logic for questions, but iirc, this should all be stable and not depend on the random strings that baker generated.
• Database race condition? We could adapt the test to retry fetching the value from the database if the first assertion fails to get some additional info. I would really like to know if the m2m row is actually still there in the database.
• Maybe race conditions with django's signals? But aren't they dispatched synchronously and deterministically?
• Django caching -- seems most plausible to me, but this should behave consistently and be reproducible.

[richardebeling]

It is an indeterminate ordering problem. The test deletes the first two question assignments of the four listed in the formset, but all are created with default order = -1 in the test. The order in which they appear on the page is undefined. The test passes when we get insert order and fails when postgres returns them in a different order. In that case, we delete other QuestionAssignment instances than the test intends, so our assertions do not match our state.

I'm able to repro by running the test repeatedly in a loop, baker seeds don't seem to matter, system load doesn't seem to matter, it just randomly fails eventually. At the failing assertion, this is our state:

(Pdb) p self.questionnaire.id
2064

(Pdb) p other_questionnaire.id
2065

(Pdb) p other_questionnaire.questions.all()
<QuerySet [<Question: Question object (3096)>]>

(Pdb) p self.questionnaire.questions.all()
<QuerySet [<Question: Question object (3095)>]>

(Pdb) p self.question
<Question: Question object (3095)>

(Pdb) p other_question
<Question: Question object (3096)>

(Pdb) pp form["question_assignments-0-id"].__dict__
{'_value': '4127',
 'attrs': {'class': ['form-control'], 'type': 'hidden'},
 'form': <Form id='questionnaire-form' />,
 'id': 'id_question_assignments-0-id',
 'name': 'question_assignments-0-id',
 'pos': 19,
 'tag': 'input'}

(Pdb) pp form["question_assignments-1-id"].__dict__
{'_value': '4129',
 'attrs': {'class': ['form-control'], 'type': 'hidden'},
 'form': <Form id='questionnaire-form' />,
 'id': 'id_question_assignments-1-id',
 'name': 'question_assignments-1-id',
 'pos': 28,
 'tag': 'input'}

(Pdb) pp [o.__dict__ for o in QuestionAssignment.objects.all()]
[{'_state': <django.db.models.base.ModelState object at 0x7fffef7727b0>,
  'id': 1,
  'order': -1,
  'question_id': 1,
  'questionnaire_id': 1},
 {'_state': <django.db.models.base.ModelState object at 0x7fffef7729f0>,
  'id': 4128,
  'order': -1,
  'question_id': 3096,
  'questionnaire_id': 2065},
 {'_state': <django.db.models.base.ModelState object at 0x7fffef772750>,
  'id': 4126,
  'order': -1,
  'question_id': 3095,
  'questionnaire_id': 2064}]