Issue Type
🛠️ Task
Affected Component
openrelik-worker-common
Description
When a worker wants to mount a disk we need to make (and document) that NODENAME is set in the docker-compose configuration across all (mounting) workers running on the same host. This to ensure that the redis lock key is the same across all containers running on the same host.
If NODENAME is NOT set and a worker tries to mount it can create race condititions that are hard to debug and can generate errors like below:
[2025-05-30 07:19:17,283: INFO/ForkPoolWorker-4] Redis lock succesfully set: cf3f5472d882-/dev/nbd0 for cf3f5472d882-/dev/nbd0
[2025-05-30 07:19:17,457: INFO/ForkPoolWorker-4] qemu-nbd: success creating /dev/nbd0 for /usr/share/openrelik/data/artifacts/4246e4aee76a474db828fa6e934b94e5/f6f1a4386a7d4b28a1f60c0398b373cd.qcow2
[2025-05-30 07:19:17,691: ERROR/ForkPoolWorker-4] fdisk: failed probing /dev/nbd0 for /usr/share/openrelik/data/artifacts/4246e4aee76a474db828fa6e934b94e5/f6f1a4386a7d4b28a1f60c0398b373cd.qcow2: fdisk: cannot open /dev/nbd0: Invalid argument
[2025-05-30 07:19:17,691: INFO/ForkPoolWorker-4] Error: Error fdisk: failed probing: fdisk: cannot open /dev/nbd0: Invalid argument
In above case it looked like a succesful redis lock was acquired for /dev/nbd0 with key cf3f5472d882-/dev/nbd0. In this case NODENAME is not set and the first part of the key is the hostname IN the container. As other containers also try to mount and have a different hostname this will succeed while in reality the device is already in use by another worker. The first part of the key needs to be the same for workers running on the same server/node.
Relevant Logs
Issue Type
🛠️ Task
Affected Component
openrelik-worker-common
Description
When a worker wants to mount a disk we need to make (and document) that NODENAME is set in the docker-compose configuration across all (mounting) workers running on the same host. This to ensure that the redis lock key is the same across all containers running on the same host.
If NODENAME is NOT set and a worker tries to mount it can create race condititions that are hard to debug and can generate errors like below:
In above case it looked like a succesful redis lock was acquired for /dev/nbd0 with key
cf3f5472d882-/dev/nbd0. In this case NODENAME is not set and the first part of the key is the hostname IN the container. As other containers also try to mount and have a different hostname this will succeed while in reality the device is already in use by another worker. The first part of the key needs to be the same for workers running on the same server/node.Relevant Logs