Skip to content

fix: fail task and release machine on unexpected guest errors - #47

Open
itamarga wants to merge 1 commit into
masterfrom
fix/handle-guest-errors-cleanly
Open

fix: fail task and release machine on unexpected guest errors#47
itamarga wants to merge 1 commit into
masterfrom
fix/handle-guest-errors-cleanly

Conversation

@itamarga

@itamarga itamarga commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

When the guest agent returns an error during analysis — e.g. an HTTP 500 from /store while the host uploads the sample — the resulting requests.HTTPError escapes CAPE's error handling and orphans the task: the VM is left running, the machine stays locked, and the task sits in running forever. The analysis timeout never fires because it only applies once wait_for_completion() has begun, which this never reaches.

We hit this in production: three submissions whose sample filename collided with a pre-existing path in the guest's %TEMP% each 500'd on upload and left all three machines locked for 3 days until manually cleaned up.

Root cause (lib/cuckoo/core/analysis_manager.py)

  • machine_running() — the stop_machine() + release() cleanup sat after the try/except, so it only ran on the no-error path. The except only catches (CuckooMachineError, CuckooGuestCriticalTimeout); any other exception through yield skipped cleanup entirely → VM left running + machine locked.
  • launch_analysis() — only finalized task status for CuckooDeadMachine (→ pending) or clean completion (→ completed). Any other exception propagated out with the task left running.

Fix

  • machine_running(): always stop/release the machine via a finally block (extracted into _stop_and_release_machine()), skipping only the dead-machine case which already removes the machine.
  • launch_analysis(): on any unexpected exception, mark the task failed_analysis and unlock the machine.

Tests

Added regression tests in tests/test_analysis_manager.py:

  • test_machine_running_releases_machine_on_unexpected_error
  • test_launch_analysis_marks_failed_on_unexpected_error

tests/test_analysis_manager.py: 19 passed, 5 skipped (pre-existing data-file skips).

When the guest agent returns an error during analysis (e.g. an HTTP 500
while uploading the sample to /store), the resulting exception escaped
CAPE's error handling and orphaned the task:

- machine_running() only stopped/released the machine on the no-error path
  and in the dead-machine branch, so any other exception left the VM
  running and the machine locked forever.
- launch_analysis() only finalized the task status for CuckooDeadMachine
  (-> pending) or a clean completion (-> completed), so any other exception
  left the task stuck in "running" indefinitely. The analysis timeout never
  fired because it only applies once the guest wait has begun.

Fixes:
- machine_running(): always stop/release the machine via a finally block
  (extracted into _stop_and_release_machine()), skipping only the
  dead-machine case which already removes the machine.
- launch_analysis(): mark the task failed_analysis and unlock the machine
  on any unexpected exception.

Add regression tests for both paths.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant