Validate updates to source while migrating

[masnax]

• Source data syncs are now permitted for instances in idle batches, however once a batch starts, manual and background syncing will be disabled for those VMs.

• A new Conflict status has been added for queue entries. This is similar to Blocked in that it is recoverable, but will not automatically self-resolve. A new POST /1.0/queue/{uuid}/:resolve endpoint exists to resolve these entries, though this does not bypass the prior validation so if a conflicted entry is resolved with no changes, it will simply become conflicted again. This is true for every case except include-expression matching, which will only trigger a conflict once.

• Batches have force_conflict_resolution as a new config option under defaults, which is false by default. This will attempt to ignore as many conflicts as possible. This can mean migrating VMs that no longer match the include expression of the batch, or migrating VMs with their initial placement and configuration, even after changes on the source.

• When starting a batch, if a prior source sync has caused a VM to no longer match the batch's include expression, a queue entry will still be created, but placed in the Conflict state

• Once all worker tasks are complete, a final fetch of target and source data is made for each VM, and changes to the instance record are committed. If a VM's properties have changed, its placement will be recomputed (including rerunning the placement scriptlet, even if rerun_scriptlets is false).

  • If the source VM has suddenly started running, it will be powered off again.
  • If the instance's UUID, name, or disks change, then the queue entry will become Errored.
  • If the instance's placement now expects a different target, project, or pools for any disk, then the queue entry will enter the Conflict state. User must then update the VM on the source, or modify the placement scriptlet to fix the issue and then mark the queue entry as resolved.
  • If the instance's placement indicates missing NICs on the target, the queue entry will enter the Conflict state and the user must update the source, target, network overrides, or placement scriptlet to fix the issue and then mark the queue entry as resolved.

---

Related changes:

• Sources now have a Syncing field which is true/false indicating whether an active sync is in progress for that source.

• Batch reset now has a --force flag which will bypass validation and allow for cleaning up target VMs that are well into migration. All queue records will be cleared, but target instances for queue records that have fully completed won't be deleted.

• Batch reset now properly starts the source VM for all deleted target VMs

• Queue cancel now has a --cleanup and --force flag. The former will perform the same cleanup as batch reset but for just this one queue entry. Force will allow for canceling and cleaning up queue entries in post-migration stages (except Finished).

• Background import verification status will not be reset for all disks if an instance's disk set changes in any way during a sync.

• Uses Incus' agent wait pattern, and repeatedly tries to start the instance if it didn't start initially for some reason.

• Changes some of the default CLI list columns to be more useful

• include expressions for instances/networks will now parse UUID types as uuid.String() internally. This means direct string comparison will work, but if someone is familiar with expr-lang and tries to directly call uuid.String(), the expression will not compile (it would end up as uuid.String().String() which is invalid).

• The placement scriptlet can now be modified while a batch is running, as long as no queue entries are in a state that precedes placement determination. This is always before instance creation, so can be bypassed by canceling those queue entries, updating the batch, and then restarting the canceled queue entries.

• Placement now includes running state, which is set according to whatever state the source VM was in when the batch started. This is to distinguish from the instance record's own running state, which will change because we perform a final source sync and commit the results after the source VM has been explicitly powered off.

• When completing migration and starting the target instance, if there are any errors, the queue entry will move to Conflict state so that last-minute tweaks can be made. The queue entry can then be marked resolved, which will run through the new source and target checks before updating the target instance again and trying to start it.

[masnax]

@presztak Some UI changes will be needed here:

• Include source syncing field in sources list

• Add a button for the POST /1.0/queue/{uuid}/:resolve endpoint, if the queue entry is in state Conflict

• Add option for force_conflict_resolution in batch defaults

• Maybe add some toggles for the force/cleanup query params on POST /1.0/queue/{uuid}/:cancel and the force query param on POST /1.0/batches/{name}/:reset, though these should probably go in the respective detail pages.