Hot Reload / Dynamic Rule Updates

[asulwer]

Hot Reload / Dynamic Rule Updates

Summary

Update a running workflow's rules without restarting the process: watch a source
file, recompile on change, and atomically swap the live compiled workflow
while in-flight executions finish safely against the old version.

What exists today (strong foundation)

• ExpressionAssemblyLoadContext (RoslynRules/Compiler/ExpressionAssemblyLoadContext.cs)
  is already collectible (isCollectible: true) with an Unload path — so old
  compiled assemblies can be released after a swap.
• Workflows are immutable after Compile() (EnsureNotCompiled guards), which
  makes reference-swap the natural, lock-free update mechanism.
• SnapshotManager already does CompileAndSnapshot / RestoreAndCompile and
  file save/load, and CompiledWorkflow.Compile(...) produces a fresh compiled unit.

Gap

There is no component that watches a source, recompiles, validates, and swaps the
active CompiledWorkflow — and importantly, verifies the previous load context is
actually unloaded so repeated reloads don't leak.

Proposed design

A WorkflowReloader that holds a volatile reference to the current
CompiledWorkflow and swaps it via Interlocked.Exchange:

1. FileSystemWatcher on the source path → debounce (coalesce rapid saves,
   handle editor write-rename).
2. Load + compile off the watched source into a new CompiledWorkflow
   (via a caller-supplied loader delegate; see open questions).
3. Validate() the new workflow. On any failure, keep serving the last-good
   version and raise a failure event — never swap in a broken workflow.
4. Interlocked.Exchange the current reference. New Execute calls get the new
   workflow; in-flight calls keep running against the old one (it's immutable).
5. Drop the old reference and Unload() its load context; verify collection so
   reloads are leak-free over time.

Concurrency semantics

• reloader.Current is always a fully-compiled, validated workflow.
• A reload never mutates a workflow in place; it publishes a new instance.
• In-flight executions complete deterministically against the snapshot they
  started with.

Public API sketch

var reloader = WorkflowReloader.Watch(
    path: "rules.workflow.json",
    parameters: new[] { new RuleParameter("customer", typeof(Customer)) },
    loader: File.ReadAllText,          // source -> text; compile handled internally
    debounce: TimeSpan.FromMilliseconds(250));

reloader.Reloaded += (_, e) =>
    Console.WriteLine($"Swapped to v{e.Workflow.Version} at {e.TimestampUtc}");
reloader.ReloadFailed += (_, e) =>
    Console.WriteLine($"Kept last-good; reload failed: {e.Error.Message}");

// Always safe to read; returns the current validated workflow.
var results = reloader.Current.Execute(
    new RuleParameter("customer", typeof(Customer), customer));

reloader.ReloadNow();   // force a manual reload
reloader.Dispose();     // stop watching, unload contexts

public sealed class WorkflowReloader : IDisposable
{
    public CompiledWorkflow Current { get; }          // volatile read, never null after first compile
    public event EventHandler<WorkflowReloadedEventArgs>? Reloaded;
    public event EventHandler<WorkflowReloadFailedEventArgs>? ReloadFailed;
    public void ReloadNow();
    public void Dispose();
}

Acceptance criteria

• Editing the watched file swaps Current to the new compiled workflow
  without a process restart.
• An invalid source (syntax/semantic/validation error) does not swap;
  Current keeps serving the last-good workflow and ReloadFailed fires.
• In-flight executions started before a swap complete against the old
  workflow and return correct results.
• Rapid successive saves are debounced into a single recompile.
• No leak: after N reloads (e.g. 100) the old ExpressionAssemblyLoadContexts
  are unloaded/collected — assert via a weak-reference / GC test.
• Thread-safety test: concurrent Execute on Current during a swap.
• Demo sample showing live editing of a rules file.

Non-goals

• No distributed/multi-node propagation (single-process reload only).
• No partial/per-rule patching — the whole workflow is recompiled and swapped.
• No hot reload of the host application's own types (only rule source changes).

Open questions

• Source format for v1: JSON Workflow (aligns with the JSON-first story),
  a WorkflowSnapshot, or a caller-provided Func<string, Workflow>? A loader
  delegate keeps the reloader format-agnostic.
• Should parameters be allowed to change across reloads, or must the parameter
  shape stay stable (recommend: stable in v1, error otherwise)?
• Expose a bool VerifyUnload diagnostic hook for the leak test, or keep internal?