Skip to content

Surface Errors from UnitOfWork actions instead of hanging silently#4697

Merged
smcvb merged 3 commits into
mainfrom
fix/psep-init-deadlock
Jul 1, 2026
Merged

Surface Errors from UnitOfWork actions instead of hanging silently#4697
smcvb merged 3 commits into
mainfrom
fix/psep-init-deadlock

Conversation

@MateuszNaKodach

@MateuszNaKodach MateuszNaKodach commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

UnitOfWork silently swallows Errors thrown by an invocation action and, as a result, hangs forever instead of failing. A single missing class on the classpath turned into a startup deadlock with no log line, no stack trace, and no exception anywhere — one of the hardest possible things to debug.

This PR makes the Unit of Work surface such failures (log + fail-fast) instead of hanging.

Why this matters

I personally lost several hours on this. Not because the problem was hard — once the underlying Error is in the logs, the cause is obvious and the fix takes a minute — but because nothing was logged at all. The app just hung with "connection refused," and I had to instrument the framework itself to find out an Error had been thrown and discarded.

That is exactly the kind of black hole we should protect our customers from. A missing/incompatible class on the classpath is one of the most common real-world failure modes, and it is precisely the situation where a clear log line saves hours. Right now the framework hides it. With this change the same situation produces an immediate, logged failure with the real stack trace — minutes, not hours.

The specific missing class below is incidental. It happened to be a Jackson annotation in my setup, but the framework defect applies to any ErrorNoClassDefFoundError, NoSuchMethodError, ExceptionInInitializerError, LinkageError, etc. — thrown by an action. Any of these will hang a Unit of Work the same way.

How it showed up (a real, very hard-to-debug case)

My app simply would not become reachable on startup. curl to its HTTP port returned Connection refused.

The single most telling detail: the last application log line was the event processor announcing it was starting — and then nothing:

... INFO ... o.a.m.e.p.s.p.PooledStreamingEventProcessor : Starting PooledStreamingEventProcessor [Automation_WhenCreatureRecruitedThenAddToArmy_Processor].
   (… no further output. ever.)

That was it. No next processor, no Tomcat started, no Started …Application, no stack trace, no error — the log just stopped mid-startup. Concretely, there was:

  • no exception in the logs,
  • no APPLICATION FAILED TO START,
  • Tomcat initialized with port 8080 … but never Tomcat started and never Started …Application,
  • a thread dump showing the main thread simply parked:
"restartedMain" WAITING (parking)
    at java.util.concurrent.CompletableFuture$Signaller.block
    at java.util.concurrent.CompletableFuture.waitingGet
    at o.a.extension.spring.config.SpringLifecycleStartHandler.start(SpringLifecycleStartHandler.java:62)
    at o.s.context.support.DefaultLifecycleProcessor...startBeans
    ...

That stack tells you a lifecycle bean's start future never completes — but gives zero indication of why. The web server is bound in a later SmartLifecycle phase than the event processors, so a PooledStreamingEventProcessor whose start future never completes blocks the whole context from finishing startup, and the port is never bound. The only symptom the developer sees is "connection refused."

It took instrumenting the framework (logging each Unit-of-Work phase transition and the state of the invocation future) to discover that an Error had been thrown and lost. Once surfaced, the cause was obvious in seconds. That gap between "impossible to find" and "obvious once logged" is exactly what this PR closes.

My setup

  • App: HeroesOfDomainDrivenDesign · 0f4742d (a DDD/event-sourcing sample on Axon 5)
  • Axon Framework 5.2.0-SNAPSHOT, Axon Server 2026.0.0 (DCB), JPA TokenStore, Spring Boot 3.5.x
  • A mixed Jackson 2.x / Jackson 3.x classpath: axon-conversion uses Jackson 3 (tools.jackson:jackson-databind:3.2.0), while the Jackson 2.x line (jackson-databind:2.21.2) transitively pinned jackson-annotations:2.21.
  • Jackson 3.2.0 references com.fasterxml.jackson.annotation.JsonApplyView, which only exists in jackson-annotations:2.22. So at runtime, the first token-store serialization threw NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonApplyView.

The dependency mismatch was my problem to fix (bump jackson-annotations to 2.22). But the framework should have failed loudly on it rather than hanging. Again — this could be any missing/incompatible class from any dependency; the framework behavior is what this PR addresses.

Root cause

The startup path is:

PooledStreamingEventProcessor.start()
  -> unitOfWork.executeWithResult(tokenStore::retrieveStorageIdentifier)
       -> JpaTokenStore serializes the config token via the (Jackson 3) converter
            -> throws NoClassDefFoundError  (an Error, not an Exception)

Inside UnitOfWork:

private <R> CompletableFuture<R> safe(Callable<CompletableFuture<R>> action) {
    try {
        ...
        return result;
    } catch (Exception e) {              // <-- only Exception
        return CompletableFuture.failedFuture(e);
    }
}

A NoClassDefFoundError is an Error, so it is not caught here. It escapes safe(...) before .whenComplete(alsoComplete(result)) is ever attached, so the result future is never completed (neither normally nor exceptionally).

Then:

public <R> CompletableFuture<R> executeWithResult(Function<...> action) {
    CompletableFuture<R> result = new CompletableFuture<>();
    onInvocation(pc -> safe(() -> action.apply(pc)).whenComplete(alsoComplete(result)));
    return execute().thenCombine(result, (executeResult, invocationResult) -> invocationResult);
}

thenCombine only fires when both execute() and result complete. execute() (the commit) does complete — exceptionally — but result never does, so the combined future hangs forever. The processor's start future never settles, and the application deadlocks during startup.

So two things conspired: (1) the Error was swallowed/lost, and (2) the future combinator waited on a future that could never complete.

The fix

1. Don't swallow Errors; log them. safe() now catches Throwable, completes the future exceptionally (so it propagates to the caller), and logs Errors loudly — because they indicate severe, easily-missed problems. Ordinary business Exceptions are intentionally not logged here (hot path; the caller already observes them via the failed future).

2. Fail fast on commit failure. executeWithResult uses thenCompose instead of thenCombine, so a failed commit propagates immediately rather than blocking on a result future that may never settle. On a successful commit the invocation phase has already run, so result is complete and its value/failure is returned as before.

Why thenCompose is the right combinator here (not just defense-in-depth)

thenCombine is meant for two independent futures running in parallel: it completes once both finish. But execute() and result are not independent — result is a byproduct of execute(). execute() (= commit()) drives the phases, and result is completed mid-way, during the INVOCATION phase, via safe(action).whenComplete(alsoComplete(result)). There is a single pipeline (the commit); nothing runs concurrently. So thenCombine here just means "wait for the later of the two (normally execute()) and yield result" — which is precisely what thenCompose(ignored -> result) expresses, but faithfully.

thenCompose preserves both original guarantees:

  1. it still waits for the full commit before returning (the lambda runs only after execute() completes), and
  2. it still returns the action's value (result, settled by then).

…and it is strictly more correct, because result is only ever completed inside the INVOCATION phase. So result never settles whenever that phase doesn't run to completion — and thenCombine then hangs forever. There are two such cases:

thenCompose short-circuits on execute()'s failure in both cases, so the Unit of Work fails fast regardless of where in the commit the failure occurs. The happy path is byte-for-byte equivalent.

Before / after

Before After
Error thrown by an invocation action swallowed; result never completes logged at ERROR; future completes exceptionally
executeWithResult future hangs forever fails fast with the original cause
App symptom silent startup deadlock, "connection refused" APPLICATION FAILED TO START with the real stack trace

Notes for reviewers

  • Catching Throwable means severe Errors (e.g. OutOfMemoryError) also become failed futures rather than propagating on the executing thread. That is the intended behavior for this boundary: the action's outcome must be observable via the returned future, and an Error lost on an arbitrary worker/callback thread is strictly worse than one surfaced to the caller.
  • The thenCombinethenCompose change is behavioral only when the commit fails (it no longer blocks on a result that may never settle — including when the commit fails before the INVOCATION phase runs); the happy path is byte-for-byte unchanged. See "Why thenCompose is the right combinator here" above.
  • I verified the fix against the failing app: with these changes the same dependency mismatch now produces an immediate, logged NoClassDefFoundError startup failure instead of a hang. Happy-path startup is unaffected.

@MateuszNaKodach MateuszNaKodach requested a review from a team as a code owner June 30, 2026 10:55
@MateuszNaKodach MateuszNaKodach requested review from hjohn, jangalinski and zambrovski and removed request for a team June 30, 2026 10:55
@MateuszNaKodach MateuszNaKodach changed the title fix(messaging): surface Errors from UnitOfWork actions instead of hanging silently refactor(messaging): surface Errors from UnitOfWork actions instead of hanging silently Jun 30, 2026
@MateuszNaKodach MateuszNaKodach self-assigned this Jun 30, 2026
@MateuszNaKodach MateuszNaKodach force-pushed the fix/psep-init-deadlock branch from fd7d05f to 320e380 Compare June 30, 2026 11:02
…ging silently

UnitOfWork#safe only caught Exception, so any Error (e.g. NoClassDefFoundError)
thrown synchronously by an invocation action escaped uncaught. The dependent
`result` future was then never completed, and because executeWithResult joined
the commit and result futures with thenCombine - which only fires once *both*
complete - the Unit of Work hung forever instead of failing.

In practice this turns a hard, fail-fast classpath/dependency problem into a
silent startup deadlock with no log, no stack trace and no exception: the only
observable symptom is that a downstream lifecycle bean (e.g. the web server)
never starts.

Changes:

- safe() now catches Throwable and logs Errors before completing the future
  exceptionally, so these severe-but-easily-missed problems are visible and
  propagate to the caller instead of being swallowed.

- executeWithResult() uses thenCompose rather than thenCombine. thenCombine only
  fires once *both* the commit and result futures complete, so a failed commit
  would hang forever whenever `result` never settles - which happens whenever the
  INVOCATION phase does not run to completion (an action throwing an Error, or the
  commit failing in an earlier phase such as begin-transaction). thenCompose
  propagates a failed commit immediately; on a successful commit the invocation
  phase has already run, so `result` is complete and its value (or failure) is
  returned, leaving the happy path byte-for-byte unchanged.
@MateuszNaKodach MateuszNaKodach force-pushed the fix/psep-init-deadlock branch from 320e380 to 72cdb67 Compare June 30, 2026 11:03
@MateuszNaKodach MateuszNaKodach added Priority 2: Should High priority. Ideally, these issues are part of the release they’re assigned to. Type: Enhancement Use to signal an issue enhances an already existing feature of the project. labels Jun 30, 2026
@MateuszNaKodach MateuszNaKodach added this to the Release 5.2.0 milestone Jun 30, 2026
Adds two regression tests for executeWithResult:

- errorsThrownInInvocationAreReturnedInFutureInsteadOfHangingTheUnitOfWork:
  an Error (e.g. NoClassDefFoundError) thrown by the invocation must fail the
  returned future instead of leaving it pending forever.
- executeWithResultFailsFastWhenCommitFailsBeforeInvocationRuns: when the commit
  fails before the INVOCATION phase runs, the result future never settles, so the
  returned future must fail fast (thenCompose) rather than block (thenCombine).

Both fail against the pre-fix code (the future never completes) and pass with it.
@MateuszNaKodach MateuszNaKodach requested a review from smcvb June 30, 2026 20:36
@MateuszNaKodach MateuszNaKodach assigned abuijze and unassigned abuijze Jun 30, 2026

@smcvb smcvb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch @MateuszNaKodach! But I got some nitty comments. Given the gravity of the Unit of Work, I'll request changes, even though they're small.

@smcvb smcvb changed the title refactor(messaging): surface Errors from UnitOfWork actions instead of hanging silently Surface Errors from UnitOfWork actions instead of hanging silently Jul 1, 2026
Apply reviewer suggestions from PR #4697:
- log escaped Throwables with an `instanceof Error` check, aligning the
  condition with the javadoc and log message that speak only of Error
- name the discarded `thenCompose` result `ignored` for clarity
- reword test comments to describe the observable contract rather than
  internal mechanics (thenCombine/#safe/'result' future)
- assert on the failed future with AssertJ `hasRootCauseInstanceOf`,
  dropping the hand-rolled unwrap helper
@MateuszNaKodach MateuszNaKodach requested a review from smcvb July 1, 2026 09:55

@smcvb smcvb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My concerns have been addressed, hence I'm approving this pull request.

@smcvb smcvb merged commit d89f515 into main Jul 1, 2026
7 checks passed
@smcvb smcvb deleted the fix/psep-init-deadlock branch July 1, 2026 10:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Priority 2: Should High priority. Ideally, these issues are part of the release they’re assigned to. Type: Enhancement Use to signal an issue enhances an already existing feature of the project.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create 5 minute introduction to CQRS / Axon

3 participants