Skip to content

Fix cross-object skeleton mutex deadlock in CCF_Skeleton::BuildState - #576

Merged
themrdemonized merged 2 commits into
themrdemonized:all-in-one-vs2022-wpo-mtfrom
gwalls:fix/skeleton-mutex-deadlock-mt
Jun 21, 2026
Merged

themrdemonized merged 2 commits into
themrdemonized:all-in-one-vs2022-wpo-mtfrom
gwalls:fix/skeleton-mutex-deadlock-mt

Conversation

@gwalls

@gwalls gwalls commented Jun 18, 2026

Copy link
Copy Markdown

Problem

On the multithreaded bone-calc path, the game can hit a silent freeze (full hang, no crash, nothing in the log) in scenes with several co-located animated objects - e.g. a stacked NPC squad, a crowded chokepoint, or NPCs throwing bolts. It's intermittent and timing-dependent.

Root cause: cross-object AB-BA deadlock

UCalc_Mutex is one critical section per skeleton instance. Two threads end up each holding their object's lock and blocking on the other's:

  • CKinematics::CalculateBones takes the object's own UCalc_Mutex (SkeletonRigid.cpp:77) and holds it across the whole body — including the IK Update_Callback at the end.
  • That callback does a foot-IK ray-pick that reaches into a neighbour skeleton via CCF_Skeleton::BuildState, which took a blocking guard on the neighbour's UCalc_Mutex.

So with the main render thread computing bones for object X and the CalculateBonesThread worker computing object Y at the same time:

Thread Holds Blocks acquiring
Main render X.UCalc_Mutex (CalculateBones) Y.UCalc_Mutex (BuildState)
CalculateBonesThread Y.UCalc_Mutex (CalculateBones) X.UCalc_Mutex (BuildState)

Confirmed with a debugger capture during a live freeze — both threads parked identically entering the second skeleton's lock:

xrCriticalSection::Enter            <-- BLOCKED
CCF_Skeleton::BuildState            <-- locks the picked object's UCalc_Mutex
CCF_Skeleton::_RayQuery
CObjectSpace::_RayPick
ik_foot_collider::collide
CIKLimbsController::IKVisualCallback
CKinematics::CalculateBones         <-- already holds THIS object's UCalc_Mutex
  main:   CMissile::UpdateXForm -> CHudItem::renderable_Render -> CRender::Render
  worker: XRay::Engine::CalculateBonesThread
Screenshot 2026-06-17 230401

Fix

Make BuildState's neighbour-skeleton lock non-blocking (TryEnter): if the object is mid-recalc on another thread, skip the lock and read its one-frame-stale bone transforms instead of blocking. This removes the only cross-object blocking acquisition, so the cycle can't form — while keeping parallel bone calculation fully intact.

Worst case is a foot-IK ray-pick seeing one neighbour's bones a frame stale (no crash — the bone array is fixed-size; a torn matrix read self-corrects next frame). This is effectively the long-standing pre-lock behaviour, only in the rare contended case.

Alternatives considered

  • Revert to per-element LL_GetTransform_safed — doesn't help; it's still a blocking acquire of the neighbour's lock, so the deadlock persists. The fix has to be non-blocking.
  • Single global lock for all bone calc — removes the deadlock but serializes all skeleton calculation, defeating the point of the MT branch.
  • Release the object's own lock before the IK callback — also breaks the cycle, but un-serializes writes to the object's own foot bones (two threads can recalc the same object), a worse race.

Testing

Reproduced reliably on the current -mt build by rushing 2–3 unprotected squads through a shapeless anomaly field (especially acidic anomalies); also reproduced via the bolt path. With this change the freeze no longer reproduces, and foot IK looks correct (no visible popping).
I'm testing with my own fork of @Priler 's Anomaly NDA.

@gwalls
gwalls marked this pull request as ready for review June 18, 2026 13:52
@themrdemonized

Copy link
Copy Markdown
Owner

That try lock guard looks useful, please move that to where xrCriticalSectionGuard is placed, call it xrCriticalSectionTryGuard

@gwalls

gwalls commented Jun 20, 2026

Copy link
Copy Markdown
Author

That try lock guard looks useful, please move that to where xrCriticalSectionGuard is placed, call it xrCriticalSectionTryGuard

Okay, I moved it over to xrSyncronize.h. Lmk if that's what you had in mind.
I'm happy to trim down the comments if they're too verbose as well.

Thanks!

@themrdemonized
themrdemonized merged commit 8d5c10f into themrdemonized:all-in-one-vs2022-wpo-mt Jun 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants