[HOLD https://github.com/Expensify/Mobile-Expensify/pull/13970] [$250] [Sentry: APP-7FJ] HybridApp iOS Fatal App Hang at main

@ZhenjaHorbach

Sentry

https://expensify.sentry.io/issues/APP-7FJ

Impact (snapshot at filing)

Users (last 14d): ~21
Platform: iOS, hybrid_app (HybridApp)
Mechanism: AppHang, fatal — watchdog termination after 2000ms main-thread block

Stack trace

Sentry's only first-party-resolvable frame is main — the body of the hang is below symbolication. We need a re-symbolicated event before we can point at a code site.

First investigation step

Re-symbolicate a recent event in Sentry against the matching dSYM (the dSYM for the corresponding release should already be uploaded). Once frames resolve, group by the deepest first-party frame and re-triage.

Why this is filed separately from APP-7CX

APP-7CX has a clear culprit (+[YAPLJS callFunction:args:] re-entrancy) and is already tracked in App#91229. APP-7FJ's only resolvable frame is main, so it could be any of:

A different JS path that also blocks the main thread for 2000ms+
A native UIKit / dispatch deadlock unrelated to JS
A separate library re-entrancy issue

Keeping it as its own issue until we can see the deeper stack.

Reproduction

Unknown.

Issue Analysis

⚠️ Preliminary analysis (medium confidence) — the root cause cannot be confirmed until Sentry events are re-symbolicated with the correct dSYM.

Root Cause: Unknown — the Sentry stack trace only resolves to main, so the specific code path causing the 2000ms+ main-thread block is not yet identifiable. However, a codebase audit reveals several candidate main-thread blocking patterns in the HybridApp iOS build that could individually or cumulatively exceed the 2000ms watchdog threshold.

Recommended Next Step: Re-symbolicate a recent APP-7FJ event against the matching dSYM. Once frames resolve, match the deepest first-party frame against the candidates below to confirm the culprit.

Candidate main-thread blocking sites (ranked by risk)

1. `RCTBootSplash.constantsToExport` — synchronous main-queue dispatch during module init

RCTBootSplash.mm:143 calls RCTUnsafeExecuteOnMainQueueSync() to read dark-mode state. If constantsToExport is called from the JS thread during module initialization while the main thread is already busy (e.g., during didFinishLaunchingWithOptions), this creates a deadlock-prone synchronous cross-thread dispatch. Combined with Fabric/TurboModule initialization, this is the highest-risk candidate for a 2000ms+ hang.

Additionally, the BootSplash module explicitly routes all its methods to the main queue (line 20-22).

2. `AppDelegate.didFinishLaunchingWithOptions` — synchronous startup chain

AppDelegate.swift:23-73 runs several heavy operations synchronously on the main thread:

React Native startup (factory.startReactNative()) — line 34
Firebase init (FirebaseApp.configure()) — line 40
Storyboard instantiation (RCTBootSplash.initWithStoryboard) — line 49
Airship LiveActivity setup — lines 64-70
Multiple UserDefaults reads/writes — lines 25, 56-58

On older devices, these operations cumulatively could approach or exceed 2000ms.

3. Distinction from APP-7CX (#91229)

APP-7CX has a clear +[YAPLJS callFunction:args:] re-entrancy culprit. APP-7FJ's main-only trace suggests a different hang mechanism — likely startup-phase blocking rather than mid-session JS callback re-entrancy. The two issues should remain separate until symbolication proves otherwise.

Possibly related issues

Expensify/App#91292 — APP-7B2: HybridApp iOS fatal hang in Fabric ComponentDescriptorRegistry (same platform, different mechanism)
Expensify/App#91229 — APP-7CX: HybridApp iOS fatal hang in YAPLJS callFunction (sibling issue, already noted in this issue)

Actionable next steps

Re-symbolicate a recent APP-7FJ event in Sentry against the matching dSYM — this is the blocker
Once frames resolve, check if the deepest frame matches any of the candidates above
If the hang is in constantsToExport / RCTUnsafeExecuteOnMainQueueSync, the fix is to make RCTBootSplash return constants lazily or move dark-mode detection off the synchronous path
If the hang is in didFinishLaunchingWithOptions, consider deferring Firebase and Airship setup to a background queue or post-launch callback

Next Steps: Reply with @MelvinBot <your feedback> to refine this analysis.

@Krishna2323

Triggered auto assignment to Contributor-plus team member for initial proposal review - @Krishna2323 (External)

Job added to Upwork: https://www.upwork.com/jobs/~022057427688022436891

🚨 Edited by proposal-police: This proposal was edited at 2026-05-21 12:00:31 UTC.

Proposal

What is the root cause of that problem?

The issue's stated problem ("only main resolves; we can't see what the main thread is blocked on") is at heart a symbolication-pipeline gap, not a missing-symbol-on-this-specific-event problem. There are two chains; the first is the actionable one in this repo, the second is the most likely candidate for the underlying hang once we can read the stack.

Chain 1 — Sentry is not getting dSYMs for HybridApp iOS releases that come from the Rock remote cache.

The Mobile-Expensify iOS build is delegated to callstackincubator/ios (Rock) in .github/workflows/buildIOS.yml#L230-L247. Rock is a remote-cache-first build system — when the cache hits, the IPA is fetched without re-running the Xcode build phases. The workflow itself acknowledges this in buildIOS.yml#L272-L275 ("::warning::No .ipa found locally (expected for Rock remote-cache hits)").

Sentry-side symbol upload for iOS is conventionally driven by the @sentry/react-native CocoaPods build phase. That phase is sensitive enough to local layout that we already patch it in patches/sentry-react-native/@sentry+react-native+8.2.0.patch#L1-L33 — but a patched Ruby script only matters when Xcode actually re-archives. On a Rock cache hit, no build phase runs, so no dSYM upload happens.

After the build step, the workflow only zips the dSYMs out of the cached archive and uploads them to GitHub Actions artifacts (not Sentry) — buildIOS.yml#L289-L313. The deploy stage downloads that artifact (deploy.yml#L337-L346) and hands it straight to TestFlight via fastlane/Fastfile#L509-L534 — which calls upload_to_testflight and nothing else. fastlane-plugin-sentry isn't even installed (fastlane/Pluginfile#L1-L6), so upload_symbols_to_sentry is not callable. The only sentry-cli invocation in this pipeline is buildIOS.yml#L248-L262 — sentry-cli build upload, which is for bundle size analysis (note --build-configuration Release), not for symbolication. sentry-cli debug-files upload is never called anywhere in this repo.

That asymmetry is exactly why we see what we see in Sentry: APP-7CX was captured against a build whose archive happened locally and ran the Sentry CocoaPods phase — its top frames symbolicate (+[YAPLJS callFunction:args:], -[BaseViewController onBarButtonItemSelected:], …). APP-7FJ was captured against a cache-hit build whose dSYM never made it to Sentry, so the only frame that resolves is main, which comes from system symbols.

Chain 2 — what the hang itself is most likely doing (mitigation hypothesis, not the actionable layer).

APP-7FJ's metadata (iOS HybridApp, fatal AppHang, ≥2000 ms main-thread block, ~21 users) matches APP-7CX's profile (same platform, same mechanism). APP-7CX's known stack already explains the family — re-entrant +[YAPLJS callFunction:args:] triggered from inside a JS log handler invoking LogSyncerManager.sendLogPacket, which calls JSC again on the same thread and deadlocks on JSC::JSLock::grabAllLocks. That code lives in Mobile-Expensify and was partially fixed in Mobile-Expensify #13930 (which closed #86334 but did not stop APP-7CX, per its current status).

The App-side amplifier worth noting is src/libs/Log.ts#L80-L82: serverLoggingCallback synchronously fans every log packet into forwardLogsToSentry(params.logPacket) inside the same JS frame that triggered the original log. Combined with src/setup/telemetry/setupSentry.ts#L13-L42 (tracesSampleRate: 1.0, enableUserInteractionTracing: true, enableLogs: true, profiling on a trace lifecycle), the inline work Sentry does inside a log call is non-trivial — when a native-to-JS-to-Log loop fires on the main thread, this widens the window in which the YAPLJS re-entry can hit. I am intentionally not claiming this is the cause of APP-7FJ specifically; without symbolicated frames it is a hypothesis, and the proposal's primary lever is Chain 1 so we can confirm or rule it out.

What changes do you think we should make in order to solve the problem?

The fix layer is CI/build, in this repo, because that is the only place that can guarantee Sentry has dSYMs for every shipped HybridApp build regardless of whether Rock served the binary from cache. Symbol-bearing builds become diagnosable; everything downstream (closing APP-7FJ as duplicate of APP-7CX, or splitting it into a new family) flows from there.

Add an unconditional sentry-cli debug-files upload step in buildIOS.yml, immediately after the existing zip-dsyms step. It should walk .rock/cache/ios/archive/Expensify.xcarchive/dSYMs/*.dSYM, upload each to --org expensify --project app, and pass --include-sources so source context is available for native Obj-C/Swift frames. Run it for the Release variant. Re-use the SENTRY_AUTH_TOKEN secret already wired into the Rock build env at buildIOS.yml#L234. Pass --bundle-id <CFBundleIdentifier> so uploads are associated with the right Sentry release that matches the release value in setupSentry.ts#L24. Do not use continue-on-error for production — a missing symbol upload should fail the build, because a silent failure is what produced APP-7FJ.
Belt-and-braces in deploy.yml: add the same sentry-cli debug-files upload step after the dSYM download (the artifact already lives at $DSYM_PATH) and before upload_testflight_hybrid runs. If the build-stage upload races/fails, the deploy stage still gets them in. Also flip the dSYM-missing check at deploy.yml#L360-L362 from a ::warning:: to a hard exit 1 on refs/heads/production — currently a lost dSYM artifact only logs and the deploy continues, which is the exact failure mode that lets APP-7FJ-class events persist.
Once dSYMs start flowing, reprocess recent APP-7FJ events (Sentry → project Debug Files → Reprocess, or the per-issue action). Re-group by deepest first-party frame. Three outcomes:
- Stack collapses into APP-7CX's YAPLJS → LogSyncerManager → YAPLJS pattern → close APP-7FJ as a duplicate of [$250] [Sentry: APP-7CX] HybridApp iOS Fatal App Hang in YAPLJS callFunction during onBarButtonItemSelected (still occurring after #86334 closed) #91229.
- Distinct JS-re-entry callsite (not onBarButtonItemSelected:) → fold into [$250] [Sentry: APP-7CX] HybridApp iOS Fatal App Hang in YAPLJS callFunction during onBarButtonItemSelected (still occurring after #86334 closed) #91229's fix work in Mobile-Expensify, since the fix that closed Fatal App Hang Fully Blocked: The user or the OS watchdog terminated your app while it blocked the main thread for at least 2000 ms. #86334 demonstrably didn't catch every callsite.
- Non-JS deadlock (e.g., UIKit/libdispatch/Pusher native handler on the main queue) → re-file as a separate issue with the new evidence.

Edge cases:

PR/Adhoc builds: keep continue-on-error: true so PR builds aren't blocked by Sentry outages; the hard fail only applies to release deploys.
Standalone iOS (the NewExpensify AppDelegate in ios/AppDelegate.swift): out of scope for this fix because HybridApp is what's affected; the standalone pipeline should mirror the change once verified on HybridApp.
Cache cold-miss: dSYMs are produced fresh into the xcarchive at the existing path, so the new upload step behaves identically.
Re-upload idempotency: sentry-cli debug-files upload deduplicates by debug-id, so retries and parallel runs are safe.

What alternative solutions did you explore?

Install fastlane-plugin-sentry and call upload_symbols_to_sentry from upload_testflight_hybrid in fastlane/Fastfile#L509-L534. Rejected as the primary fix: it moves the upload one step further from the source (the xcarchive), and it adds a Ruby plugin dependency when we already use sentry-cli directly elsewhere in the workflow. Equivalent in outcome, slightly worse in ergonomics — fine as a fallback if the workflow change is contested.
Rely on the patched sentry_utils.rb (patches/sentry-react-native/@sentry+react-native+8.2.0.patch) to upload at archive time. Rejected because that script only runs when Xcode actually archives. Rock cache hits — the common case for unchanged native code — bypass it entirely. A CI step that runs unconditionally against the produced artifact is the only design that survives caching.
Switch Sentry to AppHangV2 (enableAppHangTrackingV2: true) to use MetricKit for richer hang capture. Rejected as primary because the reported symptom is "only main resolves" — a symbol-mapping problem, not a capture-mode problem. MetricKit reports also need dSYMs to symbolicate in-process frames. Worth filing separately as an enhancement for iOS 26.x signal richness.
Wait for Sentry's iOS 26 system-symbol cache to catch up. Falsifiable and falsified: APP-7CX on the same OS does symbolicate, so missing system symbols isn't the cause — it's the app's own dSYM that's missing.
Defer Chain 2 (YAPLJS re-entrancy mitigation on the App side, e.g., dispatching forwardLogsToSentry off the main thread in src/libs/Log.ts#L80-L82) until symbolicated APP-7FJ events confirm or rule it out. Rationale: the real fix lives in Mobile-Expensify's LogSyncerManager (already partially addressed by Update version to 1.2.46-0 on production #13930); App-side mitigation without symbol evidence risks fixing the wrong layer.

Proposal

What is the root cause of that problem?

This is a watchdog termination (AppHang, fatal) — the main thread was blocked for >2000ms. The only frame that resolves is main because every frame in the blocking chain lives in the OldDot/Mobile-Expensify native layer and in JavaScriptCore — neither of which is covered by the NewDot App dSYM the event was symbolicated against. This is not a separate, unknown hang: the evidence ties it to the already-symbolicated sibling APP-7CX (#91229).

The APP-7CX event symbolicates fully and shows a JSC lock re-entrancy deadlock on the main thread:

-[BaseViewController onBarButtonItemSelected:]
  → +[YAPLJS callFunction:args:]              (1st JS entry — grabs the JSC lock)
  → … JS runs, calls Log →
  → +[YAPLLogManager logWithLevel:…] → LogSyncerManager.sendLogPacket
  → +[YAPLJS callFunction:args:]              (2nd JS entry, re-entrant, same thread)
  → JSC::JSLock::grabAllLocks → WTF::ThreadCondition::timedWait → _pthread_cond_wait   ← main thread parked

Every symbol in that chain (YAPLJS, YAPLLogManager, LogSyncerManager, BaseViewController) is Mobile-Expensify code, and the deadlock parks inside JavaScriptCore/WTF. None of them belong to the NewExpensify App binary, so when an APP-7FJ event is symbolicated only against the NewDot dSYM (or the matching Mobile-Expensify dSYM for that build is missing), the whole chain collapses to the lowest frame that does resolve — main. That is precisely the APP-7FJ signature, and it is why the issue's own first step is "re-symbolicate against the matching dSYM": doing so will expand main into the APP-7CX chain.

Two corrections to the earlier auto-analysis posted on this issue:

The didFinishLaunchingWithOptions candidate points at ios/AppDelegate.swift, which is the standalone NewExpensify AppDelegate (withModuleName: "NewExpensify"). HybridApp builds from the Mobile-Expensify submodule, so that file's launch path is not linked into the crashing binary.
The only synchronous main-queue hop the App repo itself contributes to the HybridApp binary is RCTBootSplash.constantsToExport's RCTUnsafeExecuteOnMainQueueSync (RCTBootSplash.mm#L139-L150). It reads one trait-collection bool, runs once at module init, and does not match the symbolicated APP-7CX chain — so it is not the 2000ms blocker here (worth hardening on its own merits; see alternatives).

What changes do you think we should make in order to solve the problem?

Break the re-entrant native→JS call so the second +[YAPLJS callFunction:args:] can never contend for the JSC lock already held by the first entry on the same thread. The re-entry originates from logging inside an already-running JS call, so the JS hop in LogSyncerManager.sendLogPacket (Mobile-Expensify) should be deferred off the current synchronous frame instead of executed inline:

- // sendLogPacket() — invoked synchronously from inside an active YAPLJS callFunction frame
- [YAPLJS callFunction:@"<logHandler>" args:packet];   // re-enters JSC on the same thread → deadlock
+ // defer the log's JS hop so it runs after the current JS frame releases the JSC lock
+ dispatch_async(self.logSerialQueue, ^{
+     [YAPLJS callFunction:@"<logHandler>" args:packet];
+ });

Because this lives in Mobile-Expensify, the change is in that submodule, and one fix resolves APP-7CX and APP-7FJ together (plus the same-signature groupings APP-84S / APP-82E noted on #91229). I'd confirm the dedup by re-symbolicating a recent APP-7FJ event against the matching Mobile-Expensify dSYM before closing — the expanded chain should match #91229.

What alternative solutions did you explore? (Optional)

Re-entrancy guard at the bridge: have +[YAPLJS callFunction:args:] detect that it is already executing on the JS thread (depth counter) and queue the nested call instead of grabbing the JSC lock again. More general than the sendLogPacket fix, but a much larger blast radius.
Harden the App-side sync-main hop anyway: drop RCTUnsafeExecuteOnMainQueueSync in RCTBootSplash.constantsToExport and read traitCollection.userInterfaceStyle without a synchronous main-queue dispatch (or compute darkModeEnabled lazily). This removes the one App-contributed main-thread sync block, but it does not match the APP-7CX evidence, so I would not expect it to move APP-7FJ.
Treat APP-7FJ as a distinct native hang: only justified if re-symbolication shows a chain that does not match [$250] [Sentry: APP-7CX] HybridApp iOS Fatal App Hang in YAPLJS callFunction during onBarButtonItemSelected (still occurring after #86334 closed) #91229. Given identical platform + mechanism and the dSYM-coverage explanation above, I expect it to match.

@ZhenjaHorbach

The PR was reviewing by @ZhenjaHorbach, @mountiny could you please assign them here?

@sharabai

@sharabai Kinda hard to see from the Sentry as this event captures a lot of stuff, are we sure this is fully resolved now?

@mountiny

@mountiny unfortunately, the patches didn't help. At this point, I'd consider upgrading to a new major version. The issue that prevented me from suggesting that last time is now marked as "Closed", although there’s a new report saying that those crashes are still happening. At this point, I’d go with the upgrade and ask @chrispader to weigh in on whether the issue has been resolved.

@chrispader

@chrispader What do you think about upgrade?

I think if we continue to have issues with the older V4 VisionCamera version, upgrading to latest major V5 version makes most sense.

It's worth doing the potential migration work now to fix the issue, if we plan to update at some point either way.

Also, since V5 is a fundamental re-write using NitroModules, this could potentially fix a whole lot of other (yet undetected) issues with VisionCamera. Since these days NitroModules these days is super stable, this removes lots of potential for bugs in the JSI bridging layer from previous VisionCamera V4

@mountiny

@mountiny Sounds like upgrading to V5 makes sense. Should I go ahead and start on it?

@sharabai

(Vit's agent)

@sharabai yes, let's do it. I filed Upgrade react-native-vision-camera from V4 to V5 as a standalone tracker and assigned you and me.

The scoping is already in the body: what breaks, which 11 files import the package, and what we already have in place. Good news on the prerequisites — we are already on react-native-nitro-modules@0.37.1 and react-native-worklets@0.11.1, which is exactly what V5 5.2.3 is built against. The only new dependency is react-native-nitro-image.

One caveat worth reading before starting: upstream #4026 was closed in August, but two people reported the same abort still firing on 5.2.2 afterwards, and #4156 is still open. Our signature is different and the V4 code that throws it is gone in V5, so I still think the upgrade is the right move — just not a guaranteed fix.

Navigation Menu

Repository navigation

[HOLD https://github.com/Expensify/Mobile-Expensify/pull/13970] [$250] [Sentry: APP-7FJ] HybridApp iOS Fatal App Hang at main #91293

Sentry

Impact (snapshot at filing)

Stack trace

First investigation step

Why this is filed separately from APP-7CX

Reproduction

Related

Issue Analysis

1. `RCTBootSplash.constantsToExport` — synchronous main-queue dispatch during module init

2. `AppDelegate.didFinishLaunchingWithOptions` — synchronous startup chain

3. Distinction from APP-7CX (#91229)

Possibly related issues

Actionable next steps

Proposal

What is the root cause of that problem?

What changes do you think we should make in order to solve the problem?

What alternative solutions did you explore?

Proposal

What is the root cause of that problem?

What changes do you think we should make in order to solve the problem?

What alternative solutions did you explore? (Optional)

59 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[HOLD https://github.com/Expensify/Mobile-Expensify/pull/13970] [$250] [Sentry: APP-7FJ] HybridApp iOS Fatal App Hang at main #91293

Description

Sentry

Impact (snapshot at filing)

Stack trace

First investigation step

Why this is filed separately from APP-7CX

Reproduction

Related

Activity

MelvinBot commented on May 21, 2026

Issue Analysis

1. RCTBootSplash.constantsToExport — synchronous main-queue dispatch during module init

2. AppDelegate.didFinishLaunchingWithOptions — synchronous startup chain

3. Distinction from APP-7CX (#91229)

Possibly related issues

Actionable next steps

melvin-bot commented on May 21, 2026

melvin-bot commented on May 21, 2026

oqildev commented on May 21, 2026

Proposal

What is the root cause of that problem?

What changes do you think we should make in order to solve the problem?

What alternative solutions did you explore?

yusufdeveloper2903 commented on May 21, 2026

Proposal

What is the root cause of that problem?

What changes do you think we should make in order to solve the problem?

What alternative solutions did you explore? (Optional)

59 remaining items

Krishna2323 commented on Aug 30, 2026

mountiny commented on Aug 31, 2026

sharabai commented on Aug 31, 2026

mountiny commented on Sep 1, 2026

chrispader commented on Sep 1, 2026

chrispader commented on Sep 1, 2026

sharabai commented on Sep 2, 2026

mountiny commented on Sep 29, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `RCTBootSplash.constantsToExport` — synchronous main-queue dispatch during module init

2. `AppDelegate.didFinishLaunchingWithOptions` — synchronous startup chain