Performance: Offload MediaPipe pose detection to Web Worker

[shivansh31414]

Fixes #256

---

📝 Description

Offload MediaPipe pose landmark extraction from the main thread into a module Web Worker. This change moves expensive frame processing into a background thread, preserving the public poseService API (onResults() and send(image)) while improving UI responsiveness during workout sessions.

🔹 What has been changed?

• src/services/poseService.ts
  • Replaced direct on-main-thread MediaPipe usage with a worker-backed implementation.
  • send(image) now creates an ImageBitmap and posts it to the worker; onResults(callback) receives worker results unchanged.
  • Adds per-frame frameId tracking and short promise-based backpressure to avoid flooding the worker.
• src/workers/poseLandmarkWorker.ts (new)
  • Initializes @mediapipe/pose inside the worker (dynamically imported).
  • Accepts transferred ImageBitmap frames, draws to OffscreenCanvas (when available), runs pose.send({ image }), and posts { frameId, results } or { frameId, error } back.
• No changes to src/workers/poseWorker.ts (angle/math worker) — landmarks are still posted to it as before.

🔹 Why are these changes needed?

MediaPipe running on the main thread can block rendering and lower FPS on constrained devices. Offloading landmark extraction to a worker reduces main-thread CPU load and improves smoothness and responsiveness during live tracking.

---

🛠️ Type of Change

• ⚡ Performance Improvement
• 🐛 Bug Fix
• ✨ New Feature
• 🎨 UI/UX Improvement
• 📖 Documentation Update
• 🔧 Refactoring
• 💥 Breaking Change

---

🧪 Testing

• Dev server smoke test: app boots and workout flow runs under npm run dev.
• Results delivery: poseService.onResults() receives results posted from the worker.
• No uncaught worker initialization errors in the console.
• Basic backpressure: send() does not queue indefinitely; worker timeouts/logs on stalled frames.
• Full production build (npm run build) — blocked by unrelated TypeScript error in an orthogonal file (see notes).
• Hardware test on a low-end device (or Chrome CPU throttling) to validate FPS improvement >20%.

Notes:

• Worker uses modern browser features: module workers, createImageBitmap, OffscreenCanvas. Verify on target browsers and provide fallbacks if you need older-browser support.
• The worker attempts a fallback to pass ImageBitmap directly to pose.send() if OffscreenCanvas path fails.

---

🌐 Browsers / Environments Tested

• Chrome (dev)
• Firefox
• Edge
• Safari

---

📷 Screenshots / Demo

Please attach:

• DevTools Performance timeline before/after (optional)
• Worker console logs showing initialization and results posts
• Short screencap showing smoother UI during a workout session

---

📋 Checklist

• I have read the project's CONTRIBUTING guidelines
• My code follows the project style guidelines
• I have performed a self-review of my code
• I have smoke-tested locally using npm run dev
• I ran npm run build (blocked by unrelated TS error; see Notes)
• No new console errors from worker initialization or message handling
• PR includes clear description and testing steps

---

🛠️ Implementation Notes (for reviewers)

• The poseService API is unchanged — components like WorkoutScreen do not require updates.
• createImageBitmap() is used for efficient transfer; the worker receives the ImageBitmap as a transferable to avoid copying large frame buffers.
• The worker dynamically imports @mediapipe/pose and sets locateFile to the CDN path so WASM/assets load correctly in the worker context.
• For message correlation we attach frameId to results; this can be refined to a strict frame queue map if reviewers prefer.

---

---

[vercel]

Someone is attempting to deploy a commit to the somiljain2024-4175's projects Team on Vercel.

A member of the Team first needs to authorize it.

[Copilot]

Pull request overview

This PR refactors MediaPipe Pose landmark extraction to run in a dedicated module Web Worker, aiming to reduce main-thread load and improve UI responsiveness during continuous video processing.

Changes:

• Added a new poseLandmarkWorker that initializes MediaPipe Pose inside a worker and processes transferred ImageBitmap frames.
• Updated PoseService to spawn the module worker, transfer frames via postMessage, and surface results back through an onResults callback.
• Introduced per-frame acknowledgement/timeout tracking in PoseService using a pendingPromises map.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/workers/poseLandmarkWorker.ts	New worker that imports/initializes MediaPipe Pose and processes ImageBitmap frames off the main thread.
src/services/poseService.ts	Refactors pose processing to delegate work to the new module worker and relay results/timeouts back to callers.

Comments suppressed due to low confidence (3)

src/services/poseService.ts:97

• If the Promise.race times out, the pendingPromises entry for this frame is never removed, so a hung worker will leak entries indefinitely. Also, inProgress is only cleared in the catch path, which makes it easy to get stuck if the worker later responds unexpectedly. Ensure timeout handling deletes the pending entry and resets state in a finally, and consider terminating/restarting the worker on timeout to prevent permanent stalls.

      const promise = new Promise<void>((resolve, reject) => {
        this.pendingPromises.set(id, { resolve, reject });
        // Transfer the ImageBitmap for zero-copy
        try {
          this.worker!.postMessage({ type: 'processFrame', frameId: id, imageBitmap: bitmap }, [bitmap]);
        } catch (err) {
          this.pendingPromises.delete(id);
          reject(err);
        }
      });

      // wait for processing to finish (or timeout)
      const timeout = new Promise<void>((_, rej) => setTimeout(() => rej(new Error('pose worker timeout')), 2000));
      await Promise.race([promise, timeout]);
    } catch (error) {
      console.error('PoseService.send error', error);
      this.inProgress = false;
    }

src/services/poseService.ts:56

• worker.onerror only logs. If the worker throws/crashes after send() sets inProgress = true, inProgress may never be cleared and any pendingPromises will never resolve/reject, effectively deadlocking pose processing. Handle onerror by rejecting/clearing all pending entries and resetting inProgress (and optionally recreating the worker).

      this.worker.onerror = (e) => {
        console.error('PoseService worker thrown error', e);
      };

src/services/poseService.ts:109

• close() terminates the worker but does not clear/reject any pendingPromises or reset inProgress. If close() is called while a frame is pending, callers awaiting send() may hang and the service can remain stuck. Consider rejecting/clearing pendingPromises, resetting inProgress, and nulling the callback as part of shutdown.

  close() {
    if (this.worker) {
      try {
        this.worker.terminate();
      } catch (e) {
        console.warn('Error terminating pose worker', e);
      }
      this.worker = null;
    }
  }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Somil450]

@shivansh31414 plz correct the issues suggested by copilot

[shivansh31414]

okay

[shivansh31414]

done

[shivansh31414]

please check the pr

[Somil450]

@shivansh31414 plz solve these merge conflicts