Build a Real-Time In-Browser Fitness Coach with React, Vite and Google MediaPipe Pose

Table of Contents

1. Key Highlights
2. Introduction
3. System architecture and data flow: how the pieces connect
4. Integrating MediaPipe Pose into React and Vite
5. From landmarks to insight: joint-angle math and 3D considerations
6. Designing robust repetition counting: finite-state machine and hysteresis
7. UI and feedback: delivering live HUDs without blocking React
8. Performance tuning for 60 FPS: practical strategies
9. Privacy, security, and production considerations
10. Extending the system: new exercises, multi-person, and backend options
11. Real-world examples and use scenarios
12. Troubleshooting and common pitfalls
13. Getting started: a minimal step-by-step guide
14. Choosing trade-offs: accuracy, latency, and device coverage
15. FAQ

Key Highlights

• A complete client-side pipeline using MediaPipe Pose (WASM/WebGL) delivers 33-point body landmark detection at 60 FPS with sub-15ms inference, enabling accurate exercise tracking and live biomechanical feedback without any server-side processing.
• Reliable rep counting and form assessment combine vector trigonometry, a finite-state machine with hysteresis, and selective React updates to avoid render bottlenecks and prevent false positives.
• The approach preserves user privacy (no video leaves the browser), scales to commodity hardware, and can be extended to new exercises, multi-person tracking, and optional backend analytics.

Introduction

Wearable sensors measure heart rate, cadence, and motion, but they cannot inspect joint angles or evaluate whether a squat reaches proper depth. Historically, real-time biomechanical analysis required GPU-backed servers and complicated deployment. Web technologies have moved past that constraint. The combination of WebAssembly (WASM), WebGL acceleration and Google MediaPipe Pose brings pose estimation into the browser with millisecond-level latency and production-level stability.

A practical result: a fully client-side workout assistant that captures a webcam feed, extracts 33 body landmarks, computes joint angles, counts repetitions robustly, and delivers visual and audio feedback at full frame rate. That setup keeps raw video on the user's machine, eliminates backend inference costs, and simplifies deployment with static hosting such as Vercel or Netlify.

This article breaks the system into actionable parts. Each section explains why a design choice matters, how to implement it, and what pitfalls to avoid. Code excerpts appear where clarity benefits; explanations focus on mathematical foundations, performance strategies, and how to combine these into a resilient product.

System architecture and data flow: how the pieces connect

A reliable, low-latency in-browser fitness coach must process a continuous stream of frames and convert raw pixels into actionable signals. The architecture is straightforward on paper but demands careful engineering to avoid jitter and unnecessary UI rendering.

Pipeline overview:

• Webcam feed captured via a <video> element.
• Frames passed to MediaPipe's PoseLandmarker running in GPU-accelerated WASM.
• PoseLandmarker returns 33 3D landmarks per person: normalized x, y, z coordinates plus per-landmark confidence.
• A lightweight joint-angle module computes interior angles for joints of interest.
• A finite-state machine analyzes angle trajectories to count repetitions with hysteresis.
• Visual overlays are drawn directly to an HTML <canvas> for skeleton and form cues.
• React handles persistent UI: rep counts, session controls, and non-frame-critical notifications.
• Audio feedback is triggered from the client to announce rep completion or warn about form breaks.

Key design constraints:

• Keep the high-frequency loop (60 FPS) outside React rendering cycles.
• Minimize copying and serialization of pixel data to and from WebGL/WASM.
• Use per-landmark confidence values to gate noisy predictions.
• Update React state only when the user-visible metrics change meaningfully.

Visualizing the flow clarifies responsibilities: MediaPipe provides the perception layer. JavaScript trigonometry interprets the pose. The state machine extracts semantics. Canvas and audio supply feedback. React glues the interface and persistent information together.

Integrating MediaPipe Pose into React and Vite

The MediaPipe Tasks Vision library is built for web deployment. It ships WASM assets and a bridge API: PoseLandmarker. The implementation below highlights initialization choices that affect latency and accuracy.

Install the SDK:

npm install @mediapipe/tasks-vision

Initialize the PoseLandmarker (abridged):

import { PoseLandmarker, FilesetResolver } from "@mediapipe/tasks-vision";

export const initializePoseLandmarker = async () => {
  const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
  );

  const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
    baseOptions: {
      modelAssetPath: `https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_lite/float16/1/pose_landmarker_lite.task`,
      delegate: "GPU",
    },
    runningMode: "VIDEO",
    numPoses: 1,
  });

  return poseLandmarker;
};

Why these options matter:

• modelAssetPath: selects a pre-trained model variant. The "lite" family trades accuracy for speed; choose depending on target devices.
• delegate: "GPU" leverages WebGL for acceleration. If Safari or other environments present compatibility issues, fall back to CPU.
• runningMode: "VIDEO" configures internal buffering for continuous inference.
• numPoses: limits work when single-person tracking suffices.

Practical tips for React:

• Initialize the model once (e.g., on component mount) and keep it in a ref to avoid re-initialization cost.
• Serve model assets from a CDN to reduce cold-start delays on first load and enable cache reuse.
• Detect hardware capabilities at runtime. If GPU delegate initialization fails, gracefully fall back to CPU.

MediaPipe returns landmarks in normalized coordinates (0..1 for x and y relative to the image) and a z coordinate that is relative to the landmark scale. Convert normalized values into pixel coordinates for drawing, but keep normalized values when computing proportions across different resolutions.

From landmarks to insight: joint-angle math and 3D considerations

MediaPipe gives a set of keypoints (33 landmarks). Interpreting them requires geometric reasoning. Joint angles explain how limbs move. The interior angle defined by three points A → B → C (with B at the vertex) is the key measurement.

A robust 2D angle calculation uses arctangent differences between vectors BA and BC:

export function calculateAngle(a, b, c) {
  const radians = Math.atan2(c.y - b.y, c.x - b.x) - Math.atan2(a.y - b.y, a.x - b.x);
  let angle = Math.abs((radians * 180.0) / Math.PI);
  if (angle > 180.0) angle = 360.0 - angle;
  return Math.round(angle);
}

Why use atan2 differences:

• The expression handles vector orientation smoothly and avoids the numerical issues that arise from computing inverse cosine on small magnitudes.
• It returns a signed angle corrected to the [0, 180] range, useful for defining thresholds.

Examples of joints:

• Bicep curl (right arm): Shoulder (11) → Elbow (13) → Wrist (15)
• Squat (right leg): Hip (23) → Knee (25) → Ankle (27)

3D considerations:

• MediaPipe's z is relative and not an absolute depth measurement. Use it to disambiguate occlusions (e.g., when one limb is in front of another) or to approximate forward/backward motion.
• For lateral tilt or torso lean, compute angles between shoulders and hips across the frontal plane. For example, shoulder-hip-ankle sequences can reveal leaning.
• Normalize joint positions against a torso-centered reference (for example, distance between shoulders) to make thresholds independent from camera distance.

Stability and filtering:

• Raw landmark estimates jitter, especially in low-light or with occlusion. Apply a simple exponential moving average (EMA) on landmark coordinates or on computed angles to smooth the signal: angle_smoothed = alpha * angle_current + (1 - alpha) * angle_prev
• Choose alpha based on desired responsiveness: 0.2–0.4 preserves responsiveness while suppressing micro-fluctuations.

Confidence gating:

• Each landmark includes a confidence score. Exclude landmarks with low confidence from angle computations. For example, skip rep evaluation for a frame if elbow or wrist confidence < 0.5.
• If confidence drops for a continuous period, trigger a "hold still" or "reposition camera" notification to the user rather than producing spurious counts.

Coordinate transforms for drawing:

• Convert normalized x,y to canvas pixel coordinates, accounting for aspect ratio and letterboxing if the <video> element and <canvas> sizes differ.
• If the camera feed is mirrored for the user, mirror landmark x coordinates accordingly when overlaying.

Designing robust repetition counting: finite-state machine and hysteresis

Counting repetitions requires more than threshold checks. Real exercise motion is noisy, and muscles may twitch or the user may linger near threshold values. A finite-state machine (FSM) with hysteresis prevents false triggers by encoding phases and only counting transitions that represent actual repetitions.

FSM example for bicep curls:

• States: DOWN (arm extended), UP (arm contracted)
• Thresholds: UP threshold (e.g., elbow angle < 35°), DOWN threshold (e.g., elbow angle > 160°)
• Hysteresis: ensure the FSM only transitions when an opposing threshold is crossed, preventing rapid flips if angle hovers near a boundary.

Simplified logic:

const STAGES = { UP: "up", DOWN: "down" };
let currentStage = STAGES.DOWN;
let repCount = 0;

function evaluateRepetition(elbowAngle) {
  if (elbowAngle < 35 && currentStage === STAGES.DOWN) currentStage = STAGES.UP;
  if (elbowAngle > 160 && currentStage === STAGES.UP) {
    currentStage = STAGES.DOWN;
    repCount += 1;
    triggerAudioFeedback("Good rep!");
  }
  return { count: repCount, stage: currentStage };
}

Why hysteresis matters:

• Without distinct UP and DOWN thresholds, a single pass through the threshold could register multiple times if the joint oscillates.
• Real exercises involve pauses and partial reps — hysteresis ensures only a full contraction-extension cycle counts.

Extending FSMs for complex exercises:

• Squats: track hip-knee-ankle angles and torso lean. Distinguish between partial and full-depth squats by measuring knee angle plus hip descent relative to ankle position.
• Deadlifts: monitor spine neutralization via shoulder-hip-ankle alignment and hip hinge via hip-knee-ankle angles.
• Multi-joint movements: combine several joint FSMs (e.g., squat plus forward reach) and require synchronized state transitions for a rep to count.

False positive mitigation strategies:

• Combine angle thresholds with velocity constraints: only register an UP transition if the change exceeds a small minimum rate over a short window, ensuring intentional motion.
• Minimum time between counts: require at least N frames or T milliseconds between counted repetitions.
• Use landmark confidence to suspend counting when detection is unreliable.

Edge cases and ambiguity:

• When multiple people enter frame, restrict counting to the largest bounding pose or require the user to initiate a calibration frame.
• If the camera view doesn't capture the entire joint chain (e.g., wrists out of frame), provide a helpful UI prompt instead of guessing.

UI and feedback: delivering live HUDs without blocking React

A common pitfall is tying every video-frame computation to React useState updates. That creates 60Hz rerenders and kills performance. The right approach isolates the frame loop from React and uses low-level primitives for drawing and communication.

Techniques to separate concerns:

• Keep the per-frame loop in requestAnimationFrame. This keeps inference and draw cadence synchronized to the display and avoids layout thrashing.
• Store frame-level data in useRef. Refs can be mutated without rerendering React components.
• Draw skeletal overlays on a standalone canvas using raw 2D context commands. Drawing is lightweight and immediate compared to DOM updates.
• Throttle React updates. Only set state when repCount or formWarning changes beyond a threshold or when a session event (start/stop) occurs.

Minimal draw loop pattern:

function drawLoop() {
  requestAnimationFrame(drawLoop);
  if (!poseLandmarker || !videoElement.readyState) return;

  const results = poseLandmarker.detectForVideo(videoElement, timestamp);

  // compute angles and FSM using refs (no setState)
  const { repCountChanged, warningChanged } = processFrame(results);

  // draw overlay on canvas
  drawSkeletonOnCanvas(results.landmarks);

  // update UI when important changes occur
  if (repCountChanged || warningChanged) setStateFromRefs();
}

Audio feedback:

• Play short audio cues on rep completion or long-form audio for start/stop.
• Preload audio buffers to avoid playback latency.
• Respect user preferences: allow muting and adjust for accessibility (vibration on mobile, larger text).

Accessibility and visual cues:

• Color-code overlays for correctness: green for good form, amber for borderline, red for critical fail.
• Use clear textual feedback with concise instructions, e.g., "Lower more" or "Straighten your back".
• Include a calibration step that instructs the user to stand in a neutral pose facing the camera to normalize thresholds.

Responsive UI concerns:

• On mobile browsers, camera permissions and orientation change frequently. Redraw canvas dimensions and recalc transforms when the viewport or camera stream changes.
• Consider full-screen mode for workouts; draw HUD elements with safe-area insets for phones.

Performance tuning for 60 FPS: practical strategies

Achieving consistent 60 FPS across devices requires attention to model selection, frame management, and efficient rendering.

Model and delegate decisions:

• Use a "lite" pose model for lower-latency devices where speed matters more than millimeter-level precision. Offer a toggle to switch model variants for high-accuracy use cases.
• Prefer GPU delegates on desktop Chrome/Firefox where WebGL drivers are robust. On mobile Safari or older devices, consider CPU delegates or a reduced frame rate.

Frame size and input scaling:

• Don't feed full-resolution camera frames to the model unless necessary. Downscale the input to a resolution that preserves anatomical features while reducing compute.
• Keep the aspect ratio; use letterboxing to avoid distortion. Convert model outputs to the canvas coordinate system accordingly.

Workload distribution:

• Offload non-critical computations to Web Workers if you need to decouple heavy math from the main thread. Be mindful of transferring image data; avoid copying pixels where possible.
• Use OffscreenCanvas for drawing when supported. It allows rendering inside a worker and prevents blocking the main thread.

Selective inference and throttling:

• If the device cannot maintain 60 FPS, reduce inference rate: run MediaPipe every Nth frame and interpolate or hold overlays for the frames in between.
• Use adaptive throttling: monitor average frame processing time and auto-adjust target inference rate to keep drops minimal.

Memory and asset management:

• Cache WASM and model assets on the client. Use service workers for offline reuse.
• Release references to heavy objects (e.g., poseLandmarker) on session end to free up WASM memory.

Profiling and measurement:

• Measure end-to-end latency: timestamp when the video frame is captured and when the decision (rep count, warning) is produced. That reveals whether delays are in camera capture, model inference, or post-processing.
• Track per-frame durations in development builds and present average and p95 metrics in a developer debug overlay.

Practical numbers from a well-optimized client:

• Model inference on modern desktops: ~8–12 ms per frame (GPU delegate).
• Total pipeline (capture → inference → draw): sub-15 ms on capable devices, enabling 60 Hz responsiveness.
• Mobile devices vary: expect 20–40 ms on mid-range phones; adapt inference cadence accordingly.

Privacy, security, and production considerations

Client-side pose estimation delivers a significant privacy advantage: raw video never leaves the user's device. Nonetheless, product integrity and legal compliance require attention to details.

Camera permissions:

• Always request camera permission transparently, explain the need, and show a privacy note that video never leaves the browser unless explicitly consented.
• Respect browser constraints: mobile browsers may prompt repeatedly unless you maintain a secure context and HTTPS.

Model licensing and asset hosting:

• Verify the MediaPipe model license for commercial use. Host model assets on a CDN or your own infrastructure and keep version control for reproducibility.
• Use content security policies (CSP) to allow CDN origins while blocking unauthorized scripts.

Optional analytics and opt-in telemetry:

• If you plan to store session data for analytics, collect only what users consent to. Consider computing sensitive metrics client-side and sending only aggregated, anonymized summaries to the server.
• Provide users a way to export session data locally before uploading.

Security and integrity:

• Protect against supply-chain attacks by pinning versions of @mediapipe/tasks-vision and serving WASM from trusted CDNs. Use subresource integrity (SRI) when possible for static assets.
• Avoid embedding remote scripts that can be replaced silently; use build-time pulls and serve from your origin in production if security is a concern.

Data retention and healthcare compliance:

• If the app targets clinical or PT use, ensure compliance with local healthcare data regulations (e.g., HIPAA in the U.S.) when storing identifiable health information.
• Offer strict opt-in flows and encryption for any stored video or pose data.

Cross-browser and device compatibility:

• Test across Chrome, Firefox, Safari (mobile and desktop) and on different hardware. GPU delegates behave differently across engines; provide fallback code paths.
• On iOS, camera capture constraints and backgrounding behavior can interrupt sessions; implement robust reconnection logic.

Extending the system: new exercises, multi-person, and backend options

A modular client-side pose pipeline makes extension straightforward. Consider these directions when expanding features.

Adding new exercises:

• Define exercise templates that list the joint triplets (A→B→C), thresholds for up/down phases, velocity constraints, and ancillary checks (e.g., torso angle).
• Implement a calibration step per exercise: ask the user to stand at reference pose to derive personalized thresholds based on limb proportions and camera distance.

Multi-person tracking:

• MediaPipe supports multiple poses. For a workout coach, select a primary user by:
  • Largest bounding box area,
  • Closest to the camera (z coordinate),
  • A "calibration" frame where the user explicitly indicates themselves (raise right hand).
• Design the UI to switch targets when multiple users are present.

Quality scoring and rep critique:

• Move beyond counting. Aggregate metrics per rep: range of motion, smoothness (variance of angular velocity), tempo consistency, and symmetry between left/right.
• Combine those into a composite quality score. Provide per-rep highlights: "Limited elbow range" or "Knee valgus detected".

Backend integration for optional analytics:

• Send only aggregated metrics (rep counts, average ROM, session duration) rather than raw video or landmarks, unless explicitly consented.
• Use server-side endpoints for profile sync, long-term progress tracking, leaderboards, or coach review.
• If storing landmarks, encrypt at rest and during transit.

Coach workflows and remote PT:

• Implement session sharing where a remote coach can review anonymized skeleton replays or ask for explicit consent to view a recording.
• Allow coaches to annotate rep ranges and send targeted instruction. Keep recordings short and private.

Mobile and embedded deployment:

• Progressive Web App (PWA) support enables installable experiences that behave like native apps and gain access to camera APIs consistently.
• For embedded devices (kiosks in a gym), use fixed cameras and calibrate thresholds for the known camera position to improve consistency.

Real-world examples and use scenarios

Several practical scenarios illustrate where a client-side fitness coach adds immediate value.

Home user improving squat depth: A weekend athlete wants to ensure squats reach safe depth. The system measures knee angles and hip displacement relative to ankle, counts reps, and notifies when depth falls short. Because everything runs locally, the user avoids uploading exercise videos to third-party servers.

Remote physical therapist: A therapist monitors patient compliance remotely. The patient records a session and shares aggregated metrics and flagged reps. The therapist receives a concise report showing ROM improvements and any alarming form regressions over multiple sessions.

Gym classroom or group training: In a group class, a stationary camera tracks multiple participants. The instructor toggles to the largest pose on the screen for personalized coaching. Immediate auditory cues help maintain pace and reduce risk of injury.

Rehab applications: Subtle deviations in form are critical here. By using higher-accuracy models and careful smoothing, the app can detect compensatory movements (e.g., lateral trunk lean) that indicate pain or weakness. Data-sharing flows must comply with medical privacy laws.

Gameified fitness: Rep counting feeds into gamified mechanics such as streaks, unlocks, or progress bars. Because counts are deterministic on-device, the logic remains transparent and responsive.

Each scenario benefits from client-side speed and privacy, but the product boundaries—especially around data sharing—must be chosen deliberately.

Troubleshooting and common pitfalls

Even with a robust pipeline, developers will face recurring issues. The following practical guidance addresses typical pain points.

Camera permissions and startup:

• Ensure the page is served over HTTPS. Many browsers block camera access from insecure contexts.
• Handle promise rejections when getUserMedia is denied; provide clear instructions and a retry path.

Pose jitter and flicker:

• Smooth landmarks or computed angles with EMA.
• Use per-landmark confidence thresholds and suspend rep evaluation when confidence is low.
• Encourage users to increase ambient lighting or adjust camera framing.

Occlusion and out-of-frame joints:

• If key joints are often occluded, inform the user to reposition. Consider alternate joint chains that remain visible.
• For seated exercises, customize the expected joint set, since ankles/feet may be off-camera.

False positives during warm-ups:

• Require an initial "start set" gesture or a button press to begin counting.
• Implement a minimum repetition time or a minimum amplitude to reduce accidental counts.

Performance drops on low-end devices:

• Reduce input resolution and switch to a lighter model.
• Lower model inference rate adaptively based on CPU/GPU load.
• Use a frame skipping strategy.

Browser-specific quirks:

• Safari's WebGL driver has idiosyncrasies. Test WebGL-based delegates across browsers and fallback gracefully.
• Some mobile browsers restrict background camera use; handle backgrounding and reinitialization.

Audio latency:

• Preload audio and use the Web Audio API for predictable low-latency playback.
• On mobile, be mindful of silent mode and provide visual cues as a fallback.

Calibration and personal differences:

• People vary in limb proportions and mobility. Offer a brief calibration that sets baseline thresholds automatically.

Testing strategies:

• Collect test videos to validate detection across skin tones, clothing, and lighting conditions.
• Use synthetic perturbations: add noise to landmarks, simulate jitter, and validate FSM behavior.

Getting started: a minimal step-by-step guide

A minimal bootstrapped project using React + Vite provides a quick path to a working prototype.

1. Create the project:

npm create vite@latest ai-gym-bro -- --template react
cd ai-gym-bro
npm install
npm install @mediapipe/tasks-vision

2. Initialize the PoseLandmarker in a React component (use useEffect once on mount) and store it in a ref.
3. Add a video element and request camera permission:

const stream = await navigator.mediaDevices.getUserMedia({ video: true, audio: false });
videoRef.current.srcObject = stream;
await videoRef.current.play();

4. Implement requestAnimationFrame loop that:

• Calls poseLandmarker.detectForVideo(video, timestamp).
• Computes angles via calculateAngle.
• Runs FSM logic for counting.
• Draws skeleton overlay on canvas.
• Updates React state only when rep counts or warnings change.

5. Add UI controls: start/stop session, mute audio, select exercise template.
6. Deploy to static hosting (Vercel/Netlify). Serve model assets from CDN or your static host.

For a complete reference implementation, star and clone the open-source project that inspired this guide: https://github.com/umersmx/ai-gym-bro-web

The repository includes sample UI components, exercise templates, and production-ready build scripts.

Choosing trade-offs: accuracy, latency, and device coverage

No single configuration suits all scenarios. Make explicit trade-offs and provide per-user choices.

Accuracy vs. latency:

• Use a heavier model when precision trumps real-time responsiveness (e.g., clinical assessments).
• Prefer the light model and GPU delegate when real-time audio cues and gamification require low latency.

Device coverage vs. feature richness:

• Implement a graceful degradation strategy: if the device lacks GPU delegates or cannot sustain high FPS, reduce features (e.g., fewer overlays, lower inference rate) rather than failing.
• Offer device capability diagnostics and suggested settings for different classes of hardware.

Data privacy vs. analytics:

• Default to a privacy-first posture: do not collect or transmit raw video. Offer explicit opt-in for uploads and explain exactly what will be transmitted.

Simplicity vs. personalization:

• Start with robust, conservative thresholds out of the box.
• Provide a calibration flow to unlock personalized thresholds for more accurate counting and critique.

FAQ

Q: Does the video stream leave the user's device? A: No. The entire pose detection and processing pipeline runs in the browser. No frames are uploaded unless the user explicitly consents to sharing or uploads a recording.

Q: What browsers and devices are supported? A: Modern Chromium-based browsers and Firefox provide good WebGL support for GPU delegates. Safari support is improving; test on target iOS versions. Performance varies by device; offer fallbacks for CPU inference or reduced frame rates.

Q: How accurate are joint angles computed from MediaPipe? A: MediaPipe provides robust 33-point landmarks for many poses. Accuracy depends on camera angle, resolution, and lighting. For many exercises, computed interior angles have sufficient precision to count repetitions and identify gross form errors. Use smoothing and confidence gating to reduce noise.

Q: How do I avoid double-counts or missed reps? A: Implement an FSM with hysteresis and velocity or time gating. Require full contraction and extension thresholds and ensure a minimum time between counts. Use landmark confidence to pause counting when detection is unreliable.

Q: Can this run on mobile phones? A: Yes. Mobile devices can run the lite models at acceptable latencies. Expect range depending on the phone: older or low-end phones may need lower resolutions or less frequent inference.

Q: Can I add new exercises? A: Exercises are templates: define the key joint triplets, threshold values, and optional constraints (torso angle, symmetry). Provide calibration to adapt thresholds to individual body proportions.

Q: What are common causes of poor detection? A: Low light, occlusion (clothing or objects blocking joints), camera angle (extreme foreshortening), and fast motion that causes motion blur. Improving lighting, re-framing the camera, or slowing tempo improves reliability.

Q: Is this suitable for clinical or therapeutic use? A: The client-side pipeline can support rehabilitation monitoring, but clinical deployments may require higher accuracy, validated models, and strict data governance (e.g., HIPAA-compliant backends). Treat this as a decision-support tool rather than a diagnostic device unless validated for that purpose.

Q: Where can I find starter code? A: A working open-source implementation is available at https://github.com/umersmx/ai-gym-bro-web. It demonstrates React integration, MediaPipe initialization, FSM-based counting, and performance optimizations.

Q: How do I measure end-to-end latency? A: Timestamp the moment you render a camera frame to the canvas and timestamp when your decision (rep count increment, warning) is produced. The difference is the end-to-end latency; adding model's inference time produces insight into where bottlenecks lie.

Q: Can this be combined with wearables? A: Yes. Combining inertial sensors with pose landmarks improves robustness—IMU data can disambiguate occlusions, measure rotational velocities, and provide high-frequency cadence. Architecture-wise, consume wearable streams in parallel and fuse signals in the FSM or a higher-level decision module.

Q: Are there licensing or commercial restrictions on MediaPipe? A: MediaPipe models and SDKs have licensing terms. Confirm the license terms for your intended commercial use and host your model assets appropriately. When in doubt, consult legal counsel.

Q: How do I disable audio feedback? A: Provide a mute toggle in the settings. Also present visual replacements for accessibility and silent environments.

Q: What metrics should I collect if I add analytics? A: Collect aggregated session statistics such as number of reps per session, average ROM, session duration, and per-exercise quality score—only if the user opts in. Avoid collecting raw video or personally identifiable landmark streams without explicit consent.

Q: How do I handle multi-person frames? A: Track a single primary user, selected by bounding box size, z-depth, or an explicit calibration gesture. Provide visual selection UI to switch targets if needed.

This implementation pattern turns commodity web technologies into a practical, private, and responsive workout coach. The combination of MediaPipe Pose, accurate joint-angle computation, hysteresis-based FSMs, and careful rendering strategies addresses both correctness and performance. Whether building a hobby project or the foundation for a commercial product, the architecture scales: swap models for accuracy, add analytics with consent, and expand exercise templates for richer feedback.