On-Device Workout Rep Counting in Flutter: Reliable Techniques for Private, Low-Latency Pose Tracking

Table of Contents

1. Key Highlights:
2. Introduction
3. Why straightforward geometry fails in the wild
4. From angles to signals: the conceptual shift that fixed counting
5. Signals that work and how to compute them
6. How to handle 2D perspective and monocular depth
7. Building an on-device pipeline in Flutter
8. Practical debugging: hot-swap signals and on-device A/B
9. Trade-offs: privacy, accuracy, and latency
10. Model and inference choices: what to run on-device
11. State machine design and anti-bounce hysteresis
12. Normalization and personalization
13. Evaluation and metrics: how to measure success
14. Combining IMU and camera: when fusion makes sense
15. Handling multiple exercises and extensibility
16. Future directions and advanced techniques
17. Production experiences: what broke and how we recovered
18. Deployment and maintenance considerations
19. Final takeaways for developers building on-device camera ML
20. FAQ

Key Highlights:

• Counting exercise repetitions from a single phone camera requires treating rep detection as a signal-processing problem rather than relying on raw geometric thresholds; robust features survive arbitrary camera angles.
• Practical solutions combine torso-axis orientation, normalized displacement signals, smoothing and hysteresis, and an on-device A/B workflow to iterate quickly without sending user video to servers.
• A production-ready stack uses lightweight pose models on-device (tflite), a background inference pipeline, careful normalization, and instrumentation for live debugging and per-exercise signal selection.

Introduction

Turning squats and push-ups into a pet-raising game demands an uncommon mix of machine perception, mobile engineering and user experience. Users expect instant feedback every rep, and they refuse to stream their workout to a server. That combination—privacy and latency—forces all pose estimation, inference and rep counting to run locally on the phone.

The core technical challenge looks simple at first glance: detect keypoints, calculate an angle, and count when it crosses a threshold. That naive approach works in controlled demos but fails in the field where camera positions, user height, ambient occlusion and monocular depth errors conspire to produce wildly inconsistent counts. Solving this problem required reframing rep counting as the design of robust temporal signals that persist across perspective distortion, building a toolset to compare those signals live on-device, and adopting engineering patterns that let the team iterate quickly while preserving privacy.

This article describes what broke, why it broke, and precisely what fixed it. It documents the signal types that held up in real user conditions, how to normalize and smooth them, the practical constraints of on-device pose models, and a production architecture in Flutter that keeps rep counts instant and reliable. The lessons apply to any app that needs human motion signals from a single camera without cloud processing.

Why straightforward geometry fails in the wild

Counting repetitions seems like a geometry problem: measure a joint angle and threshold it. People tried knee angles for squats, elbow and shoulder angles for push-ups, and body collinearity checks to decide whether an exercise is horizontal or vertical. All of these fail in realistic settings because a phone camera produces only a 2D projection of a 3D scene.

Perspective distorts angles. The same squat shot from different camera positions can look like a deep 90° knee bend or a shallow 150°. Even when pose models provide an estimated depth coordinate (a monocular z), that value is noisy. Thresholding on z or on an angle computed with z introduces jitter that doubles or misses reps.

Real users place phones in arbitrary places: propped on the floor, angled from the side, off to a corner. A "body-line" gate that checks whether shoulder-hip-heel are collinear works in a front-facing, centered shot but collapses when perspective bends the line in the projected image. Apps that silently decide a perfectly valid push-up doesn't count create immediate user frustration.

The core failure mode is trust in absolute geometric relationships derived from a single image. Those relationships do not survive changes in camera pose and ground truth depth. A stable rep detector must rely on features that are invariant or at least robust to perspective and scale changes.

From angles to signals: the conceptual shift that fixed counting

Successful rep counting treats the problem as extracting and evaluating temporal signals extracted from pose keypoints, not computing a fixed angle once per frame.

Two principles make this approach resilient:

1. Use robust, low-dimensional features that change consistently during a rep and are less sensitive to camera pose. The torso axis—the vector from hips to shoulders—changes orientation reliably between upright and horizontal exercises. Torso verticality is far more reliable than a three-joint colinearity test.
2. Provide multiple candidate signals and test them live with a hot-swap mechanism. Different gyms, lighting, and camera placements favor different signals. Building the ability to switch signals on-device and A/B test them with real users revealed which combinations worked across scenarios.

Signal design emphasizes relative measures and normalization. Instead of raw pixel distances or absolute angles, compute displacement relative to torso length or bounding box size. Normalize for user height and distance from camera. Track motion over time with smoothing and hysteresis to avoid double counts from jitter.

Treat rep detection as a state machine over a smoothed signal: detect a "down" phase then a return-to-"up" phase, count once, and suppress additional counts until the cycle completes. That hysteresis eliminates spurious counts caused by noisy keypoint estimates.

The meta-lesson: build a small experimental framework that lets you observe and compare signals running on-device against real movement. The winning method rarely looks best on paper; it becomes evident when tested under realistic conditions.

Signals that work and how to compute them

Below are signal types that proved robust when normalized and filtered. Each has strengths and weaknesses per exercise and camera placement. Use them as building blocks and expose them behind a debug toggle for live comparison.

1. Torso axis orientation (hips → shoulders)

• What it measures: the vector from the midpoint of hips to the midpoint of shoulders. Compute its angle relative to gravity or image vertical.
• Why it works: orientation of the torso changes dramatically between upright (squat, standing) and horizontal (push-up) movements. This vector is a compact summary less affected by perspective than multi-joint collinearity tests.
• How to compute:
  • Compute shoulder midpoint: (left_shoulder + right_shoulder)/2.
  • Compute hip midpoint: (left_hip + right_hip)/2.
  • Torso vector = shoulders_midpoint - hips_midpoint.
  • Angle = atan2(torso_vector.y, torso_vector.x) or the dot product with an image vertical vector to get a signed measure.
• Normalization: scale by torso length to express displacement in user-independent units.
• Filtering: low-pass exponential moving average (EMA) with a small alpha (e.g., 0.1 at 30 FPS) for orientation and a faster EMA for the derivative to detect movement direction.

2. Normalized limb travel

• What it measures: displacement of a limb (e.g., knee, wrist) along a principal axis normalized by a body reference (torso length or bounding box height).
• Why it works: absolute pixel motion varies with distance to camera; normalization preserves consistent thresholds across users and camera distances.
• How to compute:
  • Track the y (vertical) position of an ankle for squats or the wrist for push-ups.
  • Normalize by shoulder-hip distance or overall pose bounding box height: normalized_pos = (pos_y - reference_y) / torso_length.
  • Compute smoothed derivatives to detect motion direction.
• Notes: Use the same limb consistently and account for left/right asymmetry by averaging both limbs or selecting the more stable keypoint per user.

3. Angle-with-smoothing (robust joint angles)

• What it measures: joint angles computed with smoothing across frames, with hysteresis applied for threshold crossing.
• Why it works: raw angle values jitter; smoothing plus hysteresis turns noisy values into a stable signal for state transitions.
• How to compute:
  • Compute angle between three keypoints with standard vector math.
  • Apply Savitzky–Golay or EMA filter to reduce high-frequency noise while preserving motion peaks.
  • Use dual thresholds (enter and exit) to avoid double-counting: count when angle drops below enter_threshold and then rises above exit_threshold.

4. Torso displacement along principal axis

• What it measures: the position of torso midpoint projected along an axis (vertical or camera-normal) to detect downward/upward movement in squats or push-ups.
• Why it works: torso displacement is a direct indicator of rep phase and less susceptible to limb occlusion.
• How to compute:
  • Project the torso midpoint onto image vertical or along torso orientation.
  • Normalize by torso length.
  • Use derivative and threshold crossing with hysteresis.

5. Composite signals and classifiers

• What it measures: combinations of the above features fed into a small decision tree or a compact ML classifier trained to predict rep phase.
• Why it works: some exercises produce ambiguous readings on single features; a lightweight classifier can learn robust combinations while still running on-device.
• How to compute:
  • Build a feature vector per frame or a short temporal window: torso angle, torso displacement, normalized wrist/ankle positions, velocity features.
  • Train or tune a small logistic regression or decision tree using labeled rep phases collected from a small on-device annotation run.
  • Implement inference as a few arithmetic operations so it runs with minimal overhead on-device.

Signal selection per exercise

• Squats: normalized knee or hip vertical travel relative to torso length, torso axis orientation for coarse gating.
• Push-ups: torso axis orientation for gating horizontal vs upright; normalized wrist or chest vertical travel along camera axis when front corner shots are common.
• Lunges: differential leg displacement; combine torso orientation to filter false positives.
• Crunches: torso curl angle and torso tip displacement.

Do not hard-code a single signal per exercise without living tests. Real users expose edge cases: different camera angles, clothing obscuring joints, multiple people in the frame. Expose all candidate signals in a debug panel and pick the one that remains stable in the current camera arrangement.

How to handle 2D perspective and monocular depth

Monocular pose models give x and y coordinates in image space and often provide a pseudo z value. Treat z as a soft hint, not a measurement. The pseudo depth returned by many pose models is subject to scale ambiguity and jitter.

Practical steps:

• Normalize spatial features by torso length or pose bounding box. Doing so removes dependence on pixel scale and reduces the impact of distance to camera.
• Use relative movement rather than absolute position whenever possible. For example, a squat is defined by lowering the hips relative to shoulders, not by an absolute y coordinate.
• Where depth changes matter (e.g., when the user moves along the camera axis), use temporal smoothing to filter out small z perturbations and monitor for large z drift that indicates the user moving significantly toward or away from the camera. Prompt the user to re-center if drift persists.

Device tilt and camera rotation

• Phones tilted or rotated produce coordinate transforms that can be normalized by estimating the gravity vector from accelerometer data. Use platform APIs to determine the camera orientation and rotate keypoints accordingly before computing signals.
• If you avoid using IMU data for privacy or simplicity, instruct users to orient the device upright and provide on-screen guidance for a recommended camera placement.

Occlusion handling

• Occlusion is common, especially for push-ups with chest near the floor or when hands are obscured. Monitor keypoint confidence scores returned by the pose model and degrade gracefully: switch to signals that use visible keypoints or pause counting and show a brief message asking the user to re-adjust.

Building an on-device pipeline in Flutter

Flutter provides the cross-platform foundation, but building a low-latency camera + inference pipeline requires careful threading and isolation. Below is a high-level architecture used successfully in production.

1. Camera capture and frame preprocessing

• Capture frames at a consistent resolution and target frame rate. Lower resolution reduces CPU/GPU load and raises inference throughput; pick the smallest resolution that preserves keypoint accuracy (e.g., 256–320 px on the short edge).
• Preprocess frames to the model's expected input: resize, rotate according to camera orientation, and normalize pixel intensities.

2. Background inference isolate

• Run pose model inference off the main UI thread to avoid jank. In Flutter, use Isolates or platform-specific channels to move heavy work into background threads.
• Use a frame buffer with latest-frame semantics: drop older frames if inference can't keep up to avoid building latency.

3. Lightweight pose model with tflite

• Use tflite_flutter to run a quantized or float TFLite pose model such as MoveNet or BlazePose converted to TFLite.
• Choose delegates appropriate to the device: NNAPI on Android where available, Core ML delegate on iOS, or GPU delegates for supported devices. Fall back to CPU inference for older devices.
• Keep model size small. Quantize weights and prune unnecessary operations. A 3–20 ms inference target per frame on modern devices keeps feedback responsive.

4. Signal extraction and rep-state machine

• After keypoints are available, compute normalized features and update smoothed signals.
• The rep state machine logic should be simple and deterministic:
  • WAITING: detect start of movement by thresholding a velocity or displacement feature. Switch to DOWN or UP depending on movement direction.
  • IN_PHASE: monitor for crossing the opposite threshold to complete rep.
  • LOCKOUT: after a count, lock counting for a short window to prevent double-counting from bounce.
• Expose debug hooks to visualize raw keypoints, smoothed signals and state transitions on-device in real time.

5. Local persistence and cloud sync

• Record counts in a local database (e.g., SQLite). This ensures the app feels instant and continues to work offline.
• Cloud sync should be a fire-and-forget background operation that sends only numeric summaries and metadata, never raw frames. This preserves privacy while enabling cross-device continuity.

6. UI feedback

• Provide immediate visual and haptic feedback on each counted rep. Latency under ~100 ms between physical motion and on-screen feedback preserves the "game loop".
• Visualize signal quality and keypoint confidence in a developer/debug mode so testers can spot when keypoints are unreliable.

7. Battery and thermal management

• Running inference continuously is power-hungry. Offer sample-rate throttling or dynamic frame skipping when battery is low or device heats.
• Provide a low-power mode that reduces inference rate and relies more on slower but stable signals.

Practical debugging: hot-swap signals and on-device A/B

A blind trust in offline testing or spreadsheets leads to brittle systems. The only reliable path to production accuracy is live, on-device experimentation.

Implement a developer-facing debug switch to hot-swap among candidate signals. Each signal should be selectable without restarting the app. When testing with a user, you can toggle signals, observe the raw keypoints and smoothed traces, and immediately see which signal holds up.

A/B testing on-device

• Build a simple A/B experiment framework that randomly chooses a signal strategy for a user session and logs metadata about which strategy was active and whether the final count matched a small held-out human-labeled validation sample.
• Use these logs to compute per-signal accuracy in the wild without ever uploading video frames. Store only numeric summaries and anonymized diagnostic metrics.

Instrumentation and metrics

• Log keypoint confidence, count of detected reps, false-positive and false-negative indicators (e.g., if movement suggests a rep but state machine didn't transition), and device conditions (CPU load, camera resolution).
• Aggregate metrics server-side to identify problematic device classes or common body/camera configurations that cause failure.

On-device labeling for small-scale supervised tuning

• Allow a short "labeling" mode where the user taps when they begin and end sets or confirms rep counts after a session. This produces lightweight labels that help tune thresholds and small classifiers without capturing video.

Real-world debugging anecdotes

• A user propped a phone on the floor at a 30° angle; knee-angle thresholds that looked perfect in studio demos failed entirely. Switching to torso displacement normalized by torso length recovered correct counts.
• A push-up set filmed from a front corner showed shoulders, hips and heels not collinear on the image plane. The body-line gate stopped counting mid-set. Torso-axis orientation complemented by wrist displacement produced robust detection.

These anecdotes underline the necessity of live testing across diverse setups.

Trade-offs: privacy, accuracy, and latency

On-device counting trades server-side compute and potentially higher accuracy for privacy and responsiveness. Decide on these trade-offs explicitly.

Privacy-first restrictions

• No raw frames leave the phone. Store only aggregated metadata and telemetry. Users choose this trade-off for good reasons; many will not tolerate cameras streaming to servers.
• Some improvements require labeled footage for training. Use explicit opt-in flows for users who want to provide sample footage, with clear benefit explanations and secure upload practices.

Accuracy vs model size

• Larger models typically yield more accurate keypoints but use more CPU/GPU and battery. Optimize the model and inference pipeline to hit acceptable accuracy with low latency.
• Consider a small server-side fallback for power users who opt-in: upload encrypted, anonymized clips for offline labeling and model improvement.

Latency constraints

• Users require near-instant feedback. Keep end-to-end latency (camera capture → inference → signal processing → UI update) under 100 ms for a satisfying experience.
• Use latest-frame semantics and drop frames under heavy load rather than buffering.

Handling edge cases

• Multiple people in frame: detect multiple poses and select the primary one nearest to screen center or using user selection.
• Occlusion: fall back to alternate signals that rely on visible keypoints or pause counting with an on-screen prompt.
• Unexpected movement: if the system detects large z drift or sudden orientation changes, enter a recalibration state and ask the user to reposition their device.

Model and inference choices: what to run on-device

Several compact pose estimation models work well on mobile:

• MoveNet (SinglePose Lightning and Thunder variants): extremely fast and accurate; Lightning runs in single-digit ms on modern devices.
• MediaPipe BlazePose: provides a rich set of keypoints and a pseudo-z coordinate; widely used for fitness applications.
• Lightweight or custom TFLite models: when you need a tailored architecture or lower compute footprint.

Choosing a model:

• Measure inference latency on target devices and pick the model that meets real-time constraints.
• Use quantization to reduce model size and speed up inference, accepting a small accuracy loss.
• Consider a hybrid delegate strategy: CPU fallback for older devices and NNAPI/GPU delegates for newer ones.

Optimization tactics:

• Resize input to the smallest acceptable resolution for stable keypoint detection.
• Batch operations when possible and avoid per-frame memory allocations.
• Reuse preallocated byte buffers for input tensors.

On iOS, Core ML delegates can significantly accelerate inference. On Android, NNAPI or GPU delegates provide speedups on supported devices. Test on a representative device matrix—different SoCs and OS versions vary widely.

State machine design and anti-bounce hysteresis

A deterministic state machine prevents double-counting and makes behavior predictable. Design it around clear phases and hysteresis thresholds.

State machine example:

• IDLE: signals are stable and no rep is detected.
• DOWN (or FLEX): signal crosses the down_enter threshold.
• UP (or EXTEND): signal returns above the up_enter threshold.
• LOCKOUT: a brief window after a rep to suppress bounce or micro-movements.

Implementation details:

• Dual thresholds: down_enter < down_exit and up_enter > up_exit to create hysteresis bands.
• Velocity checks: require a minimum rate of change when entering DOWN to avoid counting slow drifts.
• Minimum rep time: enforce a minimum time between transitions to avoid counting quick oscillations.
• Confidence gating: only transition if keypoint confidence across the relevant joints exceeds a threshold.

This deterministic design simplifies debugging and yields reproducible behavior across devices.

Normalization and personalization

Population heterogeneity—height, limb proportions, camera distance—requires normalization and optional personalization.

Normalization recipes:

• Use torso length (distance between shoulders and hips) as the canonical body unit.
• Normalize vertical displacements and joint-to-joint distances by torso length or bounding-box height.
• Express velocities as normalized units per second.

Personalization strategies:

• At first-run, prompt a short calibration routine: stand at the recommended camera position, perform a slow squat or push-up, and record baseline torso length and range-of-motion. Use these to tune thresholds.
• Allow users to choose sensitivity or difficulty level; higher sensitivity yields fewer false negatives but more false positives.

Personalized thresholds improve accuracy but add friction. Offer calibration as optional and lean on normalized, conservative defaults for the majority of users.

Evaluation and metrics: how to measure success

Counting accuracy is measurable and should be tracked using objective metrics.

Key metrics:

• Rep accuracy: (true positives) / (true positives + false positives + false negatives) across labeled sessions.
• False positive rate: number of extra counts per set or per minute.
• False negative rate: number of missed reps per set.
• Latency: median and 95th percentile time from physical rep completion to UI feedback.
• Device resource metrics: average CPU, GPU utilization and battery drain during a session.

Evaluation methods:

• Human-labeled ground truth: gather small labeled sessions where users tap to mark actual rep boundaries or annotate recordings (with consent).
• Synthetic tests: use motion-capture data projected into synthetic camera views to test perspective robustness; helpful for edge-case exploration but does not replace real-world testing.
• On-device A/B telemetry: compare different signals across users and aggregate counts per workout to identify biases and device-specific failures.

Set target thresholds that align with product goals. For a consumer game, a slightly lower precision may be acceptable if the experience feels responsive and fun. For a training app promising strict form tracking, higher accuracy is required.

Combining IMU and camera: when fusion makes sense

A single camera can fail in certain scenarios. Inertial sensors (accelerometer, gyroscope) provide complementary information and enable more robust detection when fused with vision.

Fusion use cases:

• Reduce ambiguity when the user moves along the camera axis: combined data gives a stronger signal of vertical motion.
• Improve device orientation detection and compensate for phone tilt without relying on camera alignment.
• Lower reliance on z from the pose model by detecting motion phases with the accelerometer.

Privacy and battery trade-offs:

• IMU data adds little privacy risk compared to video and has low energy overhead.
• Ensure signal fusion preserves the privacy promise by never recording raw IMU streams for later analysis without consent.

Implementation tip:

• Use IMU-derived gravity vector to correct coordinate transforms for the camera keypoints, delivering more consistent measurements across device orientations.

Handling multiple exercises and extensibility

A production system needs to support many exercises and evolve without massive rewrite.

Exercise registry pattern:

• Maintain a registry of exercise configurations. Each entry includes:
  • Candidate signals prioritized for default use.
  • Thresholds or tunable parameters expressed in normalized units.
  • State machine parameters (min rep time, lockout duration).
  • Optional small classifiers or model references.
• Add new exercises by composing existing signals and tuning parameters in a short on-device test cycle.

Automatic exercise detection

• Implement a lightweight classifier that uses torso orientation and a small temporal window of features to suggest likely exercise classes (standing vs horizontal vs seated).
• Use soft gating: the classifier suggests candidate exercises but the rep system falls back to the one with the highest signal quality or with user confirmation.

Backward compatibility

• Store per-user calibration and preferences in the local database. When updating models or adjusting thresholds, migrate settings conservatively to avoid disrupting users mid-session.

Future directions and advanced techniques

Beyond the production baseline, several directions promise improved robustness:

1. Temporal convolutional or recurrent models

• Feed sequences of normalized keypoint features into a small temporal model (TCN, LSTM) to predict rep phase. These models learn temporal patterns and can handle noisy frames more gracefully than frame-wise heuristics.
• Keep models small and quantized for on-device inference.

2. Personalization via lightweight transfer learning

• Collect short labeled snippets from willing users and fine-tune a tiny classifier on-device. This adapts thresholds to an individual's motion style.

3. Multi-camera or multi-view

• When available (e.g., tablet with front and rear cameras), combine multiple views to recover depth and reduce perspective errors. Not practical for most phones but highly accurate where possible.

4. Pose refinement with temporal smoothing and physics priors

• Apply kinematic priors to reduce impossible joint configurations and improve keypoint stability, especially near occlusions.

5. Edge server augmentation

• Offer an opt-in cloud augmentation where anonymized, encrypted keypoint sequences (not raw frames) are sent to a server for batch processing and improved model updates. This preserves much of the privacy while enabling large-scale improvements.

Each advanced technique adds complexity and trade-offs; prioritize methods that deliver the most tangible improvement relative to the engineering cost.

Production experiences: what broke and how we recovered

Several failures during deployment taught practical lessons:

• Camera placement variability: users propping phones at odd heights produced inconsistent angle readings. Recovery: normalize by torso length and add torso-axis gating to detect exercise orientation.
• Double counting from jittery monocular z: naive z thresholds produced rapid flips. Recovery: treat z as a hint, smooth aggressively and rely more on normalized 2D displacements.
• Silent non-counts during sets: the body-line gate used for push-ups removed counting mid-set for corner shots. Recovery: use torso axis orientation which tolerates perspective bending.
• Performance bottlenecks on low-end devices: inference crushed CPU and drained battery. Recovery: select smaller models, reduce input resolution, and support dynamic frame-skipping modes.

These fixes converged to a consistent pattern: prefer robust, low-dimensional features; normalize; smooth; use hysteresis; and enable rapid on-device iteration.

Deployment and maintenance considerations

Operational practices prevent regressions and keep the system healthy.

• Continuous device compatibility testing: maintain a device matrix of representative phones and run nightly inference benchmarks to detect regressions as the model or pipeline changes.
• Telemetry and health signals: collect anonymized counts, failure modes, and device performance metrics. Use these to prioritize fixes.
• Gradual rollout: introduce adjusted thresholds or new models behind feature flags and roll them out progressively, monitoring user metrics for anomalies.
• Clear privacy policy: explain what is processed on-device, what numeric data may be transmitted, and obtain explicit consent for any data collection beyond that.
• User education: provide onboarding guidance for camera placement, example videos, and a quick calibration flow to maximize accuracy.

Final takeaways for developers building on-device camera ML

• Expect 2D perspective to break any feature relying on absolute angles or colinearity. Build signals that are invariant or normalized.
• Treat z from monocular models as a noisy hint; do not threshold it without smoothing and secondary validation.
• Provide a debug interface that lets you hot-swap rep signals on-device. The best solution emerges from testing in real rooms, not spreadsheets.
• Use a small deterministic state machine with hysteresis to avoid double counts and make behavior predictable.
• Optimize the inference pipeline for latency and battery. Small, quantized models plus proper delegates deliver real-time performance on modern phones.
• Prioritize privacy. Keep raw frames strictly local; only sync lightweight numeric summaries.
• Instrument, test across devices and camera placements, and roll out changes gradually.

These engineering patterns apply beyond fitness. Any application that needs motion cues from a single camera—dance coaching, physical therapy, or interactive games—benefits from a signal-first approach, robust normalization, and the ability to iterate with live users.

FAQ

Q: Which pose model should I use on mobile for rep counting? A: MoveNet and BlazePose are solid choices. MoveNet (Lightning) is extremely fast and accurate for single-person pose estimation and suits real-time needs. BlazePose provides more keypoints and a pseudo-z channel that can help with certain signals. Choose the model that meets your latency and accuracy targets on representative devices, then quantize to reduce footprint.

Q: Should I trust the z coordinate from monocular pose models? A: No. Treat the pseudo-z as a soft hint. Monocular depth estimates are noisy and can introduce more problems than they solve if used for hard thresholds. Prefer normalized 2D measures and rely on z only as corroborative evidence after smoothing.

Q: How do I avoid double-counting reps caused by jitter? A: Implement hysteresis in your state machine using dual thresholds (enter/exit) and a lockout window after each counted rep. Smooth the underlying signal with an EMA or Savitzky–Golay filter to remove high-frequency jitter before thresholding.

Q: Can I run inference continuously without killing the battery? A: Continuous inference consumes power. Reduce the input resolution, select a smaller model, use hardware delegates (NNAPI/Core ML/GPU), and implement dynamic frame skipping or low-power modes. Offer a quality vs battery trade-off setting and measure real-world battery impact across devices.

Q: How do I handle different camera placements and user heights? A: Normalize distances and displacements by torso length or pose bounding-box height. Provide a short optional calibration routine to capture personalized ranges. Offer on-screen coaching to guide users to recommended camera positions.

Q: What if multiple people are in the frame? A: Detect multiple poses and choose the primary one by proximity to the center, by user selection, or by the highest confidence. If ambiguity remains, ask the user to reframe.

Q: How do I test and improve detection without collecting video? A: Use on-device A/B telemetry that logs numeric summaries and diagnostic metrics only. Implement small on-device labeling flows where users confirm counts manually. For larger improvements, request explicit opt-in to upload anonymized samples or keypoint sequences rather than raw frames.

Q: Is it worth adding IMU fusion? A: IMU fusion improves robustness for some edge cases—device tilt compensation, axis drift, and ambiguous camera motion. IMU data is low-risk for privacy and low-cost in power usage. Use IMU fusion to correct coordinate transforms and improve movement phase detection if needed.

Q: Can I support many exercises without building a model for each? A: Yes. Compose a library of normalized signals and state machine templates. Configure exercises by selecting and tuning these building blocks. For difficult cases, add a small per-exercise classifier or lightweight temporal model.

Q: How do I maintain accuracy after model updates? A: Roll out updates behind feature flags, measure telemetry for selected cohorts, and provide a fallback mode. Maintain a device compatibility test suite that checks inference latency and accuracy on representative hardware.

Q: What are good smoothing parameters? A: Parameters depend on frame rate and motion speed. As a starting point at 30 FPS, an EMA alpha of 0.08–0.15 for position signals and a faster alpha for velocity works well. Use Savitzky–Golay filters for better derivative preservation when computing velocity, and tune using labeled sessions.

Q: Where should I put the debug tools? A: Integrate a developer-only panel that shows keypoints, raw and smoothed signals, state transitions, and a toggle for candidate signals. Make it easy to enable via a secret gesture or developer settings so you can test in the field without exposing it to end users.

Q: How precise does rep counting need to be? A: It depends on product goals. Gamified experiences tolerate minor inaccuracies; training and rehabilitation apps demand higher fidelity. Define target metrics early and validate against human-labeled ground truth.

Q: Can I open-source the rep-counting logic? A: You can open-source signal extraction and state-machine logic, but be cautious with any user telemetry or labeled datasets that contain sensitive information. Provide clear documentation and privacy guidance if you release reusable components.

Q: What final piece of advice matters most? A: Build for the real world. Lab demos are deceivingly easy. Rely on normalized, smoothed temporal signals, instrument aggressively, and iterate on-device with actual users. That workflow is the difference between a demo and a reliable, private rep-counting feature.