Table of Contents

1. Key Highlights
2. Introduction
3. System architecture and routine design
4. From detection to real-time technique correction
5. Efficient angle calculation using the dot product
6. Where pose estimation still matters
7. Rep counting and finite-state logic
8. Technique evaluation: angular thresholds and camera independence
9. Handling noisy landmarks with temporal aggregation
10. Feedback, alarm system, and error accounting
11. UX/UI and turning a raw system into a website
12. Performance bottlenecks and development challenges
13. Practical considerations before publication
14. Possible technical improvements and roadmap
15. How this sits among commercial offerings
16. Key lessons and practical tips for developers
17. Next steps: roadmap toward a public product
18. FAQ

Key Highlights

• A prototype evolved into a complete workout system that guides users through routines, counts reps, evaluates technique using pose landmarks, and issues real-time feedback.
• The implementation relies on angle calculations (dot product) for camera-independent thresholds, temporal aggregation to handle noisy pose estimates, and a Streamlit front end for an interactive experience.
• Key challenges include MediaPipe noise, frame-processing lag on web interfaces, and the need for robust thresholds and user testing before public deployment.

Introduction

Unsupervised exercise often produces flawed biomechanics. Even high-repetition sessions become counterproductive when form is poor: a single hundred squats performed incorrectly can damage knees and joints. Empirical studies and gym surveys show that a significant share of people sustaining training injuries do so because form goes unchecked. Addressing that gap motivated a recent project that turned a pose-detection proof-of-concept into a working system: a camera-driven workout coach that assigns exercises, counts repetitions, evaluates technique in real time, produces audible alarms for mistakes, and reports a performance summary at the end.

The core design trade-offs are simple but consequential. Classify every movement on the fly and then evaluate it, or prescribe the movement and immediately assess technique? Perform an isolated per-frame rule check, or aggregate evidence across a temporal buffer to overcome noisy pose estimates? Where possible, compute with the lightest operations to keep latency low and preserve a responsive user experience. The implementation described here follows a pragmatic path shaped by those trade-offs: use pose landmarks from MediaPipe, compute joint angles via dot product to make thresholds camera-independent, count reps using state transitions, and smooth decisions with short temporal windows for difficult movements like lunges.

The next sections explain how the system is organized, why particular algorithmic choices were made, how noise and lag were addressed, and which improvements still remain before public release. Practical examples and code excerpts illustrate the methods so developers and trainers can adapt the approach for their own applications.

System architecture and routine design

The system divides an entire workout into short, structured blocks rather than treating the session as a continuous stream of undifferentiated movement. The routine model is intentionally simple: a sequence of exercises grouped by purpose, with fixed short rests between exercises to allow transitions and reduce false positives during movement changes.

Typical routine structure:

• Warm-up: neck rolls, high knees, shoulder mobility.
• Lower body: squats, right and left forward lunges.
• Core and stability: bird dog, hip abduction, glute bridge, plank.

After each exercise the system enforces a 15- or 30-second rest period. These micro-breaks reduce the probability that the detector misclassifies a brief transition as a poor repetition. They also simplify the logic: when the routine sets the step to "squat", the system does not need to run a heavy classifier to determine what the user is doing; it only needs to evaluate how well they perform that specific exercise.

This step-driven design yields two main operational advantages: reduced classification overhead and clearer program flow. The system carries a per-step function that consumes pose landmarks and returns both a textual feedback label and a running rep count. That separation—routine state controls which evaluator runs—keeps the script straightforward and minimizes conditional complexity.

From detection to real-time technique correction

Early prototypes attempted to detect the performed exercise first, then evaluate technique. That approach required classifying between squats, lunges, planks, etc., which added latency, compounded errors when movements overlapped, and forced the evaluator to consider both what the user was doing and how they did it.

The alternative implemented here is prescriptive: the routine tells the user what to do. The evaluator only determines whether the user meets the target positions and transitions for that exercise. Removing the classification step simplifies the evaluator and lowers the processing budget per frame. For similar lower-body movements that share joint-angle features (for example, both squats and lunges use knee angle around 90 degrees), the system no longer needs to reason about leg orientation first; it jumps directly into technique checks.

Operationally, each routine step is mapped to a function. Example pseudocode: if step["id"] == "squat": feedback_label, squats_total = evaluate_squat(landmarks, mp_pose, frame) current_reps = squats_total

That direct mapping improves responsiveness and reduces opportunities for misclassification to cascade into incorrect feedback.

Efficient angle calculation using the dot product

Evaluating technique requires joint angles. The requirement is strict: angle measurements must be invariant to the user’s distance from the camera and cheap enough to compute every video frame.

Computing an angle between three 2D points A, B, C (where the angle is at B) can be done in several ways. Common trig-based methods use arctangent functions (atan2), but those calls are relatively expensive when executed many times per second across multiple joints. The implementation here uses a vector dot-product approach to obtain the cosine of the angle, followed by arccos to get the angle in degrees. The heavy-lifting arccos is executed only once per angle evaluation, and the vector math is simple and branch-free.

The function concept:

• Form vectors BA and BC.
• Compute the dot product BA · BC.
• Compute magnitudes |BA| and |BC|.
• cos(theta) = (BA · BC) / (|BA| |BC|)
• theta = acos(clamp(cos(theta), -1, 1)) in degrees.

Robust implementation must guard against zero-length vectors (when landmarks coincide or detection failed). Clamping the cosine value avoids numerical issues that would make acos throw exceptions or return nan.

Python-style pseudocode: def calculate_angle(A, B, C): BAx = A[0] - B[0] BAy = A[1] - B[1] BCx = C[0] - B[0] BCy = C[1] - B[1]

dot = BAx * BCx + BAy * BCy
magBA = math.sqrt(BAx**2 + BAy**2)
magBC = math.sqrt(BCx**2 + BCy**2)

if magBA == 0 or magBC == 0:
    return 0

cos_angle = dot / (magBA * magBC)
cos_angle = max(-1, min(1, cos_angle))
return math.degrees(math.acos(cos_angle))

This method returns angles in the 0–180° range and remains valid whether the subject is close to or far from the camera, because it depends on relative vector directions rather than absolute pixel distances.

Why this approach matters in practice:

• It keeps computational cost low enough for per-frame evaluation in Python.
• It makes thresholds consistent across different camera placements.
• It simplifies debugging: angles are intuitive and map to coachable cues (e.g., “knees at 90°”).

Common pitfalls and fixes:

• Forgetting to clamp the cosine produces nan values for acos under small numerical errors.
• Copy-pasting angular calculations into multiple places bloats code and prevents consistent improvements. Extracting a single calculate_angle function both reduces bugs and improves runtime performance.

Where pose estimation still matters

Prescribing the exercise obviates heavy classification, but pose estimates remain essential in selective checks. For some movements, the system must confirm the body's orientation before evaluating joint angles.

Plank example: A plank should be a horizontal alignment of shoulders and hips. If the system starts evaluating plank mechanics while the user is still standing, it will trigger false alarms. A reliable heuristic is to compare horizontal (dx) and vertical (dy) distances between the shoulder and hip landmarks. If dx > dy, the torso is more horizontal than vertical, suggesting the user is lying prone or supine and ready for plank evaluation.

dx = abs(left_shoulder_x - left_hip_x) dy = abs(left_shoulder_y - left_hip_y) lying = dx > dy

For hip abduction or side-lying glute bridges, using the shoulders as the horizontal spread indicator is useful. If the left-right shoulder separation is greater in the x-axis than in the y-axis (abs(RS.y - LS.y) < abs(RS.x - LS.x)), the subject is likely lying on their side and aligned for the exercise.

Edge cases:

• Camera angle: If the camera is tilted significantly, horizontal/vertical comparisons may be less reliable.
• Partial occlusions: When arms obscure shoulders, the heuristic fails. Supplementary checks like confidence scores from the pose model can help decide whether to evaluate or not.

The key is selective use of pose orientation checks: run them only where an orientation condition is essential to avoid false positives.

Rep counting and finite-state logic

Counting repetitions sounds trivial but becomes nuanced when pose estimation can jitter and when users hold a position for a few seconds. Simple edge detection on angles leads to overcounting if the bottom position is held. The implemented approach tracks discrete states such as "down" and "up" for each exercise, then counts a rep only when a full cycle completes.

Squat counting example:

• When both knee angles shrink below a predefined "down" threshold, set squats_down = True.
• When both knee angles return above an "up" threshold and squats_down == True, increment the rep counter and reset squats_down = False.

Pseudocode fragment:

detects a squat

if in_squat: if not squats_down and left_knee_angle < down_threshold and right_knee_angle < down_threshold: squats_down = True return "Squat", squats_total

detects when the squat is finished -> +1 rep

if (left_knee_angle > up_threshold and right_knee_angle > up_threshold and squats_down): squats_down = False squats_total += 1

This hysteresis—distinct down and up thresholds—prevents oscillation near a single threshold from causing multiple counts. It also avoids incrementing while the user remains at the bottom.

Practical considerations:

• Choose thresholds to match typical biomechanics. For example, down_threshold ≈ 100°–120° for shallow squats or ≈ 80°–90° for deep squats.
• Provide visual feedback to the user about what threshold the system expects, so users can self-correct.
• Make thresholds configurable or adaptive across sessions to accommodate mobility differences between users.

Technique evaluation: angular thresholds and camera independence

Feedback must be actionable. A label that reads “Bad Lunge: knee over toe” is more helpful than a raw angle number. The system uses angle thresholds and simple conditional rules to generate concise advice. Angular rules are preferable to raw pixel distances because they remain stable when the user moves closer to or farther from the camera.

Plank example (rules simplified):

• Arm alignment: if angle at shoulder > threshold or angle1 < another threshold => "Arms: incorrect position".
• Hip alignment: if hip angles fall outside allowed range => "Bad Hips: not aligned".
• Otherwise => "Right Plank".

Code-style rules might look like: if angle >= 100 or angle1 < 120: return "Arms: incorrect position" if 80 < angle < 100 or angle <= 60: return "Bad Arms: adjust alignment" if angle1 > 180 or 120 <= angle1 < 150: return "Bad Hips: not aligned" else: return "Right Plank"

Careful attention to numeric ranges matters. For example, the angle function returns values in 0–180°, so expecting values beyond that is a logic error. Debugging such issues is simpler when the system logs angle measurements during development and highlights impossible values.

Design principle: craft feedback text that reflects the specific failure mode and, where possible, deliver a short corrective cue (e.g., “raise hips,” “keep knees aligned with toes,” “tighten core”).

Handling noisy landmarks with temporal aggregation

MediaPipe and similar single-camera pose estimators perform well but can produce jitter and occasional landmark misassignments. Evaluating every frame independently often leads to unstable feedback, especially for movements that involve several joints or slower transitions. Solutions include temporal smoothing of landmark positions and aggregating per-frame “good vs bad” evaluations across a buffer.

Lunges present a case in point. A lunge requires evaluating more than one angle (knee, hip, torso). Frame-by-frame judgments vary because small detection errors can flip a judgment from good to bad transiently. The implemented approach uses a short buffer that records whether each recent frame passed the rule-set for the lunge. The system calculates the ratio of good frames to total frames in the buffer; if the ratio exceeds a threshold (for example, 0.8), the lunge is considered successful. If the ratio falls below threshold, the lunge is considered incorrect and a rep does not get counted.

Pseudocode: lunge_buffer.append(is_good) good_ratio = sum(lunge_buffer) / len(lunge_buffer) if good_ratio > 0.8: label = f"{front_leg} Lunge: Good"

This method reduces false positives from a few noisy frames and ensures consistency with how a human coach would judge a repetition—by looking at the movement over a short time window rather than a single frozen frame.

Other temporal techniques:

• Exponential smoothing or moving average on landmark coordinates to stabilize angles.
• Median filtering across a short window to reject spikes due to misdetections.
• Per-landmark confidence weighting using MediaPipe’s visibility or confidence outputs; downweight low-confidence landmarks.

Advice from experienced computer-vision practitioners can be decisive. The shift toward temporal aggregation for lunges in this project came on the recommendation of a more experienced colleague and substantially improved reliability.

Feedback, alarm system, and error accounting

Delivering feedback requires logic not just to detect poor technique but to present it in a way that motivates correct form without becoming annoying. The system generates a textual label for each frame or buffer evaluation; if that label includes "Bad," an audible alarm plays and the system increments the error count for the current exercise. The alarm is debounced to avoid repeated triggers because MediaPipe's noise would otherwise cause rapid, repetitive alarms.

Alarm logic specifics:

• When a label containing "Bad" is detected and the exercise hasn’t already been marked as bad in the current rep, increment exercise_errors for that exercise and set a was_bad flag for the step.
• Start the alarm only if it is not already playing.
• When the label no longer contains "Bad," start a short timer (0.7 seconds). If a good period persists longer than that debounce time and the alarm is playing, stop the alarm and clear was_bad.

Pseudocode fragment: if feedback_label and "Bad" in feedback_label: if not was_bad[step_id]: exercise_errors[step_id] += 1 was_bad[step_id] = True if not alarm_playing: alarm_playing = True play_alarm() else: was_bad[step_id] = False if good_elapsed > 0.7 and alarm_playing: alarm_playing = False stop_alarm()

This design ensures that:

• Small adjustments or fleeting noise do not rapidly toggle the alarm.
• Only meaningful errors count toward the user's final score.
• The system tracks error counts per exercise and computes correctness percentages at the end of the session.

End-of-workout reporting provides a simple performance breakdown: errors per exercise, percentage correct for each exercise, and an overall score. These metrics enable users to identify weak areas and focus future practice where it matters.

UX/UI and turning a raw system into a website

Working code is useful; an accessible interface makes it usable. The project uses Streamlit as the web framework to wrap the camera input, pose overlay, controls, and results into a single, interactive page. Streamlit offers fast prototyping and simple state management in Python, which is attractive for a single-developer project.

Interface elements:

• Three primary controls: Start workout, Pause/Stop, Reset workout.
• Dark and light themes to match user preferences.
• During an active workout, an additional “Skip exercise” button appears.
• Right-hand panel displays: current exercise name, target reps, completed reps, live feedback text (e.g., “Bad Lunge: too deep”), and a reference image depicting the expected position.

Design focus:

• Clarity: users must see at a glance what’s expected and what came out of the detector.
• Non-intrusiveness: feedback text should not obscure the live video feed.
• Responsiveness: control buttons must react quickly so users can pause or skip without interrupting their flow.

Screenshots from the prototype show a compact, coach-like layout with the video feed occupying the left and state information in the right panel. The reference image is especially useful: humans interpret images faster than textual descriptions, and a short visual cue helps the user self-correct.

Trade-offs:

• Streamlit is convenient but runs everything in Python on the host machine. That means frame processing, landmark calculations, and drawing overlays all execute in the same process, which can cause lag when computational load is high.
• Running locally avoids transmitting video to a server, preserving privacy, but limits scalability and cross-device access.

Performance bottlenecks and development challenges

Creating a working system surfaced a set of practical obstacles that are common when deploying computer vision in real time.

Testing fatigue Iterative testing required repeated actual workouts. During development, the author repeatedly performed exercises deliberately with poor form to trigger alarms. That leads to physical fatigue and demonstrates the real human cost of being both the developer and the test subject. Use external testers where possible to reduce bias and physical strain.

Frame-processing lag Processing every frame, calculating multiple angles, rendering overlays, and updating UI state imposes a heavy load. The Streamlit implementation saw significant memory use and lag, especially for squats. Several strategies mitigate lag:

• Reduce processing frequency: analyze every Nth frame rather than every frame.
• Move non-essential processing (e.g., drawing overlays) to separate threads or asynchronous loops.
• Use lower-resolution video input to reduce per-frame computational cost while preserving relative angle accuracy.
• Offload pose estimation to a GPU or a dedicated inference service when available.
• Optimize Python code paths; avoid repeated allocations inside per-frame loops.

Angles and edge-case logic Angle calculations required multiple iterations to get both performance and correctness. Common issues during development included inadvertent expectations of angle ranges larger than possible (e.g., expecting 190° from a 0–180° function) and duplicating angle calculations across functions, which made bug fixes tedious.

MediaPipe noise and landmark glitches Occasional landmark misassignments—where one limb’s position collapses or gets labeled incorrectly—produce bizarre joint angles. These glitches are somewhat ameliorated by:

• Smoothing: use temporal filters to reduce jitter.
• Confidence gating: skip evaluation when landmark confidence is low.
• Temporal aggregation: count a rep only if the buffer indicates consistent good frames.
• User-level guidelines: instruct users to wear contrast clothing and maintain even lighting.

All of these strategies reduce false positives and improve user trust in the coach.

Practical considerations before publication

The prototype currently runs locally. Turning it into a public-facing product requires attention to several operational, ethical, and engineering issues.

Privacy and data handling Camera-based coaching touches on sensitive user data. Two options exist: local-only processing (no video leaves the user’s device) or cloud processing (server-side inference). Local processing maximizes privacy but limits scalability and performance (depending on device capacity). Cloud processing enables richer models and centralized improvements, but demands robust encryption, clear consent, storage policies, and compliance with regional regulations (GDPR, CCPA).

Latency and mobile support Running a pose pipeline on mobile devices is feasible but requires careful optimization. Considerations:

• Use mobile-optimized models (e.g., TensorFlow Lite versions of pose detectors).
• Limit per-frame computation and rely on optimized native libraries.
• Offload heavy analytics to a server in hybrid setups if privacy policies allow.

Cross-device calibration Different cameras and mounting angles change detection reliability. Consider offering an initial calibration phase where the app prompts the user to assume a few reference poses and derives per-user clamps or offsets. Calibration can improve thresholds and reduce the need for overly conservative rules.

Labeling and thresholds Current thresholds were hand-tuned. For broader deployment, collect labeled data across users, body types, and camera setups to refine thresholds or train a small classifier that predicts “good vs bad” technique per rep. A bootstrap approach:

• Run the rule-based coach and allow users to flag incorrect feedback.
• Sample flagged instances for human review.
• Build a labeled dataset and retrain or augment rule thresholds.

Safety and liability Safety guidance is essential. The coach should include disclaimers and recommend a conservative approach: start with low-intensity routines, consult a healthcare professional for pre-existing conditions, and stop if pain occurs.

Accessibility Provide audio feedback for visually impaired users and ensure the UI is keyboard navigable. Keep text concise and readable.

Possible technical improvements and roadmap

The prototype is a solid foundation with clear next steps.

1. Publish and collect real-user feedback
   • Deploy a controlled beta for a small pool of testers to collect diverse examples.
   • Log anonymized pose traces and feedback where users opt-in.
2. Build multiple routines and personalization
   • Offer routines targeting goals such as mobility, strength, and weight loss.
   • Allow users to input body parameters (height, limb lengths) to refine thresholds.
3. Improve accuracy with hybrid models
   • Complement rule-based logic with supervised classifiers trained on labeled data for technique detection.
   • Use temporal models (LSTM, Temporal Convolutional Networks) to judge full movement quality across time rather than per-frame logic.
4. Advanced smoothing and confidence weighting
   • Implement per-landmark Kalman filters or exponential smoothing.
   • Weight joint angles by landmark visibility/confidence when aggregating measures.
5. Multi-camera and 3D pose
   • Multi-camera setups or multi-view inference produce more robust 3D pose estimates and reduce occlusion errors, but increase hardware complexity.
   • Explore 3D pose estimation models where feasible to measure depth-aware metrics like knee displacement relative to toes.
6. Integrate inertial sensors
   • Combine camera-based pose with wearable IMUs for robust angle estimation on demanding movements.
7. Scalable deployment
   • Offer a hybrid web + native mobile architecture: local inference for privacy-sensitive features and cloud inference for premium advanced coaching options.

How this sits among commercial offerings

Several commercial products attempt live coaching via cameras and sensors. Devices like Mirror, Tempo, and interactive platforms integrate cameras, depth sensors, or motion sensors with curated classes and human coaches. This system differs in several ways:

• Local-first, rule-based evaluation rather than centralized proprietary models.
• Focus on explicit form feedback and per-exercise error accounting rather than class engagement or production-level content.
• Lightweight design targeted for easier replication and experimentation by developers and trainers.

The rule-based approach offers interpretability: a label like “Bad Lunge: too deep” corresponds to a clear geometric reason, making it straightforward to iterate on thresholds and corrections. By contrast, deep-learning classifiers can produce opaque outputs that are harder for trainers to validate without large labeled datasets.

Key lessons and practical tips for developers

• Separate concerns: let the routine state dictate which evaluator runs. That keeps the flow tidy and eliminates expensive classification logic where it is unnecessary.
• Extract reusable math functions: a single calculate_angle function reduces duplication and prevents inconsistent behavior across exercises.
• Use temporal aggregation for complex multi-angle exercises: counting a majority of good frames yields more stable feedback than per-frame labels.
• Debounce audible alarms: short noise bursts or quick adjustments should not become repeated alarms. A 0.5–1.0s debounce window balances sensitivity and annoyance.
• Log extensively during development: capture angles and landmark confidence values in a development log to track down impossible values and edge cases.
• Test with real humans: automated unit tests are useful, but real-world testing with multiple body types, clothing, and lighting remains indispensable.

Next steps: roadmap toward a public product

• Stabilize per-frame latency by profiling and optimizing hot paths; offload heavy computations to optimized C libraries or hardware acceleration.
• Build a small labeled dataset from beta testers to refine thresholds and train auxiliary classifiers to catch failure modes the rules miss.
• Add personalization: adaptive thresholds and routine difficulty adjustments make the system more useful for long-term progression.
• Design user onboarding and safety flows, including clear privacy controls and consent for any optional data collection.

A staged roll-out—first local desktop beta, then closed mobile beta, then public release—lets the team iterate quickly while protecting users and gathering the data necessary for robust, generalizable coaching.

FAQ

Q: How accurate is technique detection with this system? A: Accuracy depends on lighting, camera angle, clothing contrast, and device performance. The rule-based approach yields consistent results when pose landmarks are stable; lunges and multi-angle movements are handled via temporal aggregation to improve reliability. Public accuracy metrics require broader user testing and labeled ground truth.

Q: Can this system run on a smartphone or must it be used on a laptop? A: The core algorithms are lightweight enough for mobile deployment, but the current prototype runs locally in Streamlit on a laptop. For mobile, swap to a mobile-optimized pose model (TensorFlow Lite, MediaPipe on-device) and reduce processing frequency or resolution to meet performance targets.

Q: How does the system avoid counting repeated reps when the user holds the bottom position? A: The rep counter uses state variables (e.g., "down" and "up") and distinct thresholds to define transitions. A rep is counted only when the system detects a full down→up cycle, preventing multiple counts while the user remains stationary.

Q: What steps handle noisy landmark detection from MediaPipe? A: Several measures help: temporal buffers that aggregate per-frame judgments, exponential or median smoothing of landmark positions, confidence gating to avoid evaluating low-confidence landmarks, and a short debounce before turning off alarms.

Q: Are movement thresholds one-size-fits-all? A: Thresholds start as hand-tuned heuristics but should be viewed as starting points. Personalization (height, limb length, mobility) and data-driven calibration improve relevance. Collecting annotated examples from diverse users enables data-driven threshold refinement.

Q: Does video leave the user's device? A: The current implementation runs locally on the host machine. If deployed to a cloud service, transmission and storage policies must be specified, and users should explicitly consent to any data uploads.

Q: Can I add new exercises? A: Yes. The architecture maps routine steps to evaluator functions. Adding an exercise means implementing a new evaluate_EXERCISE(landmarks, ...) function that computes relevant angles and states, then wiring it into the routine sequence.

Q: What improvements are planned next? A: Priorities include publishing a beta to collect real-world feedback, adding multiple goal-oriented routines, improving accuracy via hybrid rule-and-ML models, and enhancing personalization. Performance optimization and robust privacy controls are also top priorities.

Q: Where can I find the code or contact the developer? A: Contacting the project author or checking public repositories (if published) is the standard path. For this prototype, reach out via the developer’s listed professional channels to inquire about access or collaboration.

Q: Is this safe to use without supervision? A: The system is a coaching aid, not a medical device. It helps catch gross form errors but cannot replace a certified trainer or medical advice. Users with pre-existing conditions should consult a healthcare professional before beginning a new exercise program.

This system demonstrates how pragmatic engineering—combining pose estimation, efficient geometry, temporal aggregation, and careful UX—can produce immediate, interpretable feedback for at-home exercise. The prototype’s next evolution will depend on user testing, labeled data for refinement, and engineering work to make the experience smooth across devices while preserving privacy and safety.