RepWise: How a Live AI Gym Coach Uses Gemini and MediaPipe to Rescue Your Workout

Table of Contents

1. Key Highlights
2. Introduction
3. The friction that ends workouts
4. Anatomy of RepWise: From quick setup to mission complete
5. Live coaching: pose detection, rep counting, and form analysis
6. Workout Rescue: decision-making under constraints
7. Gemini’s three roles: live coaching, structured planning, and reasoning
8. Cloud architecture and reliability: Cloud Run, Firestore, and fallbacks
9. Design decisions and UX: beginner vs intermediate branching
10. Implementation details and tech stack
11. Lessons learned and engineering trade-offs
12. Limitations, safety, and privacy considerations
13. Real-world scenarios: rescue in action
14. Practical engineering patterns for similar systems
15. Integration, deployment, and reproducibility
16. Future directions and opportunities
17. What RepWise gets right for beginners—and why that matters
18. Limitations of current models and mitigating strategies
19. Testing and evaluation: measuring success
20. Ethical and regulatory considerations
21. Try RepWise: demo and code
22. FAQ

Key Highlights

• RepWise combines in-browser pose detection with Gemini live models to provide real-time coaching and an adaptive "Workout Rescue" that rewrites sessions when gym conditions change.
• The system emphasizes semantic event streaming, deterministic fallbacks, and a cloud-backed architecture (Cloud Run + Firestore) to keep sessions reliable and responsive.

Introduction

Workouts derail for predictable reasons: a machine is occupied, a sudden energy drop, unfamiliar exercises, or pain that signals something is off. Most fitness apps treat a plan as immutable, leaving users to improvise or quit. RepWise flips that model. It acts like a live coach that never walks away—counting reps, flagging form, and rewriting the remainder of a session the moment real-world constraints appear. The result is a practical, user-centered system that keeps people moving toward their goals even when the gym refuses to cooperate.

This article examines RepWise’s design and implementation, explains why the rescue capability matters, and outlines the engineering choices that make a live, browser-based coach feasible. It draws from the developer’s implementation notes and expands on real-world scenarios, safety and privacy considerations, and practical lessons for anyone building interactive fitness AI.

The friction that ends workouts

Most abandonment looks like friction, not laziness. Common triggers include:

• Equipment availability: The bench press is taken for a long set. A squat rack is occupied. Replacing a specific tool can break the rhythm of an established plan.
• Unexpected fatigue: Energy can drop mid-session for many reasons—sleep debt, poor nutrition, or an earlier unexpected task.
• Pain or discomfort: Discomfort during a movement may indicate poor form or an underlying issue. Continuing without adjustment risks injury.
• Time compression: Plans assume an allotted window. Losing half the session requires triage.
• Cognitive friction: Not recognizing an exercise or feeling unsure about form can sap confidence.

A tracker that merely logs reps and time does nothing when these events occur. A coach steps in: prioritizes safety, preserves training intent, and adapts volume and intensity. RepWise makes that transition programmatic and immediate.

Anatomy of RepWise: From quick setup to mission complete

RepWise’s user flow is deliberately short and context-rich. The app asks for a few high-value inputs up front:

• Experience level (beginner/intermediate)
• Goal (build muscle, lose fat, quick session)
• Available time
• Equipment on hand
• Current energy level
• Contextual notes (crowded gym, low confidence, soreness)

Those inputs let the planner (Gemini 2.5 Flash) generate a mission: a sequence of exercises, warmup sets, coaching cues, target sets and reps, and rest intervals. The novelty starts when the user begins moving.

During the session:

1. The phone camera streams video to an in-browser MediaPipe pose detector. All pose processing occurs client-side to minimize latency and preserve privacy.
2. The browser synthesizes semantic pose events—rep counts, set completions, and form issues—and sends those compact events to a backend via WebSocket.
3. Gemini Live receives the event stream and returns contextual coaching cues in real time. That produces a two-way conversational coaching loop.
4. If a friction event occurs, the user taps Rescue and selects a reason. Gemini then recalculates the remaining plan and returns a revised mission consistent with the original goal.

At session end, RepWise produces a grounded debrief: exercises performed, sets completed, form metrics, and recommendations for the next workout.

Live coaching: pose detection, rep counting, and form analysis

Running pose detection entirely in the browser is a deliberate choice. MediaPipe Tasks Vision provides robust landmark detection and joint-angle computation at usable frame rates on contemporary phones and laptops. Processing locally yields four advantages:

• Reduced latency: No round-trip video upload for basic detection.
• Lower bandwidth: Only semantic events travel to the backend, not video streams.
• Simpler privacy posture: Raw camera data remains on the device unless the user opts in to sharing.
• Resilience: The app can keep counting and cueing even when backend services are slow.

Counting reps and identifying form deviations require translating raw landmarks into semantic events. Consider a squat: the system tracks hip and knee angles, torso lean, and depth relative to the user’s anatomical baseline. Simple thresholds generate structured events such as: { "exercise": "squat", "rep": 8, "form_issue": "insufficient_depth", "timestamp": 1680000000 } Structured events allow the language model to synthesize coaching that’s specific and actionable. Sending raw coordinates would overload the model with low-level data and increase latency for each coaching decision.

Form analysis focuses on common, observable issues: excessive forward lean, knee valgus, insufficient depth, early lockout. For beginners, RepWise lowers the bar for technique and emphasizes reassuring cues. For intermediate users, coaching tightens, emphasizing performance outcomes like tempo and progressive overload.

Workout Rescue: decision-making under constraints

Workout Rescue is the core innovation. Rather than offering a static substitution list, RepWise recalculates the remainder of the session based on the user's unchanged goal and the new constraint. That decision process involves several steps:

1. Classify the constraint: Equipment busy, low energy, short on time, discomfort, unfamiliar exercise, or request for an easier variant.
2. Re-evaluate remaining volume and intensity targets: How much work remains to meaningfully progress toward the goal?
3. Select alternatives that preserve training stimulus: Swap a single-joint exercise for a compound movement if time-limited; lower load or range of motion if discomfort; reduce tempo and volume for low energy.
4. Communicate the change and reasoning: Provide a short justification and coaching cues so the user understands the trade-off.

Example scenarios:

• Equipment busy: The barbell bench is occupied. The model replaces it with dumbbell presses or push-up progressions, preserving horizontal pressing volume while accounting for equipment constraints.
• Short on time: A planned five-exercise circuit becomes a condensed two-exercise compound pair that targets the same muscle groups with higher intensity.
• Discomfort in a movement: RepWise reduces range of motion, swaps to a mechanically safer alternative, or prescribes mobility exercises and a lighter progression.

Rescue decisions are not arbitrary. They prioritize safety and training fidelity. The rewrite aims to deliver an equivalent training dose when possible. When equivalence is not possible—short on time by 50%—the model explains the compromise and sets expectations for the next session.

Gemini’s three roles: live coaching, structured planning, and reasoning

RepWise uses Gemini across three distinct channels, each matched to the task’s characteristics:

1. Gemini Live API (bidi-streaming via ADK): Handles real-time coaching. The system streams semantic events to Gemini and receives coaching messages via WebSocket. This channel prioritizes low-latency back-and-forth for per-rep cues, rest prompts, and safety flags.
2. Gemini 2.5 Flash for structured planning: Generates the initial workout plan, warmup sets, and exercise sequences. The model returns structured JSON that the backend parses into a deterministic plan with sets, reps, and rest.
3. Gemini 2.5 Flash for reasoning and debrief: When a rescue is triggered or a post-workout summary is needed, the model reasons about remaining goals and constraints and creates coherent explanations and next-step recommendations.

Separation matters. Real-time coaching demands a continuous stream with minimal latency and short, directive responses. Planning benefits from richer reasoning and structured outputs. Rescue and debrief tasks require deeper context, supporting the model’s ability to explain trade-offs and provide reliable guidance. Each use case uses the right model and prompt engineering to keep outputs concise and grounded.

Cloud architecture and reliability: Cloud Run, Firestore, and fallbacks

The backend runs on Google Cloud Run. FastAPI serves REST endpoints and handles WebSocket connections for Gemini live streaming. Cloud Run’s container model simplifies deployment and scaling, enabling session-based workloads that can burst during peak gym hours.

Firestore stores session-level data: rep event histories, form flags, session summaries, and an anonymous user profile. Persisting structured events allows retrospective analysis, continuity across sessions, and the generation of useful trends (form improvement, volume progression).

A critical operational choice is deterministic fallbacks. Every external model call has a local fallback that activates if Gemini is unavailable or slow. For example:

• If workout generation fails, a parameterized rule-based generator produces a plausible, conservative session.
• If a live coaching response times out, the backend returns canned cues derived from the exercise template and the most recent semantic event.

These fallbacks serve two purposes: they keep demos reliable and they model a production requirement for user-facing fitness software where interruptions can undermine trust and safety.

Testing practices reflect the need for resilience. The project ships with 111 Pytest tests that cover endpoint behavior, fallback activation, and form detection thresholds. Automated deployment via a single deploy.sh script ensures repeatability and reduces human error.

Design decisions and UX: beginner vs intermediate branching

Small UX differences produce large perceptual shifts. RepWise branches behavior early based on experience level:

• Beginner experience path: Plans fewer exercises, simpler movements, explicit warmup prompts, and reassuring, incremental coaching. Voice and text cues emphasize safety and success, reducing intimidation and cognitive load.
• Intermediate experience path: Plans include denser volume, mixed accessory work, and cues that focus on tempo, progressive overload, and performance metrics.

These variations do more than change words. They alter rest durations, rep ranges, and failure thresholds for form detection. Real-world testing demonstrates that the same algorithm tuned differently for each cohort produces genuinely different user experiences. Beginners report higher confidence; intermediate users report feeling challenged in appropriate ways.

Another design choice: prioritize semantic events over raw data for model communication. The semantic event approach compresses essential information, reduces bandwidth, and produces clearer coaching responses. It also allows the same model prompts to scale across devices with varying camera quality.

Implementation details and tech stack

RepWise uses a compact but effective technology stack chosen for latency, ubiquity, and developer ergonomics:

• Frontend: JavaScript, HTML, CSS. The UI lives entirely in the browser and uses WebSocket to maintain a low-latency channel to the backend.
• Pose detection: MediaPipe Tasks Vision runs client-side at roughly 10 FPS, computing landmark positions and joint angles.
• Backend: Python with FastAPI orchestrates WebSocket connections, Gemini API calls, fallback logic, and data persistence.
• AI: Gemini Live API for streaming coaching; Gemini 2.5 Flash for structured planning and reasoning tasks.
• Deployment: Google Cloud Run containers via Docker. A simple deploy.sh provisions and deploys the stack.
• Persistence: Google Cloud Firestore stores session data and anonymous profiles.
• Testing: Pytest covers endpoint logic and threshold behavior.

This stack minimizes heavy video infrastructure by treating the browser as the sensor layer. The architecture favors small messages—structured semantic events—over large video uploads. That choice allows the system to scale horizontally without exorbitant bandwidth or storage costs.

Lessons learned and engineering trade-offs

Several practical lessons emerged during development:

MediaPipe in the browser is capable. Running pose detection locally at 10 FPS while computing joint angles and rep counting keeps system latency low and the architecture simple. The trade-off is occasional misdetection under complex occlusion or poor lighting. For those edge cases, fallbacks and user prompts to adjust camera angle handle most failures.

Semantic events beat raw data. Structured events like {"exercise":"squat","rep":8,"form_issue":"leaning_forward"} produce richer, faster coaching than pushing raw coordinates into the model. Models respond more effectively to semantic inputs because the signal is already distilled into meaningful attributes.

Deterministic fallbacks are essential for demos and production. Every Gemini call has a local rule-based fallback for workout generation, rescue, warmup, and debrief. Demos never break; users receive predictable, safe behavior when APIs are slow.

Beginner versus intermediate branching changes everything. The same system tuned differently creates a tailored experience. Small backend logic changes—simpler templates, longer rest, softer language—drive large UX differences. That suggests a best practice: treat experience level as a first-class configuration rather than a cosmetic label.

Testing matters. Running an automated test suite covering both happy paths and fallback activation reduced regressions and made iterative prompt/policy changes safer.

Limitations, safety, and privacy considerations

The system is practical but not foolproof. Important limitations and safety considerations include:

Accuracy limitations:

• Pose detectors struggle with occlusions and some clothing or camera angles. Cameras positioned poorly will yield noisy data.
• Fine-grained injury risk assessment is beyond current in-browser vision systems. The app can flag probable issues but cannot diagnose injuries.

User safety:

• The system must avoid encouraging users to push through pain. Rescue pathways that detect discomfort should recommend conservative options and, where appropriate, suggest stopping or consulting a professional.
• Any prescriptive guidance about load or progression should be conservative and include safe failure modes.

Privacy and data handling:

• Processing pose detection locally reduces privacy risk. However, session metadata and semantic events are stored. RepWise uses anonymous profiles by design, but developers must be explicit with users about what is stored and how it is used.
• If enabling video upload or sharing, require opt-in and handle PII and biometric data under strict policies compliant with applicable regulations.

Model hallucinations:

• Language models can invent details or confidently make incorrect suggestions. Ground every coaching or rescue decision to explicit heuristics where possible. Maintain deterministic fallbacks with conservative behavior to reduce risk from model errors.

Legal and ethical considerations:

• Claims about injury prevention or medical advice should be avoided. State clearly that RepWise provides coaching and recommendations, not medical diagnoses.

Real-world scenarios: rescue in action

Concrete examples illustrate the emergency triage RepWise provides.

Scenario 1 — The bench is taken

• Context: Beginner-level chest day. User planned barbell bench press but the bench is occupied.
• Rescue flow: User taps Rescue → selects "Equipment Busy" → the system evaluates remaining training volume for horizontal pressing.
• Outcome: Replacement sequence swaps bench press for incline dumbbell press and push-up variations. Rest intervals adjust slightly. The model explains: "Swapped to incline dumbbells and elevated push-ups to preserve horizontal pressing volume. Use a weight that allows 8–10 controlled reps."

Scenario 2 — Energy crash with 15 minutes left

• Context: Intermediate user halfway through a 45-minute session, now has 15 minutes left.
• Rescue flow: User taps Rescue → selects "Short on Time" and "Low Energy."
• Outcome: The model compresses remaining work to two compound lifts—an AMRAP (as many rounds as possible) of kettlebell swings and goblet squats—focusing on maintaining stimulus with minimal setup. Coaching emphasizes tempo and a conservative rep target to avoid injury while maintaining intensity.

Scenario 3 — Pain with a movement

• Context: User experiences knee pain during lunges.
• Rescue flow: Selects "Discomfort." The system asks clarifying questions (pain location, sharpness).
• Outcome: The system swaps lunges for hip-dominant variations (Romanian deadlifts, glute bridges) and prescribes tempo reduction and mobility cues. It flags the session for a cautious follow-up and recommends seeing a professional if pain persists.

Scenario 4 — "I don't know this exercise"

• Context: A programmed exercise uses a machine the user hasn't seen.
• Rescue flow: User selects "Don't Know This Exercise."
• Outcome: RepWise offers a quick visual cue, a brief instruction (two sentences), and an easier alternative that requires no special equipment, preserving function and time.

These scenarios demonstrate how Rescue functions both as a decision engine and as a translator—turning interruptions into safe continuations rather than session-enders.

Practical engineering patterns for similar systems

If you plan to build a live coaching or real-time adaptive agent, these patterns are useful:

1. Push intelligence closer to the sensor. Processing raw video client-side minimizes latency and bandwidth.
2. Use semantic event schemas. Define a compact, descriptive event schema for rep, set, exercise, and form issues. This decouples model prompts from sensor idiosyncrasies.
3. Keep a deterministic rulebase for critical behaviors. Complement models with rules that govern safety-critical actions.
4. Separate streaming and batch model use. Real-time coaching uses a streaming low-latency model; planning and reasoning use a higher-capacity model with richer context.
5. Instrument aggressively. Persist semantic events and session metadata to enable offline analysis and model improvement. Use those logs to iterate on thresholds and templates.
6. Treat difficulty/experience level as a first-class parameter. Small changes to cues, rest times, and volume have outsized UX impact.
7. Design for degraded network conditions. If the backend is unreachable, keep counting reps locally, store events, and batch-sync later.

Integration, deployment, and reproducibility

RepWise is containerized and deployed to Google Cloud Run. The developer provided a single deploy.sh script to automate provisioning and deployment. This approach simplifies reproducibility: a container image encapsulates dependencies and runtime configuration.

Firestore acts as the canonical store for session data. It supports offline-first patterns, enabling session continuity when devices lose connectivity and sync later. WebSocket-based streaming between the browser and Cloud Run supports low-latency interactions with Gemini Live.

From a cost perspective, the architecture avoids heavy video processing on the server, minimizing storage and egress. Bandwidth is primarily semantic events and occasional diagnostic uploads when users opt in.

Open-source availability (the project repository is public) accelerates community review and contributions. Developers that extend the project should pay careful attention to privacy, security, and compliance for biometric data.

Future directions and opportunities

RepWise demonstrates that adaptive, live coaching is practical. Several logical extensions would increase utility and robustness:

• Multi-user adaptation: Group classes or remote coaching sessions where the model tracks multiple bodies and provides cohort-level adjustments.
• External sensor integration: Combine wearable data (heart rate, power) with pose events to make intensity-based adjustments with more fidelity.
• Personalized progression: Use longitudinal session data to automatically adjust macrocycles and periodization for long-term progression.
• On-device model components: Deploy smaller reasoning components on-device to further reduce latency and handle offline rescue.
• Registered professional integrations: Provide a bridge to human coaches for persistent suggestions flagged by the model, including secure session recordings and form reports.
• Clinical pathways: Partner with physiotherapists to design medically-informed rescue pathways for common complaint patterns, while maintaining a conservative scope to avoid medical practice errors.

Each extension raises new trade-offs: privacy, computational cost, and regulatory complexity. Careful incremental research and testing will be necessary.

What RepWise gets right for beginners—and why that matters

Beginners face unique barriers: unfamiliar vocabulary, technical complexity, and lack of confidence. RepWise addresses these through:

• Simplified plans with fewer exercises and clear progression.
• Explicit warmup sequences and clear, encouraging language.
• Frequent, small success signals—short-term goals that build confidence.
• Conservative defaults on load and complexity to prioritize safety.

These considerations increase the likelihood of long-term adherence. A system that adapts when things go wrong reduces dropout. For beginners, that matters more than marginal improvements in training efficacy.

Limitations of current models and mitigating strategies

Language models are not domain experts by default. They may produce plausible-sounding but unsafe advice. Strategies to mitigate risk:

• Anchor model outputs to explicit constraints and rule-based checks (e.g., limit recommended loads, require conservative movement substitutions).
• Avoid prescriptive medical advice. Use conditional phrasing only when tied to objective thresholds and fallback messaging that recommends human consultation.
• Log and monitor model outputs to catch recurring failure modes. Use offline review by domain experts to refine prompts and guardrails.

These steps make the system safer and help preserve trust.

Testing and evaluation: measuring success

RepWise uses automated tests and manual evaluation to ensure correctness and safety:

• Unit tests for parsing, event handling, and fallback activation.
• Integration tests for end-to-end flows (workout generation → live coaching → rescue → debrief).
• Human-in-the-loop evaluation: domain experts validate coaching cues, rescue logic, and debrief wording.
• Field testing with users across levels of experience and different environments to validate pose robustness under real gym conditions.

Key metrics to track:

• Session completion rate (how often users finish a session despite friction).
• Rescue success rate (percentage of rescues that lead to session continuation).
• Safety incidents (reports of pain or injury reported post-session).
• User sentiment and adherence over time.

These metrics provide a feedback loop to iterate on coaching strategies and system thresholds.

Ethical and regulatory considerations

Live coaching systems occupy a sensitive space between consumer guidance and professional health advice. Responsible design should include:

• Transparent data practices and consent flows for biometric data.
• Conservative boundaries around clinical claims.
• Accessible means to contact human experts when the system identifies red flags.
• Clear disclaimers about model limitations and the scope of advice.

Regulation varies across jurisdictions, especially where devices or software claim to diagnose or treat. Developers should consult legal counsel when scaling features that approximate medical guidance.

Try RepWise: demo and code

A live demo and the code repository are publicly available for testing and study:

• Live demo: repwise-384586125133.us-central1.run.app
• Code: github.com/sujnesh/repwise

The repository contains the frontend, backend, prompt templates, and deployment scripts. Reviewers can explore the semantic event schema, fallback rules, and test suite.

FAQ

Q: What devices and browsers does RepWise support? A: The app runs in modern browsers that expose camera APIs and can run MediaPipe efficiently—recent versions of Chrome, Edge, and Safari on smartphones and laptops. Performance varies by device; older phones may run pose detection at lower frame rates, which can affect rep counting.

Q: How accurate is the pose detection and rep counting? A: Accuracy depends on camera angle, lighting, and occlusions. Under typical gym conditions with a clear view, MediaPipe at 10 FPS provides reliable rep counting and detects common form deviations. Edge cases (occlusion, crowded racks, heavy clothing) reduce accuracy. The app includes prompts to adjust camera position when detection degrades.

Q: How does RepWise protect my privacy? A: Pose detection runs locally in the browser, and only structured semantic events (rep counts, form flags) are sent to the backend. Session data is anonymous by default. If you opt to upload video for coaching review, the app requests explicit consent and treats uploads under stricter data handling policies.

Q: Is RepWise a replacement for a human coach or medical professional? A: No. RepWise provides coaching cues and conservative exercise substitutions but does not diagnose injuries or replace professional medical advice. It flags issues and recommends consulting a professional when appropriate.

Q: What happens if the Gemini API is unavailable? A: Deterministic fallbacks provide safe, conservative behavior for workout generation, live coaching responses, rescue adaptations, and debriefs. The app continues to function using local heuristics and cached templates.

Q: How does Workout Rescue preserve training goals? A: When you trigger Rescue, the system evaluates remaining workload and selects alternatives that maintain the same movement patterns, intensity targets, or volume when possible. If equivalence is impossible, the model explains the compromises and sets expectations for future sessions.

Q: Can I share my session data with a coach? A: The architecture supports sharing session summaries or exporting rep histories, but sharing is opt-in. Any data sharing should comply with user consent and privacy policies.

Q: Is the code open-source and reusable? A: Yes. The RepWise repository provides the frontend, backend, prompts, and deployment scripts as a reference implementation. Developers can adapt the code but should follow best practices for privacy and safety when deploying to production.

Q: How does the app handle pain during exercise? A: When a user reports discomfort, the system asks clarifying questions and recommends conservative alternatives, reduced range of motion, or stoppage. If pain is sharp, sudden, or persistent, the app recommends seeking professional evaluation.

Q: What are the next planned features? A: Potential enhancements include wearable sensor integration, on-device reasoning components for offline rescue, and more personalized long-term progression based on session history. Any extension will prioritize privacy and conservative safety defaults.

RepWise demonstrates a practical path for AI that does more than record movement. By blending robust in-browser sensing, semantic event protocols, and purpose-built model prompts, it keeps workouts alive when the gym refuses to cooperate. The system’s emphasis on deterministic fallbacks, user-level branching, and safety-oriented rescue logic offers a blueprint for builders who want AI that acts like a coach—decisive, contextual, and accountable.