Gym‑Jams: Turning Albums into Music‑Driven Workouts with Simple Audio Analysis and LLM Planning

Table of Contents

1. Key Highlights
2. Introduction
3. The Design Rule: The Album Is the Timer
4. How the System “Reads” an Album: Deterministic Music Information Retrieval
5. From JSON to Workout: Language Models as Structured-Data Reasoners
6. The Time‑Signature Problem: Why One BPM Can Be a Lie
7. Implementation Details: Backend, Prompting, and the Command Contract
8. When the Model Says No: Trustworthy Declination and Trade-Offs
9. Song‑Structure Segmentation and Cue Accuracy
10. Adaptive Track Selection vs Strict Album Order
11. Live Coach Mode: From Cards to Calls
12. Practical Use Cases and Real‑World Examples
13. Edge Cases and Limitations
14. Measuring Success: Metrics and User Feedback
15. Open Source, Community, and Extensibility
16. Safety, Ethics, and Best Practices
17. Roadmap and Research Opportunities
18. FAQ

Key Highlights

• Use the album itself as the workout timer: deterministic audio features (duration, tempo over time, energy arc) mapped to exercise blocks, so the music runs the session with no mid-workout phone fiddling.
• Combine classical music information retrieval (librosa-derived features) with a language model that reasons over structured JSON plus an intent prompt to generate precise workout cards.
• Robust handling of tempo shifts and unstable pulse ensures accurate cadence cues and prevents misleading guidance; the model can decline impossible requests and offer explicit trade-offs.

Introduction

A typical workout app interrupts the flow: tap to start a timer, switch tracks, adjust intervals. Gym‑Jams flips that workflow. Instead of forcing music to conform to a predesigned timer, it treats an album as the timer—play from the first note, finish when the record ends. That simple rule eliminates mid-session decisions and leverages music’s innate tempo and energy structure to guide effort, cadence, and rest.

This approach depends on two clearly separable problems. First, extract measurable, deterministic features from the audio: how long is the album, how loud is each section, where does the tempo shift. Second, interpret those features against an exercise intent—“full body, kettlebells and a pullup bar, go hard”—and produce a workout plan that maps effort windows, movement types, and cadence cues to the record’s timeline. Gym‑Jams uses well-established music information retrieval (MIR) tools for the first half and a language model for the second. The result is a minimal, reliable pipeline that reads a record and writes a session with human-level reasoning about structure and constraints.

The project demonstrates an important point: you don’t need a neural network to “listen” like a human to glean the information that matters for most exercise planning. A handful of precise features, coupled with text-based reasoning over structured data, creates a surprisingly capable system. This article unpacks the design, technical choices, edge cases, and roadmap for turning albums into workouts that start with track 1 and end when the record stops.

The Design Rule: The Album Is the Timer

Gym‑Jams rests on a single design rule: the album is the timer. That rule shapes the user experience. Start the music, follow the card, stop when the music does. No timers, no pattern-orbiting between tracks, no mid-session phone adjustments.

Why this works:

• Many albums are deliberately sculpted with an energy arc—introductory calm, build, peak, resolution—and that arc maps cleanly to warm-up, effort blocks, and cool-down.
• Track order imposes a fixed progression; designers of albums often intend a narrative flow that complements a workout’s phases.
• Using the music as a running clock engages the athlete: the song transitions cue cognitive shifts that can anchor intensity changes better than arbitrary countdowns.

The gym-jams command-line experience demonstrates the simplicity: initialize, generate a plan, take the card to the gym. The tool treats the audio as authoritative and designs the workout to finish when the album finishes, aligning music cues directly with exercise cues.

How the System “Reads” an Album: Deterministic Music Information Retrieval

Gym‑Jams does not rely on a neural network to interpret music. The planner needs only a small, deterministic set of features extracted from the audio. Those features are reliable, explainable, and cheap to compute locally. Typical output for a single track looks like this JSON snippet:

{ "title": "02 Tohogd", "duration_sec": 245.0, "bpm": 172.3, "tempo_arc": [172.3, 172.3, 172.3, 117.5], "pulse_stability_cv": 0.032, "energy_arc": [0.47, 0.92, 0.97, 0.9], "onset_density": 3.04 }

Each value is meaningful for workout design:

• duration_sec: How long the track will occupy the session clock.
• bpm: A global estimate of tempo, useful for rough cadence mapping.
• tempo_arc: Median local tempo measured per quarter of the track; reveals tempo changes across sections.
• pulse_stability_cv: Coefficient of variation of inter-beat intervals; low values indicate a steady pulse that can support cadence-based instructions.
• energy_arc: Loudness or perceived energy per quarter, normalized so the loudest quarter across the album maps to 1.0. Peaks suggest natural moments for max-effort blocks.
• onset_density: How rhythmically busy a track is (onsets per second), which can influence movement selection or intensity.

librosa, a widely used Python audio analysis library, computes these features quickly—on the order of 20 seconds per album on modest hardware. The results are cached for repeat use.

Why deterministic MIR is the right tool here:

• Exercise planning needs measurable facts: how long is a section, does tempo change, is energy rising. These are simpler to extract than tasks requiring human-level semantic interpretation (genre, mood nuance).
• Interpretable outputs make it easier to write rules and prompts for downstream planning.
• Local computation avoids privacy or streaming integration complexities and works offline with a user’s own music library.

From JSON to Workout: Language Models as Structured-Data Reasoners

Once the audio features are available, the problem becomes textual reasoning over structured data: map the JSON describing an album plus a one-line intent into a sequence of workout blocks. That’s where a language model excels—not by “listening” to audio but by synthesizing planning instructions from declarative information.

A typical prompt gives the model:

• The JSON for the album (tracks with durations, tempo and energy arcs, stability scores).
• A short user instruction: e.g., "full body, kettlebells and a pullup bar, go hard".
• Constraints and rules: preserve album order unless using adaptive mode, end when the album ends, shift prescription at detected tempo changes, avoid cadence guidance if pulse stability is low.

The model returns a “card”: a human-readable plan that assigns blocks to track sections. Example output:

Tamebsz — 7:57 @ 123 — Peak effort block. Q4 hits 1.0, the album's loudest point. Heavy KB complex: clean → press → front squat, max effort in the final quarter as energy climbs 0.71→1.0.

Notice two things:

• The model interprets the numeric features into actionable guidance tied to time windows (e.g., Q4, percent segments).
• The output includes movement choices consistent with the user intent.

This division of labor is efficient. Deterministic MIR supplies precise, time-aligned facts. The language model applies domain knowledge about training structures and translates those facts into an exercise prescription.

Prompt engineering is critical. The prompt embeds explicit heuristics for mapping audio features to workout demands. Example heuristics:

• If a quarter's energy >= 0.9, designate a peak effort block.
• If tempo_arc shifts by more than X BPM between adjacent quarters, place a cadence transition at that boundary.
• If pulse_stability_cv > 0.1, avoid cadence-specific instructions.

These heuristics anchor the model’s reasoning and reduce hallucination. The model's role is to reconcile the user’s intent with the album’s structure and to explain trade-offs when a perfect match is impossible.

The Time‑Signature Problem: Why One BPM Can Be a Lie

A single BPM per track is often inadequate. Many bands vary feel, tempo, and meter within a song; reporting one global BPM treats the track as homogeneous and can produce misleading guidance.

The project’s test case illustrates this: an Angine de Poitrine track (Tohogd) reported a global BPM of 172.3. Listening shows the final quarter shifts into half-time. If a plan averaged tempo over the whole track, it would instruct athletes to “hold effort through Q4” while the music has actually eased into a different feel. Another track reported "143.6 BPM" that existed only in one quarter.

Two observations follow:

• Detecting meter labels like 7/8 is research-level work and not required for workout design.
• What matters is tempo over time—the local tempo and pulse stability—and aligning prescriptions to those changes.

Fix applied:

• Replace global BPM with a tempo_arc: local median tempo computed per-quarter (or per configurable segment) of a track.
• Compute a pulse_stability score (coefficient of variation of beat intervals) to detect how reliable cadence-based guidance will be.
• Add prompt rules: when tempo_arc shifts, change the prescription at the shift; when pulse stability is low, avoid cadence cues.

The regenerated card reflected this approach. For Tohogd, the card correctly noted that the final quarter drops to half-time and recommended easing to a recovery jog at that transition instead of averaging through the change.

This solution avoids the complexity of formal meter detection while preserving the essential information for training: does the music change its feel in ways that affect cadence and intensity?

Implementation Details: Backend, Prompting, and the Command Contract

Gym‑Jams organizes the model backend as a single configurable slot that follows a stdin/stdout contract. This keeps integration straightforward and flexible. Example configuration:

command = "claude -p --model {model}" model = "sonnet" music_dir = "~/Music"

One-line prompts piped into the model yield cards on stdout. The default example uses an authenticated local client (claude -p), so no API keys are embedded in the codebase. Anything that reads stdin and writes stdout can be substituted, and there is an OpenAI-compatible HTTP backend for deployments that prefer remote endpoints.

Album resolution is simple: with music_dir set, users name an album fragment and the tool resolves candidates. Rather than guessing, gym‑jams lists ambiguous matches so the user explicitly selects the intended record. That small UX decision avoided immediate confusion when “vol.1” matched two different albums.

Prompt structure matters. A robust prompt includes:

• A brief style guide for card formatting (time stamps, energy cues, cadence notes).
• The JSON with per-track feature arrays.
• A list of movement libraries and constraints (e.g., if the user has a kettlebell and pullup bar, prefer compound movements that use those tools).
• Failure modes and explicit preference for truthful declinations rather than padding.

This setup keeps the code lean. The audio feature extractor produces a predictable JSON shape, and the model maps that into a concise, readable plan.

When the Model Says No: Trustworthy Declination and Trade-Offs

A surprising and valuable behavior emerged: the model refused to fake a workout. Asked for "20 minute easy recovery" against an album that only contained about 14.5 minutes of calm per energy arcs, the model selected appropriate tracks totaling 14:27, explained why other tracks were unsuitable, and offered two explicit trade-offs instead of padding to 20 minutes.

That behavior is important for user trust. Two failure modes are worse than a simple decline:

• The system fabricates details (e.g., claims 20 minutes of calm when none exists).
• The system produces a plan that violates safety or training principles (e.g., prescribing max efforts on a sustained quiet segment that doesn't support it).

The model’s transparent trade-offs—pick louder tracks and accept higher intensity, or split the session into two albums—give users clear options rather than silent failure or misleading output. Design choices that encourage this honesty include:

• Penalizing hallucination in the prompt: request explicit evidence from the JSON when the model asserts total durations or cadence.
• Requiring the model to cite the sections or tracks that justify a claim.
• Giving the model the authority to refuse impossible tasks and to propose alternatives.

Real users value these honest constraints. They calibrate expectations and reduce the likelihood of injury or dissatisfaction.

Song‑Structure Segmentation and Cue Accuracy

Quarter-based segmentation works but is blunt. The next step is song-structure segmentation: detecting actual section boundaries—intro, verse, chorus, bridge, drop—so exercise cues land on musical moments rather than arbitrary quarter markers.

Why segmentation matters:

• Peaks and drops frequently occur within quarters. Aligning a max-effort block with a genuine drop provides a stronger cue and better psychological anchoring for athletes.
• Some tracks have long transitions where a quarter boundary might slice through a buildup, producing confusing cues.

Approaches to segmentation:

• Use energy and onset density to detect likely section boundaries; abrupt changes in energy or sudden increases in onset density often mark transitions.
• Implement change-point detection on spectral features or novelty functions, which identifies times where the audio signature shifts significantly.
• Optionally, allow human confirmation: present suggested cue points and let the user accept or nudge them before the session.

Segmentation introduces complexity but improves the quality of cues. For now, quarter-based tempo and energy arcs provide a good balance of simplicity and fidelity, and they resolve the major pitfalls like tempo shifts. Segmentation will tighten alignment and improve the coach-mode’s timing for live prompts.

Adaptive Track Selection vs Strict Album Order

Gym‑Jams currently supports both philosophies:

• Strict-album-order mode: preserve the artist’s sequence and design the workout to match it exactly. Useful for listeners committed to the album as narrative or for curated sets where order matters.
• Adaptive track selection: choose a subset of tracks that best match the requested intent (e.g., recovery ride) irrespective of album order. This was the mode that produced the honest 14:27 recovery selection.

Trade-offs:

• Adaptive selection leads to better matches to intent but breaks the “album as timer” rule unless the selection forms a continuous playback. You can still treat the selected playlist as the timer.
• Strict order preserves artistic intent but can force compromises between the user’s training goals and the album’s structure.

User controls can make this explicit: let users choose a mode, or present the model’s best-fit plan and let the user toggle between strict and adaptive outputs. For sessions designed around an album’s narrative or for live DJ sets, strict mode is essential. For context-specific goals (easy recovery, high-intensity interval training), adaptive selection generally yields better outcomes.

Live Coach Mode: From Cards to Calls

A planned feature is a live coach mode that plays the record and calls the blocks in real time. It will need:

• Precise cue timing: mapping plan blocks to sample-accurate timestamps.
• Audio playback control: detect track changes and start prompts relative to first-beat alignments.
• A voice component capable of concise, context-aware prompts: “Start KB clean sequence in 3 — 2 — 1,” or “Ease to recovery jog now — tempo drops.”

Practical issues:

• Beat alignment is critical. If beat trackers are off by fractions of a second, a coach call can feel out of sync. Aligning to onsets or detected downbeats reduces perceptual jitter.
• Latency between detection and speech needs buffering. Precompute exact timestamps rather than relying on real-time detection.
• Allow user overrides: a physical button or wearable gesture can silence or delay a cue while preserving session integrity.

A live coach that respects musical moments and avoids distracting the athlete will transform gym‑jams from a planning tool into a guided experience that removes the phone from sight and hands.

Practical Use Cases and Real‑World Examples

Multiple training scenarios benefit from this approach. Below are illustrative examples that show how the album-as-timer design applies across modalities.

Running: Cadence-matched tempo mapping

• Use a steady electronic album with a low pulse_stability_cv (< 0.05) and a tempo_arc around 160–170 BPM to guide fast turnover runs. A 170 BPM record can support 170 steps per minute (or 85 strides if using two-beat cycles). If the tempo arc drops mid-track, the model inserts a cadence shift and recommends easing pace.

Kettlebell and barbell complexes: Energy-aligned peak blocks

• An album with a clear energy peak (energy_arc with a quarter at 1.0) is ideal for a max-effort complex. Gym‑jams assigns heavy kettlebell complexes to that window. The system might map the buildup quarters to warm-up and technique-focused sets, saving the loudest section for maximal lifts.

HIIT with odd meters or changing feel

• Bands that change feel mid-song can still be used: the tempo_arc detects shifts, and the model assigns intervals that align to sections with consistent beat intervals. If pulse_stability is low, the system avoids cadence-dependent instructions and instead prescribes time-based blocks (e.g., reps or AMRAPs) tied to loudness spikes.

Recovery sessions from mixed albums

• For a recovery request, the model may select three tracks that total the nearest calm duration and explain why louder tracks were excluded. If the requested duration is longer than calm sections available, the model presents alternatives: accept higher energy, add quiet tracks from a different album, or split the session.

Group classes and circuit sessions

• Use a curated album with clear transitions for instructor-led sessions. The band’s intentional arrangement creates predictable build and release patterns, making it easier for instructors to cue groups without constant timekeeping.

These examples show the system’s versatility. The common theme is alignment: match training needs to what the music actually provides, not to what a naive global estimate would claim.

Edge Cases and Limitations

Gym‑jams handles many cases well, but there are inherent limitations and failure modes to address.

Live albums and crowd noise:

• Loud audience sections inflate energy measurements without indicating a musical peak conducive to exercise. The model should detect sudden increases in non-musical energy (spectral characteristics, broadband noise) and discount them when assigning effort.

Tracks with silence or hidden tracks:

• Long silences can mislead duration-based plans. Silence detection and trimming are necessary before feature extraction.

Mixed-genre albums:

• Albums that jump between genres present both tempo and energy discontinuities. Adaptive mode often works better, or strict mode with explicit user awareness that the album will force varied training phases.

Unreliable beat tracking:

• Very sparse or highly syncopated music can break beat trackers. high pulse_stability_cv signals this and prompts the system to avoid cadence guidance.

Safety and movement appropriateness:

• The model can recommend high-intensity moves during loud sections, but it should respect user constraints (e.g., injuries, equipment limits). Explicit user profiles and movement blacklists are essential.

Streaming and licensing:

• The current project works with local music. Integrating with streaming services adds authentication and potential licensing constraints for automated playback and cueing.

These limitations suggest practical mitigations: robustness checks, user-supplied constraints, and conservative defaults. The system should default to safe, explainable recommendations.

Measuring Success: Metrics and User Feedback

Effectiveness can be measured several ways:

• User engagement: Do users complete sessions without mid-session phone interaction? The album-as-timer design should increase completion rates and reduce fiddling.
• Perceived sync: Users rate whether cues felt timely and motivating.
• Adherence to intensity targets: Heart-rate data or power metrics (for cycling) can indicate whether energy-aligned blocks produce the intended physiological response.
• Safety incidents: Track reports of discomfort or injury and see if certain recommendations correlate with issues.

A/B tests can compare gym‑jams sessions against conventional interval timers:

• One cohort uses music-driven sessions; another uses a standard timer with the same target durations.
• Compare motivation, perceived enjoyment, and objective performance (e.g., reps completed or time-in-zone).

Qualitative feedback remains vital. If a model’s movement selection or cue wording confuses users, adjust the movement library and prompt style. The language model can be tuned for concision, clarity, or motivation depending on the audience (e.g., athletes vs. general exercisers).

Open Source, Community, and Extensibility

Gym‑jams is available as an open-source project. That decision encourages community contributions across several axes:

• Movement libraries for different training modalities and equipment pools.
• Localization and language variants for coach prompts.
• Improved audio segmentation models contributed by researchers.
• Integrations with wearables and music players to support live coach mode.

Open-source contributions can also attack thornier MIR problems: better beat trackers, robust silence and crowd-noise detectors, and music-structure segmentation. Community-sourced playlists curated for training intents could provide ready-made albums for specific workouts.

Extensibility should follow the project’s minimal-contract philosophy: keep the audio feature shape stable and let different model backends (local or remote) plug in via stdin/stdout. That allows experimentation with different LLMs, including smaller offline models for edge deployments.

Safety, Ethics, and Best Practices

Design choices must prioritize participant safety and transparency.

User profiles and constraints:

• Collect equipment availability, movement restrictions, and fitness level. Constrain movement selection based on those parameters.

Explainability:

• Always present the reasoning: list the tracks and specific data (e.g., “Q4 energy 0.97 — recommended max-effort window”). Users should understand why a recommendation exists.

Avoid overclaiming:

• The system should not provide medical or injury advice. It should encourage users to consult professionals when necessary.

Privacy and local music:

• Favor local computation of audio features to respect user privacy. For cloud services, be explicit about what data leaves the device.

Accessibility:

• Provide text and auditory cueing options for different needs. Visual cards alone are insufficient for some users; conversely, voice-only prompts can be disruptive in shared spaces.

These practices build trust and lower the risk of harm.

Roadmap and Research Opportunities

Several practical and research directions will improve fidelity and usability.

Short-term product improvements:

• Song-structure segmentation for better cue alignment.
• Precomputed cue timestamps for live coach mode.
• A marketplace of curated album packs optimized for training intents.

Research topics:

• Robust beat and downbeat detection under variable meters and live recordings.
• Novelty detection tuned to musical events that are meaningful for human perception (drops, choruses).
• Lightweight on-device models for segmentation that balance accuracy and compute.

User-experience research:

• Investigate whether musical cues aligned to drops improve adherence and perceived exertion compared with arbitrary timers.
• Study whether artist-intended album arcs lead to better subjective enjoyment and long-term adherence.

The project sits at the intersection of MIR, LLM-driven planning, and exercise science, so collaboration across disciplines will yield the best outcomes.

FAQ

Q: Does gym‑jams “listen” like a human? A: No. It extracts deterministic, measurable features—tempo over time, energy per segment, pulse stability—using established MIR tools. A language model then reasons over that structured data to design workouts. The system does not attempt genre or emotional interpretation in the human sense.

Q: Can it handle songs with odd meters like 7/8? A: Yes. Odd meters are not inherently problematic if the beat is steady. The system focuses on pulse stability and tempo over time. If the beat intervals are steady, cadence guidance is viable; if the music changes feel mid-song, gym‑jams detects tempo shifts and adjusts prescriptions accordingly.

Q: What happens if the album doesn’t match my requested workout length or intensity? A: The model will explain the mismatch and propose explicit trade-offs rather than fabricating a plan. Options might include selecting alternative tracks, combining albums, accepting a different intensity, or splitting the session.

Q: Is gym‑jams safe for beginners or people with injuries? A: The tool makes recommendations based on audio structure and a movement library. Users should provide movement constraints and fitness levels. The system is not a medical tool and should not replace professional guidance. It is designed to be conservative and transparent about risk.

Q: Can I use streaming services like Spotify or Apple Music? A: The prototype operates on local music due to privacy and playback control reasons. Streaming integration is possible but adds authentication and licensing considerations that require separate implementation.

Q: How accurate is beat detection and tempo measurement? A: For most modern recordings, beat tracking and local tempo estimates are reliable. Challenges arise with very sparse, highly syncopated, or noisy recordings (e.g., crowd noise). The system computes a pulse_stability metric to detect unreliable beat information and adjusts guidance accordingly.

Q: Can I choose strict album order or adaptive selection? A: Yes. Strict-album-order mode preserves the artist’s sequence. Adaptive mode selects tracks that best match the requested intent. Each mode has trade-offs: strict mode keeps the narrative flow, while adaptive mode better matches training goals.

Q: Will there be a live coach that talks over the music? A: A live coach mode is planned. It will use precomputed cue timestamps aligned to musical events and deliver concise voice prompts timed to those moments. Buffering and beat alignment are critical to timely cues.

Q: Is gym‑jams open source? A: Yes. The project repository is public and welcomes contributions for movement libraries, MIR improvements, model backends, and curated training albums.

Q: How do I get started? A: Install the tool, point it at your local music directory, run a plan command with a short intent, and review the generated card. Start the album and follow the plan; stop when the music stops.

Gym‑jams demonstrates that combining straightforward audio feature extraction with focused language-model reasoning produces practical, transparent, and compelling music-driven workouts. The album-as-timer rule keeps the phone out of hands and lets the music guide the session. With careful prompt engineering, tempo-over-time analysis, and a clear contract between MIR and planning, albums become ready-made training sessions: play from track 1, follow the card, and finish when the record does.