At WWDC26, Apple introduced Music Understanding, a framework that analyzes music directly on Apple platforms and returns structured information about rhythm, key, loudness, pace, structure, and instrument activity. For rhythm specifically, it can return timestamps for detected beats and bars, along with a global BPM value. Apple’s framing is explicit: the framework handles signal processing and model inference, so developers don’t need signal-processing or machine-learning expertise just to use the API.

That sounds like a major boundary shift for audio development. Until now, even a relatively straightforward feature such as BPM detection could pull an iOS or macOS developer into AVFoundation, raw PCM buffers, digital signal processing, and concurrency.

I have worked through that stack while building a multiplatform BPM analyzer for Mac, iPhone and iPad, with a watchOS companion. The pipeline involves AVFoundation and AudioToolbox, real-time buffer manipulation, DSP, beat-detection algorithms, raw-audio parsing, frequency analysis, and moving CPU-heavy work away from the main thread.

So if Apple can now return BPM in a few API calls, does an iOS developer still need to understand any of that? Yes. But the reasons have changed.

Before Music Understanding, BPM detection meant owning the pipeline

A BPM analyzer is a good example because the final output looks deceptively simple: one number. The work required to get there is not.

A typical custom pipeline looks roughly like this:

audio input → PCM buffers → signal processing → beat detection → BPM result

AVFoundation can get audio into a form the application can process, but at that point the developer still has to make sense of the samples. That can mean examining energy over time, converting data into the frequency domain, finding transient events, looking for periodic patterns, and deciding which candidate tempo is most plausible.

In my BPM analyzer, decoding compressed audio into samples accounted for roughly three quarters of the CPU work. Most of the remainder went into onset detection, including FFTs over short windows. One of the biggest speedups came not from changing the BPM algorithm but from replacing a decoding API that had been serializing the work. On a 100-track batch, that cut processing time from 52 seconds to 12. Another gain came from removing an analysis stage whose output was not used.

Choosing the algorithm is only part of the engineering work. Audio processing can also become expensive very quickly. A few minutes of stereo audio contain millions of samples. If you transform, filter, compare, or correlate each window, the computation grows fast.

At that point, application architecture matters as much as the algorithm itself. I moved orchestration to Swift Concurrency, using async/await and a task group with a bounded number of files in flight. The blocking work, however, stayed on a private GCD queue.

That distinction mattered because audio decoding can block a thread inside the system framework rather than suspend cooperatively. With enough files in flight, Swift Concurrency’s worker pool could end up with every thread parked: zero percent CPU, no completed work, and a UI that looked hung even though the main thread was not doing the heavy processing.

The practical lesson was simple: blocking audio work needed its own queue, and that lesson outlived the custom pipeline. Music Understanding parks on the same cooperative pool, and because the blocking happens inside the framework’s own tasks, moving your code to a private queue doesn’t help; cancellation and timeouts don’t free it either. The fix is to cap how many analyses run at once.

What Music Understanding takes off the developer’s plate

Music Understanding moves much of that subsystem behind a system framework.

A MusicUnderstandingSession can work with a complete AVAsset or with a streaming source of AVReadOnlyAudioPCMBuffer values. The developer selects analyses, calls analyze, and gets structured results back.

Apple recommends avoiding work the app does not need, but my on-device timing adds an important nuance: across 300 tracks, requesting all six analyses in one call cost 1.3× what requesting rhythm alone cost, while running the same six analyses as separate sessions cost 4 times as much.

Much of the cost appears to be shared front-end work that every call repeats. So if an app needs two or more dimensions, one combined analyze() call and a cache can be cheaper than a sequence of narrowly scoped calls.

For rhythm, the public result is already close to what many applications need:

public struct RhythmResult: Codable, Sendable {
public let beats: [CMTime]
public let bars: [CMTime]
public let beatsPerMinute: Float?
}

Apple’s documentation says beats contains the timestamp of every detected beat, bars contains bar start times, and beatsPerMinute provides the global tempo when enough audio has been processed. The setup for a file-based source is equally high-level:

let asset = AVURLAsset(url: url)
let session = try await MusicUnderstandingSession(asset: asset)
let result = try await session.analyze(for: [.rhythm])
let bpm = result.rhythm?.beatsPerMinute

The API is shipping in the iOS/macOS 27 SDK, so the shape is settled, and the architectural shift is already clear. In a 100-track corpus whose tempos came from the files’ own ID3 tags — imperfect ground truth, partly written by the analyzer under test — Music Understanding’s beat array, folded into the same tempo range, tied the custom pipeline exactly on both the strict count and the one that forgives half- and double-time. It fixed four tracks the pipeline missed and broke four it got right.

The gap appears when you take the framework’s own beatsPerMinute property instead of deriving tempo from its beat timestamps. Unfolded, it scored 63 against the custom analyzer’s 78, and it still lost when half- and double-time were allowed. The main difference was interpretation rather than raw beat detection: a 174 BPM drum & bass track, for example, can plausibly come back as 87 BPM. When I folded Music Understanding’s beat-interval estimate into the same 88–175 BPM search range used by my own analyzer, the two matched exactly on both metrics.

That gap is corpus-dependent, and the direction matters: on a separate 200-track set of steadier material, the same folded estimate beat the custom analyzer, 96.5% exact against 94.5%. A tempo range is worth more the more octave-ambiguous the music is, and close to nothing when it isn’t.

The public API has no BPM range parameter. A custom analyzer can constrain the search space to resolve half-time and double-time ambiguity; Music Understanding has to choose a perceptual interpretation on its own.

Before, the application might own:

PCM → preprocessing → feature extraction → beat detection → tempo estimation

Now, for a standard rhythm-analysis use case, the application can own:

audio source → Music Understanding → RhythmResult

For a standard rhythm-analysis feature, most of the custom DSP layer disappears.

This does not make PCM buffers disappear

Music Understanding still leaves some low-level audio work to the application. Apple also supports a custom audio provider that supplies an AsyncSequence of AVReadOnlyAudioPCMBuffer. In other words, developers can still feed streamed PCM data into the framework when the application architecture requires it.

The framework can own the musical interpretation while the app may still own capture, decoding, routing, or buffering.

The API supports that split: an application can load audio with AVAudioFile, read it into a read-only PCM buffer, and hand it to Music Understanding for rhythm analysis. Developers can still touch audio buffers while leaving signal analysis to the framework. For many apps, that is exactly the abstraction they want.

Where the system API stops being enough

If a product needs BPM, key, structure, loudness, beat positions, or pace, there is now a strong reason to try the framework before building a custom algorithm. But the six dimensions are not equally mature in practice.

In my 300-track test — 18.5 hours of audio, all six analyses — loudness stood out: its integrated values matched ffmpeg to within half a unit on every file, and it was the only result that arrived while audio was still streaming in. Vocal detection worked well as a gate: across a 100-track control of vocal pop, not one was wrongly reported as instrumental.

Beat timestamps remained useful even when the global BPM landed on the wrong half-time or double-time interpretation. Key detection was usable with caveats, including enharmonic spellings such as F♯ and G♭ and, on every file, a single key range.

The type is an array, so the framework can express modulation; it simply never did here. The major/minor call is the caveat that matters, though: 236 of the 300 tracks came back minor, on a corpus with a hundred pop tracks in it. BPM itself was the weakest fit for a drop-in replacement because metrical interpretation matters to products such as DJ tools.

A custom DSP pipeline still gives the developer more control over how tempo is interpreted. My analyzer relies on inputs and intermediate data that the public API does not expose: a tempo search range, alternative BPM candidates, confidence, and the onset curve underneath the estimate. Those are not minor diagnostics. The search range can define which metrical interpretation wins in the first place.

The underlying models appear to compute considerably more information than the public API returns. The framework ships Core ML models whose declared outputs include a per-frame distribution over candidate tempos and another over beats per bar alongside beat positions.

The bins themselves don’t say what tempos they cover — that mapping lives in the framework’s postprocessing, not in the model file. The key model similarly produces probabilities across keys, and the structure model produces section labels. In other words, some confidence, alternatives, and structural detail live inside the model boundary but are intentionally abstracted away at the framework boundary.

Those hidden decisions matter when the framework’s interpretation conflicts with the product’s requirements. DSP knowledge remains useful here even if the developer no longer implements the baseline algorithm.

In my own analyzer, Music Understanding is not replacing the tempo engine; it runs alongside it as an optional cross-check that flags tracks the two estimators disagree on.

Across 200 labelled tracks, it caught 6 of the 7 genuine, non-octave errors, but about half the flags were false alarms, so it is a cue to listen again, not a verdict. It is off by default: turning it on takes a batch scan from a fraction of a second per track to about two.

Real-time processing is a separate question

Music Understanding can accept streaming PCM buffers, but that does not automatically make it a replacement for every real-time audio engine.

The framework is designed around musical analysis. An app that performs live effects, audio synthesis, monitoring, instrument processing, or latency-sensitive interaction may need control over buffer scheduling and execution that a semantic analysis framework does not expose. There is also an important difference between “streaming input” and “continuous low-latency output.”

In the shipped SDK, loudnessResults is the session’s streaming AsyncSequence; there is no equivalent rhythmResults stream. Rhythm arrives only when analyze() resolves. Streaming audio into the session therefore does not mean streaming rhythm results back out.

That matters in systems that already perform continuous tempo analysis. My own live mode feeds a rolling window of the most recent audio from the microphone, produces a new estimate about once a second, and stabilizes the displayed number by voting across recent history.

It takes roughly ten seconds to say anything useful and about a minute to become confident. Music Understanding cannot currently drop into that path: its beat model works in fixed 40.96 s blocks, which suits batch analysis but not a rolling, low-latency window.

Developers can feed it a live stream, but it only streams loudness back while the audio is arriving; rhythm is available at the end of the analysis.

For latency-sensitive audio, knowledge of sample rates, frame counts, buffer sizes, scheduling, Audio Units, AVAudioEngine, and thread behavior remains useful regardless of whether a high-level music-analysis API exists.

DSP knowledge now helps you choose the abstraction

After WWDC26, developers have less reason to own the DSP layer simply to obtain a BPM value. The knowledge still matters when deciding whether the system API is sufficient for a particular product.

If all the product wants is “what is the BPM of this song?”, implementing beat detection from scratch is increasingly difficult to justify. Apple has a first-party on-device implementation that works offline and keeps analyzed audio on the device. But audio-domain knowledge still helps developers evaluate the result.

A global BPM does not tell you everything about musical motion. Apple separates rhythm from PaceResult, which represents perceived energy and momentum independently of fixed tempo. Apple documents pace as independent of fixed tempo, so it shouldn’t resolve octave ambiguity.

I measured it anyway — “shouldn’t” and “doesn’t” are different claims: across 300 tracks, pace correlated 0.55 with the framework’s own BPM estimate and essentially zero (-0.06) with the reference tempo.

In practice, it was tracking perceived energy more than metrical level. That makes pace potentially useful for playlist feel, workout pacing, or section intensity, but not as a reliable tiebreaker for BPM octave decisions.

Beat timestamps, bar boundaries, and BPM also serve different purposes. The framework exposes all three because different products need different levels of musical structure. Understanding those distinctions makes it easier to use the high-level API correctly.