The beat grid: replacing detected beats with a formula

For three weeks, VidViz had beats the obvious way: the audio analysis returned an array of timestamps, and everything downstream — the ruler, the beat-reactive nodes, section snapping — indexed into that array. This week the array lost its job. Beats in VidViz are now a formula, and the week of migration convinced me this is the only sane architecture for musical time in a visualiser.

What the analysis actually does

The Python backend chews a track with librosa: per-frame RMS, spectral centroid, mel bands, chroma, frequency bands, optionally LUFS loudness and key estimation — everything the visual nodes might want to feed on, padded to a fixed frame count and cached against the file's path, mtime, and analysis config, so re-opening a track is instant.

Tempo comes from Essentia's multifeature rhythm extractor (run in a worker with a 30-second timeout — Essentia takes its time and its time is not the UI's), with librosa's beat tracker as fallback. And here's the turn: the raw beat timestamps it produces are not what ships. Detected beats jitter. Track them through a synth pad and they wander tens of milliseconds; two consumers reading neighbouring beats can disagree about where "now" is. Every consumer of the array grew its own off-by-a-little.

BPM + phase: two numbers instead of four hundred

The pipeline now reduces detection to exactly two numbers. The BPM is normalised into the 70–160 range (halving or doubling — is it 75 or 150? for visuals, pick the felt pulse) and rounded. Then the phase: with BPM fixed, slide a candidate grid across one beat interval in a hundred steps and keep the offset whose grid points land on the highest average RMS — the alignment where the energy actually is. Detection's whole output is bpm and offset. Every beat thereafter is:

beatTime(i) = i * (60 / bpm) + offset

That formula lives in one shared utility — beatGrid.ts — with the derived family around it: snapToBeatGrid, beatIndexAt, beatPhaseAt, barPhaseAt. The playhead, the beat ruler, the quantise logic, section snapping, and the BeatDetect node all import the same file. The old BeatSync module, which had been quietly accumulating its own slightly different arithmetic, is deleted. One notion of musical time, one implementation, zero negotiations.

The BeatDetect node is the purest expression of the shift: it receives the per-frame data (which still carries an isBeat flag from the raw analysis) and ignores it, recomputing from BPM + offset + current time, firing its trigger port whenever the computed beat index increments. The raw flag survives only as a fallback for tracks where tempo detection failed outright. Detection proposes; the grid disposes.

A ±500ms latency nudge rides on top, because the perceived beat is the sum of detection phase, audio output latency, and render latency — no algorithm sees your speakers. One slider moves the entire world, which is only possible because the world is derived from one formula.

Sections stand on the grid

Section detection (the S1/S2 song-structure markers) runs downstream of the same idea: beat-synchronised MFCC and chroma features, a cosine self-similarity matrix, a checkerboard kernel convolved down the diagonal to get a novelty curve, peaks picked — and every boundary snapped to the bar grid, because song sections change on bars, not on wherever a novelty peak wobbled.

Two refinements this week made it actually pleasant. First, adaptive sensitivity: the backend used to apply a fixed threshold and return its verdict; now it returns all candidate boundaries with strengths, and the frontend filters through a sensitivity slider. Detection tuning became a live UI gesture instead of a re-analysis — the general trick of "return scored candidates, threshold client-side" applies to basically every detector, and I keep forgetting it.

Second, a proper race condition died. Analysis and section detection ran in parallel, so sections often snapped to a grid that hadn't been computed yet — quantising against BPM zero, garbage in a trench coat. The fix is embarrassingly sequential: wait for the grid, then detect sections. Parallelism between a producer and its consumer isn't parallelism, it's a coin flip.

Skips bend time; the grid doesn't break

The sequencer can skip sections during playback, which poses a neat question: after the playhead jumps over sixteen bars, is the next beat still a downbeat? With a beat array this is a nightmare of index bookkeeping. With a grid it's arithmetic: count the beats inside skipped regions before the current time, subtract, and the adjusted index keeps bar phase honest — visuals that pulse on the downbeat stay on the downbeat across any pattern of skips. The skip regions travel to nodes in the same per-frame data, so even node processors can be skip-aware.

Everything the user tunes — BPM, offset, beats-per-bar, markers, section candidates, sensitivity — persists per track in a local cache keyed by file path, so the grid you corrected once follows the song across every graph and project that uses it. Corrections belong to the music, not to the document you happened to make them in.

The moral

The instinct when a detector is imperfect is to improve the detector. Sometimes the better move is to shrink its authority: let it estimate two parameters of a model, and derive everything else from the model. The grid is wrong in a way the array never was — a real drummer drifts, and a fixed-BPM grid won't follow — but it's wrong consistently, every consumer agrees, and one slider fixes perception globally. For electronic music, which is what people visualise, a consistently-wrong grid beats out an inconsistent one that's occasionally closer, every time. Models over measurements, wherever you can afford them.