sdm

sdm looks after a DJ music collection. It only reads and reports: nothing in the library is ever moved, renamed or deleted.

• Duplicates. It compares how tracks actually sound, so it catches the same song saved twice under different file names or in different formats, which a search by name would miss.
• Audio quality. It measures what each file’s audio really is (its true bitrate, where its frequencies stop, its loudness, how often it clips) and puts that beside what the tags and Rekordbox claim. A “320 kbps” file whose sound stops at 16 kHz was made from a 128, and gets flagged.
• Rekordbox analysis. It reads the beat grid and song structure (intro, verse, chorus, outro) that Rekordbox worked out for each track, straight off a USB stick, without Rekordbox installed.
• Rekordbox library. It exports the library with each track’s tempo, key and length.
• Shopping list. Give it a Spotify playlist and a music folder, and it lists the songs you still need. It knows an extended mix and a radio edit are the same song, but a remix by someone else is not.
• Catalog. It lists everything in a folder, then checks the list against the folder so nothing is quietly left out.

It runs on your own computer, from the command line or as a page in your browser. In the browser you can play duplicate copies side by side and switch between them without losing your place in the song.

Part of it also runs online, inside your browser tab: the catalog, the Rekordbox analysis, and the shopping list from a CSV export of the playlist. You pick a music folder or a Rekordbox USB and the tools read it where it is, so nothing is uploaded. Finding duplicates and checking quality need programs a web page can’t run, and a Spotify link needs a login, so those stay on your computer.

It grew out of spdl, a Spotify downloader.

Under the hood

The rest of this page is the technical version. All numbers come from my own library, a clone of a Pioneer USB with about 3,660 tracks. There is no test suite. Instead, these measured runs are the regression checks: if a run on the same folder reports very different numbers, something changed.

Structure

sdm is one Python CLI with a subcommand per feature. A feature may import from a small shared core (the track model, tag readers, fingerprinting, the analysis reader, report writers) but never from another feature. When two features need the same code, it moves into the core. Heavy dependencies such as pyrekordbox and pandas are optional extras, imported inside the command that needs them, so sdm --help and the lighter commands work on a machine with neither installed.

Finding duplicates by sound

Detection runs in two stages. First, a prefilter buckets tracks of similar length, within 2 seconds by default. Duration is the one field every file has, because it is decoded from the audio rather than read from a tag. BPM and key can be added to the prefilter, but only half my files carry a BPM tag, so requiring it would silently drop half the library. On my library the prefilter turns 6.7 million possible pairs into 64,272 candidates.

Each candidate pair is then compared by its Chromaprint fingerprint. Similarity is one minus the Hamming distance over the 32-bit sub-fingerprints, across their overlapping length. The threshold is 0.85 rather than something near zero, because unrelated audio scores about 0.5: two random 32-bit words agree on half their bits by chance. Confirmed pairs go through union-find, so a track saved three times comes out as one group instead of three pairs. A full run takes about nine minutes, nearly all of it fingerprinting at roughly 0.14 seconds a track. It confirms 178 of the 64,272 pairs, which form 158 groups: 148 pairs and 10 triples.

Getting fingerprints out took some debugging:

• sdm runs fpcalc -raw -json directly instead of using pyacoustid. pyacoustid returns the compressed fingerprint, and decoding that needs the native libchromaprint library, which the standalone fpcalc.exe does not ship. The first version fingerprinted every track and then failed on the first comparison.
• If no fpcalc is installed, sdm downloads a pinned Chromaprint release into a scratch folder for the run and deletes it in a finally block, including on Ctrl-C. On Windows the executable that just ran can stay locked for a moment, so the cleanup retries instead of failing silently.
• A pre-flight fingerprints five files before the real run. Without it, a broken binary would only show up hours later, as a report of zero duplicates.

Reading Rekordbox’s analysis

When Rekordbox exports to a USB, it writes analysis files for each track under PIONEER/USBANLZ/. Each file is a chain of tagged binary sections: the beat grid (PQTZ), the phrase structure (PSSI), several waveforms, and more. sdm takes the binary layouts from pyrekordbox, but walks the chain itself and parses only the sections it needs, one at a time.

That has two benefits:

• Speed. Waveforms are almost all of the bytes, and skipping them takes a full scan from 8½ minutes to 49 seconds.
• Resilience. pyrekordbox parses each file as one structure, so one section it dislikes loses the whole file. It declares a PQTZ header word as the constant 0x00080000 when only the upper half is fixed, and that alone discarded 119 of my 3,665 files. Read per section, all 119 grids pass a sanity check, and beat-grid coverage goes from 3,541 tracks to 3,655. Sections that newer Rekordbox versions write and pyrekordbox has never seen (PVDI, PVB2) now cost only themselves.

Rekordbox analyzes each track in one of three moods, and the mood decides what a phrase is called. Mid and low name phrases by kind alone: Intro, Verse 1–6, Bridge, Chorus, Outro. High uses fewer kinds and keeps the variant in separate flag fields: Intro 1/2, Up 1/2/3, Down, Chorus 1/2, Outro 1/2. That makes 20 labels, and all 76,118 phrases on my USB resolve to one of them. An unknown kind is reported as unknown rather than given a plausible name, so a format change would show up in the output.

Rows are written as tracks are read, so the millions of individual beats are never all in memory. The grid also checks the tags: of the 1,814 tracks with a BPM tag, 17.3% disagree with the grid by more than 0.5 BPM, and 97 are tagged at exactly half the real tempo.

Reading the Rekordbox library

Sources are tried in order: a master.db on the USB or one you point at, then the local Rekordbox install’s master.db, then the raw tags. The local database wins even when a USB is given, because it is the only source with analyzed BPM and key. It is then scoped down to the tracks actually on the USB. The USB’s own exported database (export.pdb, exportLibrary.db) is encrypted DeviceSQL, so it is skipped.

Measuring audio quality

Each file is decoded exactly once. ffprobe reads the stream layout, then a single ffmpeg pass measures loudness and true peak with its ebur128 filter, while the decoded samples stream into numpy for clipping, the spectrum, bit depth and stereo correlation.

• Bitrate is the bytes of audio packets over the decoded length, so tags and cover art don’t inflate it. ffmpeg only prints the packet total at -v verbose, which makes that flag load-bearing. The command this replaced divided decoded PCM bytes by duration, which gave every 44.1 kHz file the same 1,411 kbps.
• The spectral cutoff is the highest frequency within 60 dB of the 1–8 kHz median, and it counts as an encoder’s lowpass when the level falls at least 25 dB across it. The smoothing pads with edge values: zero padding lifted every lossy file’s floor near Nyquist above the threshold, and the whole library looked full-band.
• Bit depth is exact, because a 24-bit integer survives float32, so 16-bit audio padded out to 24 bits shows.

Claims come from the file’s own header and tags, and from Rekordbox’s local database. A USB copy is matched to its database row by path, then by name and size (3,064 of 3,069), then by name only if no other row shares it. A shared name matches nothing, rather than putting another file’s claims beside this one’s measurements. Every threshold was set against the full library, and a flag with a “?” (transcode?, upsampled?) is evidence, not a verdict.

A full run takes 52 minutes and flags 484 of 3,664 files: 227 that clip, 196 likely transcodes (123 of them sold as 320 kbps), 66 where Rekordbox has the bitrate wrong, 23 mono tracks stored as stereo, one file claiming 8:04 that holds 7:00, and an AIFF whose header claims 1,118 bytes of a 48 MB file. The 66 are VBR files without a Xing header: Rekordbox reads the first frame, so 48 of them show 32 kbps. Honest encodes cut off where their bitrate says they should, 128s at 16.1–16.9 kHz, 192s at 18.8–19.1 kHz and 320s at 19.5 kHz and up.

It is slower than it should be on this machine, and not because of Python. Each track costs about 2 seconds of ffmpeg, but 8 jobs on 8 cores ran only 1.3–2.5 times faster than one, and plain ffmpeg writing to nowhere scaled just as badly. The antivirus was using about 2.5 cores to scan every file ffmpeg opened. Cancelling had its own trap: worker threads kept starting fresh ffmpegs from the queue in the half second before the main thread noticed Ctrl-C, and each decoded a whole track, so a cancel took 9.7 seconds. Every decoder is now registered, and a cancel kills those in flight and starts no more, which takes 0.8 seconds. Killing is safe here, unlike for sdm itself, because the decoders only read.

Matching a Spotify playlist against a folder

Login is OAuth with PKCE on a one-off loopback server, so no client secret exists anywhere, and it asks only for read-only scopes. Since Spotify’s February 2026 API changes, playlist contents come back only for playlists you own or collaborate on. For anyone else’s playlist, sdm reads a CSV export from a tool like Exportify, finding the columns by name.

Matching is by title and artist, because only 1.6% of my files carry an ISRC. Both sides are normalized:

• Words that name a release (original, extended, radio edit, remastered 2011) are ignored. Words that name a different recording (a remixer, dub, VIP, instrumental) have to agree.
• Artist lists are split on /, ,, &, x, vs and feat, and one shared artist is enough. Spotify credits remixers as artists, while tags usually put them in the title.
• Each file is read three ways: its tags, an “Artist - Title” inside the title tag, and its cleaned-up filename. The worst-tagged files are rips that carry the name in only one of those places.

Each track comes out as owned, missing or maybe. “Maybe” means a different version, or a file with no artist tag to confirm it. It exists because the costly mistake is a false “owned”: it hides a track you then never go and buy.

The catalog’s audit

The export writes a sidecar file listing every path it visited. Then it walks the folder a second time and compares the two sets, reporting anything missing, extra or listed twice. A clean run proves the export is complete instead of assuming it.

The web interface

The server uses only the Python standard library (ThreadingHTTPServer). The page is React and TypeScript, built with Vite. Its forms are generated from the CLI’s argparse tree, so a new flag on any command shows up in the browser with no web change. Submitted values are turned back into arguments and checked by a real parser before anything runs.

Each run is its own subprocess, for three reasons: commands report by printing, the output folders are process-wide settings, and a process can be stopped where a thread can’t. Cancelling doesn’t kill the run. The child runs in its own process group and receives CTRL_BREAK_EVENT, which it turns into a KeyboardInterrupt, so finally blocks still run and a borrowed fpcalc still gets deleted. Only a child still alive after 15 seconds is killed.

The server binds to 127.0.0.1, but any website open in the same browser can still send requests there. So it guards the API in layers:

• Each launch prints a link with a fresh token, which is exchanged for an HttpOnly, SameSite=Strict cookie.
• The Host header is checked, against DNS rebinding.
• Cross-site requests are refused by Sec-Fetch-Site and Origin, because SameSite treats every port on 127.0.0.1 as the same site.

Two Windows traps came up along the way. Python’s mimetypes reads the registry, where .js is often text/plain, which breaks module scripts, so the types are hard-coded. And SO_REUSEADDR lets two sockets share a port on Windows, so it is turned off there and SO_EXCLUSIVEADDRUSE is turned on.

Running in the browser

The online version is the same React page, built a second time with its API swapped for one that answers inside the tab. Python runs there through Pyodide in a Web Worker and calls the same cli.main as the command line, so no feature has a browser version of its own. The forms come from the same argparse tree, cut down to the commands that can run there. The site is static files on a Cloudflare Worker, rebuilt from the repo on every push.

• Nothing leaves the browser, and the browser enforces it. The page’s Content-Security-Policy allows connections only to its own origin, which serves static files and has nothing to receive an upload. That is also why everything, Pyodide and the Python wheels included, is self-hosted: about 16 MB on a first visit, cached after.
• Only what a command reads is handed to Python: audio files and the Rekordbox analysis folder. On the USB that’s 14,851 of 29,593 files; the rest is artwork.
• Reads are cached. Every read of a picked file is a round trip to the browser costing about a millisecond whatever its size, and tag readers make many small ones. Reading through 64 KiB blocks, four kept per open file, takes the missing-metadata check from 33 seconds to 18.
• Reports are byte-identical to a native run’s, apart from the paths. The catalog writes rows in the order the file system lists them, so the worker lists each directory the way NTFS does.
• pyrekordbox needs a stand-in. Importing its analysis reader runs the package’s __init__, which loads the database layer, and that needs SQLCipher, which has no Pyodide build. A bare module goes into sys.modules first, so only the analysis code loads.
• Cancel works like Ctrl-C. The page is cross-origin isolated, which gives it shared memory with the worker, so Cancel can interrupt Python mid-run and finally blocks still run. Each run writes into an empty output folder that is kept only if the run wasn’t cancelled, so a cancelled run leaves earlier reports as they were.

On the same USB, compared with running natively: the catalog takes 21 seconds (8.3 natively), analyze 85–88 (68), and the shopping list from a CSV 21 (8.3).