The AI research that helps pull a voice out of a crowded recording

The voices in an old recording may already contain everything an editor wants to hear. The difficulty is that instruments, room noise and other voices arrived at the microphone together. Pulling them apart turns a single piece of audio into something that can be worked on again.

A profile published by MIT News on 28 October follows Paris Smaragdis, whose research helped separate sounds in the archival material used for The Beatles: Get Back. Adam Zewe’s report traces the professor’s path from music and synthesis to machine listening, including work on systems that understand sound and tools that could help musicians improve.[1]

Smaragdis now holds appointments in MIT’s music and electrical-engineering programmes. His own research page places source separation at the centre of his audio work. Two earlier papers make the underlying challenges unusually tangible: identifying the parts of a mixture, and choosing a representation that does not lock a model to one recording format.[4]

A voice is not a separate track until the system estimates it

In a 2015 paper, Smaragdis and colleagues studied separation from a single audio channel. This is an underdetermined problem: several combinations of original sounds could produce the same mixture. Their method learned to estimate the component sources and used a time-frequency mask to allocate portions of the mixture to each one.[2]

A mask is useful to picture as a changing filter. At one moment and frequency it can retain more of the signal for the voice; elsewhere it can assign more to the accompaniment. The network learns from examples, while the masking relationship connects its estimates back to the recorded mixture.[2]

The paper evaluated speech separation, singing-voice separation and speech denoising against earlier models. Those experiments establish results for defined datasets and tasks. They are earlier research, not a specification of the production system used for Get Back.[2][1]

For an editor, the resulting control is the attraction. If a dialogue track and an instrument can be handled separately, changing one need not require making the same change to the other. Separation creates room for a human decision about the balance of the recording.

The format can become part of the problem

A second paper, published in 2021 with Krishna Subramani, asked why audio models should depend on a fixed grid of frequency bins. Changing the sampling rate or analysis window can change the input dimensions expected by a conventional model, even when the underlying sound remains recognisable.[3]

Their alternative represented audio as points carrying frequency and magnitude, with time included when needed. In environmental-sound classification experiments, that approach could accommodate varying representations and use fewer model parameters. With full inputs, training iterations required two to three orders of magnitude more operations than the conventional comparisons. Subsampling reduced that count by roughly a factor of ten, with a small accuracy loss.[3]

This is a useful distinction for audio software: a compact model is not automatically a cheap one to train. The experiment opened up a more flexible way to describe sound while exposing another cost that a practical system would need to address.[3]

Taken together, the work shows how much of machine listening happens before a system can answer a question about a recording. It must decide how to represent the signal and how to distinguish the sources within it. Better answers at those stages can give a musician or editor something very concrete: access to a sound that was present all along.

Report an error or suggest a correction