The tiny AI changing a synthesiser’s sound between notes

At Canberra’s SoundOut festival this year, Charles Patrick Martin’s instrument combined a Roland S-1 synthesiser, a QuNeo controller and a small computer running IMPSY. His published account shows the equipment he brought to performances with Australian and international musicians, including collaborators he had not met before. The AI was part of a playable setup in an ensemble.[1]

Martin’s NIME 2026 paper traces five instrument designs through his own performances and recordings between 2024 and 2026. One discovery was especially productive: let a model change several sound controls together, then let the musician intervene through the instrument’s ordinary controls.[2]

IMPSY itself produces no audio. It sends instructions to a synthesiser or other sound-making software, using connections such as MIDI. Those instructions can choose notes, but they can also change the settings that determine a note’s tone. This puts the model inside the process of playing an instrument.[3]

A tenth of a second changes the relationship

With an Arturia MicroFreak, Martin assigned the system note control and seven timbral parameters. It could alter several settings at once, changing the sound between phrases and sometimes between notes. He tried reducing the handover delay to 0.1 seconds. The model’s contribution could then fit inside very small gaps in his playing.[2]

The current configuration documents the rule precisely. In call-and-response mode, IMPSY waits for a chosen interval without user input, then generates its response. When the musician moves again, the model stops. The trigger is a performance gesture; the player does not need to step outside the music to issue an instruction.[4]

This makes the waiting time a musical control. A long pause gives a reply its own space. A very short one lets the sound change as the musician alternates between notes and knob movements. The same control relationship can produce a spacious exchange or a restless, continually interrupted texture.[4]

The later Roland S-1 setup gained controllers with larger knobs and visual feedback, then a QuNeo with pads and touch strips. Martin describes those physical changes as making the instrument easier to manage across different improvising groups. Improving the interface created new musical possibilities without requiring a new model for every setup.[2]

Learning the shape of a gesture

IMPSY uses a recurrent neural network to follow a sequence of control values and predict their continuation, including when the next values should occur. The training material can be a musician’s own recorded interaction. The toolkit logs incoming performance data, provides a step for turning the logs into a dataset and then trains a model that can be loaded for another session.[3]

Timing is part of that learned material. In the documented configuration, a nine-value model represents eight musical parameters plus the interval between events. A timescale control stretches or compresses the predicted intervals. Other controls adjust how tightly predictions stay near the learned patterns and how much variety sampling introduces.[4]

The model’s inputs and outputs are mapped separately. A stream of learned values can therefore be connected to different controls or sound generators. Changing that mapping is a form of instrument design: the musician decides what a model’s movement will mean in the particular setup. The project deliberately keeps those connections separate from the trained network.[5]

That separation also makes the model portable. IMPSY runs on ordinary computers and Raspberry Pis; its current companion app brings the same model format into MIDI effects workflows on iOS and macOS. The app’s mapping screen lets each dimension control a note, a controller message or pitch bend. It can record session logs for later training.[5][6]

For someone who already uses AUM on an iPad or Logic Pro on a Mac, the documented app route offers a way into this kind of playing without assembling a Pi instrument. It requires iOS or iPadOS 17, or macOS 14 on Apple Silicon. The sound still comes from the instrument receiving the MIDI messages.[6]

Martin’s festival account moves easily from the equipment to the people he played with. That is an appealing destination for an AI music system: a component a musician can learn, adjust and bring into a room with other musicians, ready to contribute to whatever they play together.[1][5]

Report an error or suggest a correction