Pitch-class Steering for Diffusion-based Music Generation via Latent-space Probes

Researchers published a paper on arXiv, proposing a lightweight method based on convolutional probes for guiding audio generation based on diffusion models. This method involves training approximately 125,000 parameters of frozen convolutional probes, using paired audio and MIDI data to decode frame-level pitch categories in the latent space. During the inference phase, these probes act as differentiable loss functions, whose gradients relative to the denoised latent variables guide the generation process toward the user-specified pitch sequence, without the need for re-training or modifying the underlying model architecture. In 27 evaluation experiments involving 9 text prompts and 3 target melodies, the method resulted in 2.4 times higher coherence of generated melodies compared to the unguided baseline (p < 1e-5), demonstrating that recoverable and controllable musical meaning structures can be created in the diffusion-based musical latent space.