He collaborates with an ensemble of AI agents: "I think it could open exciting possibilities for music production"

Have you ever seen birds flying across the sky in shifting, mesmerising patterns? Or ants using their own bodies to form a living bridge that other ants can walk across?

In nature, organisms we normally think of as simple can solve remarkably complex tasks when they act together. For example, they can travel distances that would be impossible for a single individual.

This kind of behaviour can be recreated using mathematical models and algorithms to help us better understand how nature works. But it can also inspire artificial intelligence (AI) systems, including those used in music.

“With the AI boom of the last few years, many people have become interested in how humans and machines can create music together. One way to explore this is to look at how we, as humans, can collaborate with an ensemble of AI agents,” says Pedro Pablo Lucas Bravo.

Studying the interaction between agents and humans

Bravo is pursuing a PhD at the University of Oslo's RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion. There, he draws on insights from animal behaviour in his research.

His research has two main parts:

“First, I study how a swarm of AI agents behaves when they are involved in creating music. What do they do together, and how do they shape the music?” he says.

The second part is to study how humans can collaborate with these agents.

“If we can establish a good form of collaboration, I think it could open exciting possibilities for music production and performance,” he says.

The systems require very little data

AI is already widely used in music production, both as a creative assistant and as a tool for speeding up routine tasks.

Tools such as Suno allow users to generate music, including vocals and instrumentation, given text or audio prompts.

When AI systems generate realistic-sounding music, they typically rely on huge datasets: millions of audio samples or many thousands of hours of music.

Working with a swarm of AI agents is different. These systems are built on very simple rules and often require little data – sometimes almost no data at all, explains Kyrre Glette, a professor at RITMO and Bravo’s PhD supervisor.

“It provides a different way of expressing yourself creatively. You’re working with a model based on micro-rules, and together they create a fascinating, dynamic system,” he says.

The basic idea is to start with a type of biological behaviour and translate it into a set of rules. Usually, all the agents follow the same simple set of rules – for instance, one inspired by how ants move and coordinate.

But when you put many agents together, their interactions can lead to unpredictable yet rich musical behaviour, like complex musical patterns in melodies or harmonies.

A live AI experience

So-called swarm systems, or multi-agent systems, are already used in areas such as robot control and computer animation. However, relatively few people have explored their potential in music.

In his research, Bravo has designed and adapted swarm systems that enable interaction with humans. He calls these systems Human-Swarm Interactive Music Systems (HS-IMSs). He built on earlier research in the field.

He explains that the systems can provide a 'live' experience in which humans and AI improvise music together.

“The swarm produces its own music, and the user tries to use the swarm's interactive capabilities to shape a different intention than the autonomous behaviour. That's why there's a trade-off between control and autonomy, since the user might not have full control over everything happening,” he says.

Using his hands to shape the sound

When Bravo explores the behaviour of the agents, he works in what is known as mixed reality.

He wears an XR (extended reality) headset that lets him see both the physical room he’s in and the virtual agents. They appear as floating building blocks arranged in different formations around him.

He can use his hands to move these blocks closer together, push them apart, or rearrange them in space.

From the outside, it looks as though he’s just gesturing randomly in the air.

But in reality, he’s improvising music.

By manipulating one agent, he explores how the others respond. How does the overall sound change? Which musical textures emerge? How much control does he have over the swarm?

If one drum fails, the others keep going

“The agents know which group they belong to, and they know they must move if one of the others is moved. But we can’t predict exactly what the result will be,” says Bravo.

One key advantage of swarm systems is their robustness. If one agent – for example, a virtual drum – fails or is removed, the others continue to do their job. The overall system keeps functioning.

The systems he has developed are currently designed for musical improvisation.

The long-term goal is for them to also be used to explore musical ideas, create new music, and perhaps even enable new forms of live performance, he explains.

“The potential of these systems lies in balancing their chaotic tendencies with good control mechanisms. The agents have a behaviour of their own, but if we can find a sweet spot where humans have just the right amount of control over them, we can create exciting musical experiences,” says Bravo.

Reference:

Bravo, P.P.L. Human-Swarm Interactive Music Systems, Exploring Collaborative Sonic Creation Between Humans and Autonomous Agents, Doctoral thesis at the University of Oslo, 2026. 

This content is paid for and presented by the University of Oslo

This content is created by the University of Oslo's communication staff, who use this platform to communicate science and share results from research with the public. The University of Oslo is one of more than 80 owners of ScienceNorway.no. Read more here.

More content from the University of Oslo:

• 

  The likelihood of relationship breakdown may be influenced by our genes

• 

  Technology is making the music industry more individualistic and less equal

• 

  Researchers: Digital ID management in Norway is a catastrophe

• 

  Archaeology is helping researchers understand the effects of forced displacement

• 

  For 'Amalie,' going to prison was devastating. But it helped her get sober

• 

  “My pulse quickens every time someone says that children and young people are digital natives"