Objective
use a feature-based audio mosaic tool to create a musical statement
use a feature-based audio mosaic tool to create a musical statement
Outcome
an audio mosaic of bird songs that visualizes and sonifies the AI model's parameter k, a measure of similarity retrieval
an audio mosaic of bird songs that visualizes and sonifies the AI model's parameter k, a measure of similarity retrieval
How It Works
The project uses the audio programming language Chuck and the game engine Unity to create a musical statement. In ChucK different audio features are extracted from the GTZAN dataset and real-time concatenative sound synthesis is performed. My project allows for the exploration of the parameter k, a value for similarity retrieval used by the k-means machine learning algorithm to generate sound based on an input sound file. The minimalist educational tool lets the user explore the correlation between similarity retrieval and the sound that the model produces. It is an attempt at abstractly visualizing the dimensions of this music model and how the AI synthesizes sound - by taking one sample from the pool of similar samples in n-dimensional feature space and making an audio mosaic from it. The project is called K-TILE, as k decides which tile is picked for the mosaic of sounds.
The project uses the audio programming language Chuck and the game engine Unity to create a musical statement. In ChucK different audio features are extracted from the GTZAN dataset and real-time concatenative sound synthesis is performed. My project allows for the exploration of the parameter k, a value for similarity retrieval used by the k-means machine learning algorithm to generate sound based on an input sound file. The minimalist educational tool lets the user explore the correlation between similarity retrieval and the sound that the model produces. It is an attempt at abstractly visualizing the dimensions of this music model and how the AI synthesizes sound - by taking one sample from the pool of similar samples in n-dimensional feature space and making an audio mosaic from it. The project is called K-TILE, as k decides which tile is picked for the mosaic of sounds.
The choice of visuals and colors was inspired by artworks of Piet Mondrian and the artistic movement De Stijl. De Stijl was influenced by Bauhaus which was a school of design with strong modernist influences. Bauhaus emphasized simplicity and effectiveness as well as modularity in their views on architecture. Like Bauhaus design, the audio mosaic is modular allowing for an abundance of expression through combining ready-made elements in new ways. The choice of sound might be surprising but I though that the juxtaposition of the audio making process - something so highly technological - and the most natural thing - the song of a bird might be a combination that elicits wonder and joy. It is not an attempt at humanizing AI, but rather a way of saying that not everything AI-related needs to be serious.
Acknowledgement
The starter code for this project was provided by Ge Wang, Yikai Li and Andrew Zhu Aday.
The audio is synthesized from a video of 50 species of European birds and their bird songs by Wildlife World.
GitHub Repo
By using the W and S key, k can be increased and decreased respectively to change the sound and the visuals.
Tools
ChucK - Audio programming
Unity - Visuals
GTZAN dataset - Music Genre dataset
The starter code for this project was provided by Ge Wang, Yikai Li and Andrew Zhu Aday.
The audio is synthesized from a video of 50 species of European birds and their bird songs by Wildlife World.
GitHub Repo
By using the W and S key, k can be increased and decreased respectively to change the sound and the visuals.
Tools
ChucK - Audio programming
Unity - Visuals
GTZAN dataset - Music Genre dataset
