Making Music With Machine Learning: Introduction to Magenta Music.js
The idea behind Google’s Magenta is fascinating — combining machine learning with art, in an effort to “[build] smart tools and interfaces that allow artists and musicians to extend (not replace!) their processes”. The demos page illustrates how fun and dynamic these tools can be — here are a few of my favorites.
The @magenta/music library is abstracted over the WebAudio API (through Tone) and TensorFlow. That combination means Magenta is capable of both playing music and using machine learning models to generate music. There are also models for generating visual art and working with images, but I won’t go into those here.
Naturally, I wanted in on the fun. In case you do too, here’s how I approached this.
Want to read this story later? Save it in Journal.
For the sake of convenience, I just made this in an HTML file, but the package is available through npm as well.
<!DOCTYPE html>
<html>
<head>
<meta name='viewport' content='width=device-width, initial-scale=1'
<script
src="https://cdn.jsdelivr.net/npm/@magenta/music@^1.0.0">
</script>
<link rel="stylesheet" href="styles.css">
</head>
</html>And just like that, we’re ready to begin.
Magenta music is built around the idea of NoteSequences, which are numerical representations of musical notes. Here’s a NoteSequence representing “Twinkle Twinkle Little Star” that I borrowed from the docs:
TWINKLE_TWINKLE = {
notes: [
{pitch: 60, startTime: 0.0, endTime: 0.5},
{pitch: 60, startTime: 0.5, endTime: 1.0},
{pitch: 67, startTime: 1.0, endTime: 1.5},
{pitch: 67, startTime: 1.5, endTime: 2.0},
{pitch: 69, startTime: 2.0, endTime: 2.5},
{pitch: 69, startTime: 2.5, endTime: 3.0},
{pitch: 67, startTime: 3.0, endTime: 4.0},
{pitch: 65, startTime: 4.0, endTime: 4.5},
{pitch: 65, startTime: 4.5, endTime: 5.0},
{pitch: 64, startTime: 5.0, endTime: 5.5},
{pitch: 64, startTime: 5.5, endTime: 6.0},
{pitch: 62, startTime: 6.0, endTime: 6.5},
{pitch: 62, startTime: 6.5, endTime: 7.0},
{pitch: 60, startTime: 7.0, endTime: 8.0},
],
totalTime: 8
};It’s important to note what time means in a given context: in the sequence above, those times refer to seconds (ie, play the first note for half a second). This is what magenta would refer to as an “unquantized” sequence.
A quantized sequence, on the other hand, means the notes are defined in “steps”. This allows for greater customization and better consistency: here’s another example of a quantized sequence (in this case a drum pattern) from the docs:
DRUMS = {
notes: [
{ pitch: 36, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true },
{ pitch: 38, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true },
{ pitch: 42, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true },
{ pitch: 46, quantizedStartStep: 0, quantizedEndStep: 1, isDrum: true },
{ pitch: 42, quantizedStartStep: 2, quantizedEndStep: 3, isDrum: true },
{ pitch: 42, quantizedStartStep: 3, quantizedEndStep: 4, isDrum: true },
{ pitch: 42, quantizedStartStep: 4, quantizedEndStep: 5, isDrum: true },
{ pitch: 50, quantizedStartStep: 4, quantizedEndStep: 5, isDrum: true },
{ pitch: 36, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true },
{ pitch: 38, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true },
{ pitch: 42, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true },
{ pitch: 45, quantizedStartStep: 6, quantizedEndStep: 7, isDrum: true },
{ pitch: 36, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true },
{ pitch: 42, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true },
{ pitch: 46, quantizedStartStep: 8, quantizedEndStep: 9, isDrum: true },
{ pitch: 42, quantizedStartStep: 10, quantizedEndStep: 11, isDrum: true },
{ pitch: 48, quantizedStartStep: 10, quantizedEndStep: 11, isDrum: true },
{ pitch: 50, quantizedStartStep: 10, quantizedEndStep: 11, isDrum: true },
],
quantizationInfo: {stepsPerQuarter: 4},
tempos: [{time: 0, qpm: 120}],
totalQuantizedSteps: 11
};You’ll notice that the stepsPerQuarter key is set to 4. This means there are 4 steps per quarter note. The qpm key in tempos refers to how many quarter notes per minute. If you adjust the qpm, the tempo changes accordingly. the totalQuantizedSteps tells Magenta how long the sequence you’re passing it is — its value should match the quantizedEndStep value of the last note(s) played.
There are just a few more steps we have to do before we can give magenta our sequences. First, we need to set up a player for our sounds. Here are a few options:
//the basic player will play a synth sound by defaultplayer = new mm.Player()//the SoundFont player will play any soundfont you want--this one in particular plays a piano sound.soundFontPlayer = new mm.SoundFontPlayer('https://storage.googleapis.com/magentadata/js/soundfonts/sgm_plus');
Personally, I prefer the SoundFont player. But either works beautifully. To start and stop playing, Magenta offers very intuitive commands.
soundFontPlayer.start(TWINKLE_TWINKLE);
soundFontPlayer.stop()Ok, so now that we can play a noteSequence, let’s move on to the next step — sending it to Magenta and letting it continue the melody.
Part of what makes @magenta/music so wonderful to work with in the browser is that it gives you access to pre-trained models through a simple api call.
Since my goal is to continue a melody, I’m going to use the MusicRNN model (which specializes in doing exactly that). First, we’ll initialize the model and point it to the checkpoint. You can find a full list of the available checkpoints here.
//
music_rnn = new mm.MusicRNN('https://storage.googleapis.com/magentadata/js/checkpoints/music_rnn/basic_rnn')
music_rnn.initialize()Then, we create a new player to play the response noteSequence we’ll get back from the model.
rnnPlayer = new mm.Player();function play() {
if (rnnPlayer.isPlaying()) {
rnnPlayer.stop();
return;
}
Next, we’ll have to quantize the Twinkle Twinkle Little Star note sequence (using a convenient helper method magenta provides)
//quantizing the former note sequence (the 4 referencing the stepsPerQuarter value)
const qns = mm.sequences.quantizeNoteSequence(TWINKLE_TWINKLE, 4)//this number will determine the length of response, so I set it high to hear a longer continuation of the melody.
let rnnSteps = 40//this controls how close to the original melody the response will be: a higher "temperature" means more randomness, a lower one means less.
let rnnTemp = 1music_rnn
.continueSequence(qns, rnn_steps, rnn_temperature)
.then((sample) => rnnPlayer.start(sample));
}
And voila! Just like that, we can hear magenta continue the melody.
There are so many crazy things magenta gives you access to, and I’ve only barely scratched the surface. If you want to learn more, I recommend:
This blog post that helps walk you through getting started
This blog post describing making a neural melody maker
I also want to emphasize that I’m new to magenta and machine learning generally, so if I’ve misunderstood something please let me know. But I’ve been floored by how easy it is to work with magenta, and I’m very excited to do more projects with it in the future. I encourage everyone to try it out.
📝 Save this story in Journal.
👩💻 Wake up every Sunday morning to the week’s most noteworthy stories in Tech waiting in your inbox. Read the Noteworthy in Tech newsletter.
