Music control with ml5.js

ml5.js is a library that provides easy integration of machine learning algorithms and models in the browser and is built on TensorFlow.js.

In this example, you can control the speed of music in a simple way with the interaction of an image classification machine learning model.

Getting Started

Since ml5.js runs on the browser you need to include the scripts to the header of your html file, including p5.js:

<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.8.0/p5.min.js"></script>   
<script src="https://cdnjs.cloudflare.com/ajax/libs/p5.js/0.8.0/addons/p5.dom.min.js"></script>   
<script src="https://unpkg.com/ml5@0.4.3/dist/ml5.min.js"></script>

Add the music into your html file, in this example I am playing The Symphony №5 in C minor of Ludwig van Beethoven.

Let’s create the model

Teachable Machine is a fast and easy way to create machine learning models for your sites, you just need to collect your examples, then train the model and export the model.

For my model, I have created 3 clasess, left, right and nothing:

Left: to play slow

Right: to play fast

Nothing: no action

Once finished the training, get the model as a sharable link.

Add the model to your js file

First, create some variables for video and model:

Set up

Now, let’s create the functions:

1. In preload() function, we load the model to be sure is ready before we can apply the image classification.
2. In setup() function we create the canvas and flip the video so it is rendered as a mirror.
3. In classifyVideo() function, we classify the flipped video again. In gotResult() function, we use two parameters, one to log the error if there is an error or to return the results of the classifier.
4. In draw() function, we render the video, the labels and the commands to set the rate at which the media is being played back.

“The audio is muted when the fast forward or slow motion is outside a useful range (for example, Gecko mutes the sound outside the range 0.25 to 5.0)”

Play!