Piano-Playing Robot

It took him around four months, on and off, to develop his Piano-Playing Robot. As shown in his YouTube project video (), it features 15 solenoids mounted on a wooden frame placed on top of an electric piano keyboard.

“Basically, the system is quite simple,” reveals Étienne. “Each solenoid pushes on a piano key and is controlled by a specific GPIO pin of a Raspberry Pi 3 Model B.” Since the latter doesn’t have enough electrical power to activate the solenoids directly, an electromechanical relay module acts as a buffer between the two.”

Fast ‘fingers’

The system can read a MIDI file, convert it into on and off signals for each note, and play it. A basic UI allows the user to choose between scales, arpeggios, or melodies.

“As of now, the robot can play a full scale in just under two seconds,” reveals Étienne. “That is, at the fastest rate, it plays about eight keys per second.” This impressive speed makes it possible to get the timing of the notes just right.

“However, the system is limited by the push and pull times of the solenoids, which add up a few tens of milliseconds,” he says. “At a first glance, it may not seem like much, but in a fast melody it can make a difference. Our robot still has a long way to go to overtake human-beings: the fastest pianist in the world claims to be able to reach up to almost 20 keys per second!”

Sight reading

So far, he has programmed the robot to play six scales, six arpeggios, and five different melodies. The most impressive feature, however, is its ability to read previously unseen sheet music – using a Raspberry Pi Camera Module – and play it.

“A camera was added to the system to read and decode new musical scores that had never been shown to the robot before,” says Étienne. “It then scans, decodes, and plays them.” To do this, it uses a free open-source tool called Audiveris. “It is a complex program for optical music recognition (OMR), which is recognition of musical symbols. Unlike optical character recognition (OCR), this task requires decoding the configuration of symbols in two dimensions. In addition, the order of magnitude of the symbols is extremely wide (sometimes ranging from tiny dots above musical notes to symbols that are drawn across a large part of the page).”

In brief, the recognition process is done with classifiers and neural networks that make matches with symbols previously learned. “For my project, the decoding part was done on a laptop since Audiveris was more easily compatible with Windows,” he explains. “On average, it takes around 30 seconds for the robot to decode a sheet of music containing a few staves.” Étienne is hoping to add extra features to the robot. “In particular, I would like to extend the concept to 88 solenoids. This way, the robot would theoretically be able to play all the existing melodies on the piano! Obviously, there is a lot of development to do in terms of Optical Music Recognition, but I like to imagine a piano concert performed entirely by a robot!”