For Gabriel Johnson's video documentary about this project click here.
In July 2017, following the success of the ‘Sensing Culture’ project involving blind and visually-impaired students from across East Kent, students from Simon Langton Grammar School for Boys were invited to develop a set of musical compositions to paintings at the Beaney House of Art and Knowledge. Using Cal Hewitt’s extraordinary new creation in a gallery for the first time meant that the students were able to bring those paintings to gallery visitors at a number of public events, and make a particular impact on visitors with visual impairment.
Here is the project in the words of Luke Rennells, a student at Simon Langton Grammar School for Boys who composed for the events of that summer:
N.B. ‘The Butterfly Machine was the early name for what later became known as Tonesight’
To most Langton News readers, ‘The Butterfly Machine’ will be an unfamiliar and rather strange name. It is the title of a project in the Music Department, employing the help of a range of students (including myself) to develop a unique piece of technology. Since the likes of it have never been seen before, here's how it works...
The Butterfly Machine was originally thought up by Cal Hewitt of the Simon Langton Sixth Form, combining his passion for music and computing. It began with experimenting how movements and a Kinect sensor could be use to interact with music spatially through a Python program. This developed into the idea of interacting with paintings to produce sounds. The initial test was with a butterfly exhibit (hence the name) at the Beaney Museum on Canterbury. Loops of music were mapped to points on the painting, and through the sensor, a circle of interaction on the painting would play the music wherever the user pointed. The Butterfly Machine was born! We now aim to help visually impaired students appreciate artwork through this aural interpretation.
Fast-forward to July 2017. Having introduced the idea to KS3 students, a team was assembled to develop the first prototype of the Butterfly Machine at an exhibition on Friday of the same week. This included Mr Navarro-Pollott as project manager, Cal Hewitt as the producer, A-Level design students Dominic Schoen and Shannon Keough, and a group of composers, including Gabriel Johnson as cinematographer.
Step 1: Head to the Beaney gallery for inspiration. Off we went one morning, inventing melodies and motifs for every aspect of our chosen paintings. In future, we hope, artists themselves can present their own aural version of their pieces, not limited to music. Happy with our impressions, it was back to school with us to bring our ideas to life using software such as Sibelius or Logic Pro X, or recording us playing everything from violins to a tuba.
Meanwhile, the design students were hard at work in the workshop, creating the machine itself. The current prototype reuses a shopping trolley, which was cut in half. It consists of a touchscreen laptop with the software loaded, a Bluetooth speaker, a Kinect sensor, and all the wires in-between. The idea was to create something mobile and easy to use.
The software, too, is easy to understand. Musicians can create their own individual tracks as sound files. Each file can be placed as a green dot on a picture, so that if the hand is detected over it, it will loop in time with the rest.
To the user, they are presented with a list of the paintings, which when tapped on, will activate the specific project. They can then move around, zoom in or out, and experience the music of the painting, all with their hand.
With some rehearsals, we packed up for the Beaney. We had everything we needed for the reveal at the Friday exhibit before an audience of the blind, partially-sighted and anyone else who wanted to have a look! There were live performances of an extended version of an ensemble piece ('A Canterbury Pilgrimage') I composed originally for the Butterfly Machine, a lyrical song with piano backing by Greg Schoen and Sonny Sheehan, and even Acey Court's one-man tuba piece with a loop station! Following that, everyone got a go with the Butterfly Machine first-hand. Everything went perfectly, and everyone left satisfied. The Butterfly Machine was in business!
This was followed in August by another workshop at the Beaney Museum for blind and partially-sighted children and adults, and again in September, when the Butterfly Machine became one of the star attractions at the museum's 5th birthday celebrations.
Looking ahead, we will continue to work on developing the Machine through to the New Year and beyond! The ultimate ambition is to create at least two fully-certified copyrighted prototypes of the Machine for public use, with one at the Beaney. Also, we hope to have five more 'shells', if you will, available for purchase by other establishments.
Cal may also work on creating an open-source version of the basic software for use in school environments. We hope to make this all possible with the help of grant funding, and hope to show it at an RNIB 'Sensing Culture' conference in February 2018, so that our invention can be an asset to music departments and galleries alike.
But fundamentally, we just want to help those who are visually challenged, aspiring composers, and, indeed, anyone else, to experience art in a completely new way, and appreciate an entirely new depth to every piece.
We encourage anyone with interest in the powerful potential of music to contribute to this amazing long-term project.
For those interested, here is a video summarising the above - or you could just search for 'Langton Butterfly Project' on Youtube!