This experimental project investigates potentials of translating data as art technique by building a synaesthetic environment.
Materials
Arduino UNO, stepper motor, servo motors, laptop, LCD screen, stereo, synthesizer, wires, laser-cut MDF, laser-cut paper, tracing paper, RGB LEDs.
A Synaesthetic Machine for Real-time Colour-Sound-Motion Correlation
What are the aesthetic potentials if an image (i.e. painting) can be converted into kinetic movements and music? What are the possibilities of constructing a synaesthetic environment by digital means? Experiential Converter investigates such notion by applying ‘data mapping’ as an art technique. It constructs colour – sound – motion correspondence in a real-time setting. It explores synesthesia experience, colour-sound correspondence, and technical data processing as a method of experience design.
Integrating a technical process of “reverse engineering”, Experiential Converter analyzes colour information of a painting and generates music by mapping the colour information into sound tones. This project demonstrates a process of sensory data-conversion and explores how images can be perceived with alternative sensory organs. Experiential Converter demonstrates how images can be experienced as “synaesthetic abstraction”, in a way it is generative and enriched through musical and kinetic performance.
Technical Description
This experiment contains three prototypes:
A software that maps images into colour wheel.
Colour Collector Machine
Motion Converter Machine
Firstly, a software is design in Processing to map an image into a colour wheel. The image is scanned to obtain all its colour information which includes the RGB values and the frequency colours are used. In order to decrease colour difference from the original (i.e. oil paintings), images are obtained from online digital archives. The colour wheels are printed out on card boards.
The Colour Collector is a machine designed to detect and deliver colour information from the colour wheels. It is mainly a composition of a motor driving turntable and a colour sensor. The Colour Collector delivers RGB values to two domains. By Open Sound Control (OSC) protocol, the RGB value is send to PureData to generate sound. Higher colour tones generate higher-pitch sound to provide a perceivable correspondence for the viewer. In the mean while, the RGB values are sent to Motion Converter machine through I2C pins on Arduino boards.
Motion Converter is a machine that converts colour into spatial positions. It is mainly designed with three servo motors and RGB LEDs. By mapping the R, G, B values into motor angle between 0 – 180 degree, each color can be represented with a unique position in space. Therefore, the higher contrast an image is, the more kinetic motion it becomes. The LED lantern is showing the colour the machine is processing.
Codes archive at Github.
編碼分享於Github開放編碼庫：https://github.com/pingyehli/ExperientialConverter
Some documentation of this work and other researchers’ works:
