I started by doing research on digital instruments design, where a good resource for research is New Digital Musical Instruments[1], which presents an extensive analysis of a wide range of projects. Although the frameworks presented are particularly focused on instruments rather than on experiences, most of their principles can be applied to my project as analysis tools. The authors state that there are six steps for designing DMI’s, outlined below:
- Decide on the gestures used as controls
- Decide the gesture capture strategies
- Define sound synthesis algorithms
- Map the sensor outputs to the synthesis and music-control inputs
- Decide on feedback modalities
GESTURES
Gestures can be classified into two types according of their physical interaction with the controls. The Semiotic (or Naked Gestures) and the Ergotic (or Manipulative Gestures)[2]. On the former, there is no direct physical contact with the device that crates the sound. An example for this would be David Rokeby’s Very Nervous System, where sound is controlled through a videocamera reading body movements. In contrast, an ergotic interface could be any traditional instrument, or the Reactable, where there is some sort of physical contact with the device.
Most of the prototypes I’ve constructed have explored semiotic gestures, so I thought it would be valuable to try ergotic gestures at this stage. The Pyramid and Pendulum prototypes explore around this idea, where there is a direct interaction with the controllers, so the interaction is mainly ergotic. However, using computer vision techniques allow semiotic gestures to be involved as well, creating an interesting blend of two territories.
CAPTURE STRATEGIES
The Play Station Eye camera has been the choice for most of my prototypes because of its fairly good optics compared to a regular webcam. As a capture technique, color tracking has been successful so far, but I’m aware that it does not work well in low-light conditions. Using a camera that is capable of capturing infrared light could solve this issue, allowing any kind of light conditions. In this sense, the Microsoft’s Kinect camera features an infrared sensor so it worths experimenting with it. The Kinect also features a depth sensor based on laser beams, providing a new spatial dimension that can be used as control. I will definitely test how this camera behaves in the following prototypes.
SOUND SYNTHESIS
Throughout the prototype’s iterative process, I’ve created a model that has given good results for audio synthesis. As explained in the Capture Strategies section, the gestures are captured using camera vision, more specifically, color tracking techniques. A program written in openFrameworks[3] analyses the data obtained from the tracking points and maps this information into parameters. The variations of these parameters are translated into messages and sent–using OSC protocol–to a second software called Osculator[4], that transforms this data into midi messages. A third software, capable of receiving midi messages from other softwares called Ableton Live[5], is used to produce sound. It was chosen for its capacity to easily produce controllable sound loops with a good sound quality.
MAPPING
So far, trying different mapping alternatives has been one of the main differences between prototypes. Probably the way the gestures are mapped into sound is the most influential variable, and it profoundly impacts on the kind of interaction: it sets the rules, affordances and constraints for the experience. Although a good amount of alternatives have been explored, I cannot make a definitive commitment to a certain mapping at this point.
However, one of the most interesting experiences took place with prototype # 5, the Pendulum, that incorporates–for the first time–the use of physical objects operated by participants. I plan to further explore the use of objects that inherently imply physical challenges of operation.
FEEDBACK
To balance visual and sonic feedback is challenging. Visual feedback is very useful for understanding how the system works, but an excessive presence can diminish the level of attention between players. When the visual feedback is strong or extremely seductive, people tend to give most of their attention to it, diminishing awareness about the musical and social experience[6]. Taking this into account, my project will use primarily sound feedback, making use of the visual aspects as a complement for the sound experience.
Having a system that is highly responsive to the participants’ gestures will be extremely important. As novice players are the main target, the musical range will be highly constrained, but this can be balanced by providing different control modalities, like controlling low-pass filters. A good balance between constraints and controls will allow to have expressive and very reactive feedback to users’ gestures.
THE MISSING VARIABLE
As the model I’ve been using for this section aims specifically for DMI’s, when applied to experiential design, it lacks of a last step: choosing a proper context. I feel that as my goal is not to create an instrument, but more an interactive sound space, the context where the experience takes place is as relevant as the interface or the sound itself. When I performed tests with Trinidad, I realized that this was an extremely important component to provide the appropriate conditions for the experience. I can see now that the interface is not to be designed alone, but as a tandem with the space that contains it.
Some questions arise here: How large needs the space to be? What are the lighting conditions? What are the sound conditions? Will the space take an active or a passive role in the interaction (i.e. the light conditions can be controlled by the participants, as a complementary reaction to the inputs).
In conclusion, some of the decisions I’ve made during this stage are: Will continue exploring the use of physical controllers, as the interaction seems to be more appealing when they are present. Mapping, while reducing the range of musical possibilities (melodies), must augment the sense of control and expressiveness. The context must be adequate for the scale (number of participants), and encourage intimate interactions.
___________________
[1] Miranda, Eduardo R. and Wanderley, Marcelo M. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Wisconsin: A-R Editions, 2006.
[2] Miranda, Eduardo R. and Wanderley, Marcelo M. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Wisconsin: A-R Editions, 2006.
[3] http://www.openframeworks.cc/
[4] http://www.osculator.net/
[5] http://www.ableton.com/live-8
[6] Blaine, T. and Perkis, T. The Jam-O-Drum Interactive Music System: A Study in Interaction Design. In Proceedings of Symposium on Designing Interactive Systems. 2000, 165-173.