Download the pdf version HERE

I started by doing research on digital instruments design, where a good resource for research is New Digital Musical Instruments[1], which presents an extensive analysis of a wide range of projects. Although the frameworks presented are particularly focused on instruments rather than on experiences, most of their principles can be applied to my project as analysis tools. The authors state that there are six steps for designing DMI’s, outlined below:

• Decide on the gestures used as controls
• Decide the gesture capture strategies
• Define sound synthesis algorithms
• Map the sensor outputs to the synthesis and music-control inputs
• Decide on feedback modalities

GESTURES
Gestures can be classified into two types according of their physical interaction with the controls. The Semiotic (or Naked Gestures) and the Ergotic (or Manipulative Gestures)[2]. On the former, there is no direct physical contact with the device that crates the sound. An example for this would be David Rokeby’s Very Nervous System, where sound is controlled through a videocamera reading body movements. In contrast, an ergotic interface could be any traditional instrument, or the Reactable, where there is some sort of physical contact with the device.

Most of the prototypes I’ve constructed have explored semiotic gestures, so I thought it would be valuable to try ergotic gestures at this stage. The Pyramid and Pendulum prototypes explore around this idea, where there is a direct interaction with the controllers, so the interaction is mainly ergotic. However, using computer vision techniques allow semiotic gestures to be involved as well, creating an interesting blend of two territories.

CAPTURE STRATEGIES
The Play Station Eye camera has been the choice for most of my prototypes because of its fairly good optics compared to a regular webcam. As a capture technique, color tracking has been successful so far, but I’m aware that  it does not work well in low-light conditions. Using a camera that is capable of capturing infrared light could solve this issue, allowing any kind of light conditions. In this sense, the Microsoft’s Kinect camera features an infrared sensor so it worths experimenting with it. The Kinect also features a depth sensor based on laser beams, providing a new spatial dimension that can be used as control. I will definitely test how this camera behaves in the following prototypes.

SOUND SYNTHESIS
Throughout the prototype’s iterative process, I’ve created a model that has given good results for audio synthesis. As explained in the Capture Strategies section, the gestures are captured using camera vision, more specifically, color tracking techniques. A program written in openFrameworks[3] analyses the data obtained from the tracking points and maps this information into parameters. The variations of these parameters are translated into messages and sent–using OSC protocol–to a second software called Osculator[4], that transforms this data into midi messages. A third software, capable of receiving midi messages from other softwares called Ableton Live[5], is used to produce sound. It was chosen for its capacity to easily produce controllable sound loops with a good sound quality.

MAPPING
So far, trying different mapping alternatives has been one of the main differences between prototypes. Probably the way the gestures are mapped into sound is the most influential variable, and it profoundly impacts on the kind of interaction: it sets the rules, affordances and constraints for the experience. Although a good amount of alternatives have been explored, I cannot make a definitive commitment to a certain mapping at this point.
However, one of the most interesting experiences took place with prototype # 5, the Pendulum, that incorporates–for the first time–the use of physical objects operated by participants. I plan to further explore the use of objects that inherently imply physical challenges of operation.

FEEDBACK
To balance visual and sonic feedback is challenging. Visual feedback is very useful for understanding how the system works, but an excessive presence can diminish the level of attention between players. When the visual feedback is strong or extremely seductive, people tend to give most of their attention to it, diminishing awareness about the musical and social experience[6]. Taking this into account, my project will use primarily sound feedback, making use of the visual aspects as a complement for the sound experience.
Having a system that is highly responsive to the participants’ gestures will be extremely important. As novice players are the main target, the musical range will be highly constrained, but this can be balanced by providing different control modalities, like controlling low-pass filters. A good balance between constraints and controls will allow to have expressive and very reactive feedback to users’ gestures.

THE MISSING VARIABLE
As the model I’ve been using for this section aims specifically for DMI’s, when applied to experiential design, it lacks of a last step: choosing a proper context. I feel that as my goal is not to create an instrument, but more an interactive sound space, the context where the experience takes place is as relevant as the interface or the sound itself. When I performed tests with Trinidad, I realized that this was an extremely important component to provide the appropriate conditions for the experience. I can see now that the interface is not to be designed alone, but as a tandem with the space that contains it.
Some questions arise here: How large needs the space to be? What are the lighting conditions? What are the sound conditions? Will the space take an active or a passive role in the interaction (i.e. the light conditions can be controlled by the participants, as a complementary reaction to the inputs).
In conclusion, some of the decisions I’ve made during this stage are: Will continue exploring the use of physical controllers, as the interaction seems to be more appealing when they are present. Mapping, while reducing the range of musical possibilities (melodies), must augment the sense of control and expressiveness. The context must be adequate for the scale (number of participants), and encourage intimate interactions.

___________________

[1] Miranda, Eduardo R. and Wanderley, Marcelo M. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Wisconsin: A-R Editions, 2006.
[2] Miranda, Eduardo R. and Wanderley, Marcelo M. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Wisconsin: A-R Editions, 2006.
[3] http://www.openframeworks.cc/
[4] http://www.osculator.net/
[5] http://www.ableton.com/live-8
[6] Blaine, T. and Perkis, T.  The Jam-O-Drum Interactive Music System: A Study in Interaction Design. In Proceedings of Symposium on Designing Interactive Systems. 2000, 165-173.

On the previous section I identified three types of roles for interfaces: intercommunication, mediation and facilitation. Intercommunication role will not be explored in my project, that said condition is not adequate for the general aim of this thesis, where communication between participants is priority. Consequently, the ground between mediation and facilitation is the direction I will pursue for designing the system.

I also have discussed how the experience should be in terms of interaction. However,a remaining question is: What is this for? There is a wide range of expectations towards the musical experience that this project could address. Ranging from entertainment to musical performance, the wide spectrum of expectations deeply influence on the characteristics of the interface. The final product could then range from a toy–in the case of entertainment–to a musical instrument in the case of music performance. Although at this point I cannot say exactly what my project will be, I am certain about what I don’t want: is not going to be a toy, and is not going to be an instrument.

Since this project is targeting group experiences, the internal social relationships between participants need special attention. The question here is: Who will participants collaborate with? It is completely different if they collaborate with their couples, with their bosses or with a complete strangers. Different levels of intimacy between participants require different contexts, features and roles for the interface. Trying to understand where my project should fall in this wide social spectrum, I created a bi-dimensional graph plotting level of intimacy and context. These two conditions influence on each other, so it is important to find the most adequate combination of both. For instance, an interface meant to be used in the living-room would be completely different from one that is set up in a public plaza. The decision to work in the public or in the private sphere is relevant (and difficult!) so I’m not completely decided on this yet, however making this graph helps me to narrow the scope of possibilities by choosing an area.

Making research on social gatherings in diverse contexts, one of the social activities that caught my attention are rituals of Native Americans around drum circles. In this rituals, many of the things I’m interested in are present: collaboration around a sound context, communication between participants, a sense of belonging to a community with common goals. The interaction is interesting too, a very simple gesture–hitting the drum in a constant pace–creates something really powerful and beautiful: humans singing to the spiritual world. Here, no musical expertise is really needed and the most important thing is the social experience.

Another example are community music groups. There are some institutions that create activities around music for members of a given community, where people with different levels of musical expertise can participate in collective experiences. One of the most noticeable institutions is the Developmental Community Music[1], that organize experiences for different communities such as neighborhoods, office spaces, schools and any kind of community interested in participating. The goal of DCM is to facilitate musical group experiences with the aim of developing self-improvement, communicational skills, teamwork skills and spiritual fulfillment. The power of sound.

Getting into spaces, I started looking to places that create intimacy and behavioral change, trying to understand what are the conditions to allow this. The karaoke bars and booths–so popular in Japan–are perfect examples of spaces that meet those conditions. Karaoke spaces are usually used by groups of friends looking for an entertaining experience. Within this space, people behave in a completely different way as they would do outside; entering a karaoke booth implies some behavioral rules responding to special rules that exist only inside the room. As described By Salen and Zimmerman[2], a Magic Circle has its own set of rules, and these rules create a parallel space of reality:

“…Within the magic circle, special meanings accrue and cluster around objects and behaviors. In effect, a new reality is created, defined by the rules of the game and inhabited by its players.”

So, what conditions make spaces become a place for intimacy and behavioral change? It seems that the basic variables are: light conditions, area of space, sound conditions, space distribution and services. In a karaoke space for instance we have dim lights, a closed space (proportional to the number of participants), a good sound quality, a right distribution of furniture that encourages the activity, and food and drinks at hand. The combination of all these provide the necessary environment that karaoke requires.

In a similar fashion, Photo-Booths are also spaces with their own behavioral rules. It is interesting that although all what these booths do is nothing more than taking a set of pictures, using one is usually a fun experience and it produces a profound behavioral change on  the participants. There is something very unique about entering a very tight box, making people share their personal space with others. Again, the photo booth conditions: dim lights, very tight space, small seats, and a curtain that separates the booth-world from the ordinary. In this case, just like the karaoke example, the interface sets the rules and drives the behavior; the time between each picture, the amount of pictures, the view area of the camera, are things that encourage people behave in a certain way for a certain period of time.

I want to point out something I saw one night at a local bar in Brooklyn: a group of ten to twelve people were playing traditional American music. They were not officialy a band, they were  people getting together to make music. Following the tradition of the musical style, violins, mandolins, double basses, guitars, violas, and the essential banjo were part of the ensemble. People were displaced in a circular shape, distributing along various layers of distance from the center. Closer to the center, the more experienced musicians played their instruments, conducting the performance. Moving away from the center, a second layer of intermediate musicians followed the ones in the center, assuming a supporting role to the ensemble.

On a third layer, behind the intermediate layer, people that either were learning how to play the instrument or were getting to know the style also assume supporting roles, sometimes just observing and listening to the music. I had the chance to talk to them, and they explained that they were not a band. They were people meeting every two weeks to play together, “The group is different every time we meet depending on who shows up, we always have new people that we never met before” said one of the participants. The most experienced musicians have their own bands, they do this because it’s fun and for meeting and sharing with people with similar interests. “Anyone with an instrument and can participate and learn.”

The beauty of these meetings is that novices and experts come together, and feel comfortable in the same  experience. Even though is clear that there are different levels of participation according to the different skill levels, socially everyone is equal, and everyone’s invited. I witnessed that myself: when they ended playing a couple of songs, one of the guys in the center introduced himself with the one sitting on his side, after talking a while, they exchanged their guitars, this seemed to me a welcoming ritual for a stranger.

The findings from this section were very enlightening, now I have a better understanding about the conditions of spaces that encourage collaborative interactions. The idea of a booth remains in my mind, and definitely some of their characteristics will be incorporated into my project.

_____________________

[1] http://playsinglaugh.com
[2] Salen, Katie and Zimmerman, Eric. Rules of play: Game Design Fundamentals. London: The MIT Press, 2004.

Based on the methodology described in the previous section, I’m currently focused on defining the characteristics of the experience. Putting it in simple words: What do I want people to experience with my project? What kind of interactions will the system require? How can interactional requirements shape the social experience?

Doing research on the HCI aspects of these experiences, will allow me to better understand in what ways the interaction between participants and the system can influence on the overall experience, setting different moods. Understanding this better will help me to later take decisions about the target users, interface characteristics and context requirements. By doing research on different projects that relate–directly or indirectly–to my own, I can get a better idea of all the variables that need attention, what are the common points, and goals. In short, is evaluating others’ projects so I can see the strengths and pitfalls that are common in this field and apply those findings into my own project.

Since the 1950‘s there has been an increasing interest on exploring collaborative sound experiences using non-traditional instruments, examples as John Cage’s Symphony of 12 radios [1] were trying to push the traditional boundaries of what was known as collective performances. Since the 70’s, when computational systems became more accessible, artists and scientists have been able to let go their imagination seeking for new ways of interacting with new musical devices and systems. In the last ten years, there has been strong interest among the academic and design communities on this subject. The existence of NIME, the annual conference on New Interfaces for Musical Expression is a proof of that.

Today, the current state of technology and the influence of DIY culture, are allowing more people to explore new approaches to collaborative musical expression, and new projects emerge every time more rapidly. So, my path is not unique, I’m obviously not the first one to investigate this issues nor the only one with the same objectives and passion about this. The good thing is that there’s a community and I’m not alone. I here quote Scott Pobiner, my thesis Professor: “stand in the shoulder of giants”–he said. The challenge is then to find how this thesis contributes to the field. How will it be different from other projects? How will it be better? These are some of the questions that I need to address in order to make this thesis unique.

A study presented for NIME 20102 outlines a broad classification of experiences of ISM (Interactional Sound and Music) by breaking them out into three dimensions: By situation: co-located or remote collaboration, by focus: task-oriented or creative engagement (improvisation), and by immediacy: real-time sound generation or sound-editing. Although the categorization presented here is very useful, I see it with critical eyes because it posits the focus as a binary framework. I believe that having an experience that demands the completion of tasks does not exclude creative engagement, and should be space for both approaches to coexist. However, this statement requires further investigation, developing prototypes will be a way of submitting this classification to test, to see how this relation works. I’m really interested in the following question: can a task-oriented interface allow space for improvisation? My gut feeling says it should, but I won’t be sure until I research more on this point.

Based on my research on different projects, I think that the role that the interface assumes can be classified in three categories: intercommunication, mediation and facilitation. The role of intercommunication implies that most of the communication between participants is done through the interface. For example, in the case of console games such as the Microsoft’s Dance Central designed for the Kinect, the interaction encourages participants to focus their attention to the screen, and all the messages sent between each other are mediated by a digital representation the players. Participants and their interactions become part  of the avatar world, making the physical world less important than the virtual.

The second type of role is mediation, where the interface influences and shapes–up to a certain extent– the communication between participants. An example for this would be the Jam-O-Drum [3], where the interface sets the rules for communication but participants can still generate direct interactions between each other. The third type is facilitation, where the interface is in the periphery of attention, and plays the role of enhancing or catalyzing direct communication. An example for this case could be Lucas Werthein’s installation Samba Surdo [4], where sound is the only feedback and the internal mechanics of the interface are hidden from the participants’ attention.

To conclude, some of the main questions that arise from this section are: Can the experience be rich in communicational terms when the interface is mediating the interaction? Can improvisation (creative engagement) be fostered with a task-oriented experience? How can participants’ behaviors can be influenced or modified by the interface?

_____________________

[1] http://www.guardian.co.uk/music/2009/aug/06/john-cage-symphony-for-radios
[2] Bryan-Kinns, N., Fencott, R., Metatla, O., Nabavian, S., & Sheridan, J. (2010). Interactional Sound and Music: Listening to CSCW, Sonification, and Sound Art (Demo/Poster). In Proceedings of NIME 2010, Sydney, Australia.
[3] Blaine, Tina, and Tim Perkis. 2000. The Jam-O-Drum interactive music system: a study in interaction design. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, 165–173. DIS ’00. New York, NY, USA: ACM.
[4] http://www.lucaswerthein.com/?p=227

I read some texts from Tina Blaine at the beginning of this process, most of them analyze different projects from various authors, but she also has designed a prolific amount of musical interfaces. One of the most relevant for my investigation is the Jam-O-Drum, an interactive tabletop interface that can be considered as a predecessor of the Reactable. Blaine along with her collaborator Tim Perkins wrote a paper for DIS 2000 [1], where they explain in detail  the process of designing the system: they divided the study into two initial phases. On the first, they performed tests to define the interactional and graphical aspects separately. In Phase Two, the results from Phase one were incorporated into an integrated model, testing again the results.

My plan is to incorporate a similar iterative methodology, testing specific aspects of interaction through prototypes, and by the end of the semester integrate all the individual findings into a final, integrated proposal.

Prototyping will be an important aspect for my research: through user’s reactions to these prototypes, I will able to see what are the most suitable solutions. Prototypes will become then the spinal cord, the main road that will lead to answers and discoveries, and eliminate or confirm assumptions.

I’m framing my research into three main areas: The experience, the interface and the context. My intuition says that defining the experience first will lead naturally into taking decisions about the users, interface and space that supports the experience.

So inspired by Blaine’s model, the sequence should be divided into three steps:

• Define what kind of experience
• Define users
• Design an interface to facilitate the defined experience for the defined users.

Along the multiple examples I’ve studied doing research, I’ve seen a trending challenge when designing interactive sound systems for novices: balancing the level between expressivity and complexity. It is of the most importance, as this balance determines the entry-level. Let’s take an example from traditional instruments: a xylophone requires a simpler gesture than a violin, but so are the expressive possibilities of creating different timbres. Conversely, mastering a violin is harder than mastering a xylophone, but the expressive range of the instrument is wider. This is a trade-off I need to be address carefully, because my goal is to provide a very low entry-level, but to make the interface as expressive as possible.

Probably the entry level is one of the most important things I will have to put attention to.  A different issue that also deals with this idea is to decide if this will be exclusively designed for novices. What I think it would be the perfect situation is a space where novices and experts can share the same experience, having similar amounts of joy. Maintaining the state of flow for both levels of expertise is a difficult task, and I feel that the answer is focusing more on the experience rather than in the musical aspects.

____________________

[1] Blaine, Tina, and Tim Perkis. 2000. The Jam-O-Drum interactive music system: a study in interaction design. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques, 165–173. DIS  ’00. New York, NY, USA: ACM.

In this section, I identify the different disciplines that will be relevant for my research. Not necessarily looking for answers within these disciplines at this stage, I’m focusing on visualizing the scope of my research. Although this list will definitely be growing over time, this is where I’m starting.

On the humanistic side, psychology will help me understand the social dynamics between participants, thus is very important for this research. There is a vast body of research around HCI (Human Computer interaction), a subset of psychology. This discipline is relevant because it deals with how humans interact with artificial systems, and what are the psychological aspects within that interaction. CSCW (Computer Supported Cooperative Work), an even more specific area of HCI, studies how people interact collaboratively through computer-mediated systems. CSCW addresses the concerns and rules for designing a system to generate the desired behaviors.

On the design realm, looking into game design, software and musical instruments design will help understand the variables for designing a system that deals, at the same time, with all of their concerns. On the technical side, spatial design and sound design will play their role in determining the kind of space and sound conditions to support the experience. As I could realize when testing Trinidad, the context is very important as it influences on people’s predisposition to behavioral changes. Just imagine a dance club with bright, fluorescent white lights, it simply wouldn’t work.

At this stage of research, I am focusing more on the humanistic side. It makes sense to me that a good first step is understanding how people operate in experiences that require collaboration. What are the conditions needed to generate engagement with an activity? What influences the communication between participants? How does flow works?

In The Psychology of Happiness [1], Michael Argyle refers to a chart from the American Psychological Association, that plots the classification of moods. It maps all the adjectives associated to moods into a positive and negative scale, crossing the dimensions of pleasant-unpleasant and arousal-sleep. This maps broadens the scope of what I was considering a “fun” experience, breaking it down into more specific states. There are different kinds of pleasurable experiences, we sometimes experience as much pleasure when staring at a sunset in a calm beach as we do when playing Black Jack at the casino. Same feeling, but shaped in an entirely different way. This brings me to the question: What kind of pleasurable experience I’m looking for? Is it going to be something that makes you feel relaxed or something that makes you feel euphoric? Both sides are interesting, but it is important to notice that aiming to one side or the other changes everything. I don’t have the answer at this point yet, but having this in mind is helpful to define an initial area to explore through prototyping.

I also did further research about the state of flow. According to Csikszentmihalyi [2], this state can be visualized as a channel that arises when skills and challenges are properly balanced.

“ In our studies, we found that every flow activity, whether it involved competition, chance, or any other dimension of experience, had this in common: It provided a sense of discovery, a creative feeling of transporting the person into a new reality. It pushed the person to higher levels of performance, and led to previously undreamed-of states of consciousness. In short, it transformed the self by making it more complex. In this growth of the self lies the key to flow activities.” (Csikszentmihalyi 1990, 72-77)

Taking these ideas from different studies, mostly from Csikszentmihalyi, it can be said that there are four conditions for systems to facilitate flow:

• Contain rules that require the learning of skills
• Set up goals
• Provide feedback
• Make control possible

___________________

[1] Argyle, Michael. The Psychology of Happiness. New York: Methuen&Co.,1987.
[2] Csikszentmihalyi, Mihaly. Flow: The Psychology of Optimal Experience. New York: Harper & Row, 1990.