Multimodal Music Stand


The Multimodal Music Stand (MMMS) incorporates four Theremin-type electric field sensors, a video camera, and a microphone as input sources for the performance of electroacoustic and computer music compositions. It is designed for interactive music performances using computer vision techniques (such as tracking the position of an instrument), audio analysis / transformation of the performer's playing, and enhanced expressivity through bodily gestures and movements that are tracked via the four electric field sensors (also known as capacitive sensing). The MMMS gives the performer added dimensions of musical control in order to influence the interactive aspects of a composition without having to modify their instrument or tether themselves with wires.

While the electric field sensing technique is based on the front end of the original Theremin circuit topology, it is done entirely in the digital domain using the CREATE USB Interface (CUI), and the ThereminVision-II. The interface to the computer is accomplished by detecting timing signals using custom firmware for the CUI, and sending the four proximity values to the host as continuous ranges (16-bits each) at an update rate of 100Hz. Using four channels of capacitive sensing makes it possible for the MMMS to provide full 3-dimensional input - the collective E-fields of all four antennas are used to determine the Z-axis gestural input. In this way, different sensing modalities are used to reinforce one another where each has advantages and disadvantages (e.g. capacitive sensing is better at Z-axis tracking than computer vision).

Below is a video of Lance Putnam demonstrating the electric field sensing capabilities during initial prototype testing of the Multi-Modal Music Stand.

  • MMMSdemoSmall.mov - 10MB mpeg4, recent version of quicktime required.
  • MMMSdemo.mov - 64MB mpeg4, higher quality version of video above.

    The first composition for the Multimodal Music Stand is "timeandagain" by JoAnn Kuchera-Morin, here an excerpt performed by Jill Felber:

  • timeandagainRehearsal.mov - 20MB mpeg4, recent version of quicktime required.
  • timeandagainPerformance.mp3 - 13MB mp3, 256kbps




    Publications

    Interactive Musical Performance Using the Multimodal Music Stand
    Presented at the NSF Integrative Graduate Education and Research Traineeship annual meeting, (Santa Barbara, California, 26 January, 2007).

    The Multimodal Music Stand
    Proceedings of the New Interfaces for Musical Expression conference, New York, New York 6-9 June 2007.