Vox Pluralis
Overview
[A second version of the glove using Eeonyx fabric and improved circuits is planned for this summer.]
Vox Pluralis is a musical performance system for augmented singing. The singer’s natural gestures and vocal prosody are used to multiply, transform and extend the voice. By opening a hand, the voice is opened into choral polyphony, by closing it, it is pulled back into singularity. By cocking back the wrist, the voice is stretched and frozen in time. This is accomplished with a glove that senses the positions of the fingers and wrist coupled with software extracting prosodic features from the voice. The glove transforms intuitive hand gestures into correspondingly intuitive, yet fantastical sonic manipulations.
Project Description
The Vox Pluralis glove is an computer input device for controlling sound processing applied to the voice of the user. It is designed to be as transparent as possible and to take advantage of intuitive hand gestures. The device consists of a single piece of stretch conductive lycra cut into long strips glued directly to the top of the hand and extending around the wrist to the palm. As the fingers and wrist bend, the material is stretched. This stretch is measured for each finger and the wrist and used to parametrize various sound processing algorithms. Thus, the device is not so much a glove as a sensate second skin applied to the hand.
The device is designed to track intuitive gestures and to map these gestures to vocal effects that are perceptually associated with the gestures. The primary gesture is that of opening the hand. This gesture activates a chorus effect, multiplying one voice into many. The gesture can be thought of as one of ‘opening’ the voice into a chorus. The act of spreading the fingers can be linked visually to the process of ‘splitting’ one singular voice into a polyphony. Following this metaphor, the individual fingers control parameters of separate groups of ‘voices’ in the chorus algorithm, allowing the singer to control the virtual chorus as if each finger were a group of singers. Likewise, by bending the wrist back, the singer activates a phase-vocoder based time stretching algorithm hacked to work in realtime. This gesture resembles the ubiquitous “stop” gesture, a natural choice for time-stretching, which can be seen as slowing or stopping time.
The device is designed with the biomimetic principle of ‘instrinsic functionality’ in mind. In engineering, there is a separation drawn between structural elements (a garment, the frame of a building) and functional elements, or in this particular case, information processing devices. Just as all parts of a living body serve both structural and functional roles (for instance bones serve both to support a body and to manufacture blood) so does the glove. The only structure that it can be said to have also constitutes the sensor. This is a property that has come to be called “intrinsic sensing”: the material has sensing properties by the very nature of its structure. There are not sensors “in” or “on” the ‘device’, it is itself a sensor.
Going further, we can suggest that the glove can be considered nearly part of the human body. While the device is for now fabric glued to the outside of the skin, it suggests future devices which may be painted or sprayed on, tattooed, or embedded directly in the skin. Typical wearable sensors are built into ad hoc garments or strap-on assemblies designed to be removed. What if wearable sensors were instead ‘applied’ or ‘implanted’ into the body, designed to last for long periods of time and serve manifold purposes?
Technical
The glove consists of a single piece of Stretch Conductive Fabric, a silver-doped lycra from Less EMF, cut into a pattern that follows and loops around the tips of the fingers and extends into the palm. The the various ‘fingers’ of the textile are united at the wrist. Conductive lycra can be used as a resistive stretch sensor by connecting electrodes to two ends and measuring the resistance. The common connection at the wrist serves as the grounding electrode for all of the ‘sensors’ of the gloves while at the end of each ‘finger’ is attached an electrode carrying 5 volts. The electrodes lead to a circuit that performs a resistance to voltage conversion (here, we tried a voltage divider and a Wheatstone bridge). The derived signal can then be conditioned and converted into a digital bitstream to drive audio software.
Obtaining a reliable signal from the particular conductive lycra that we used proved difficult. The material provides a resistance range of about 120Ω, resulting in a voltage range of a few decivolts. Because our analog to digital converters expected a voltage varying between 0 and 5 volts, this small change needed to be amplified. The combination of a Wheatstone bridge and a precision instrumentation amplifier (which is designed to amplify and scale small changes in voltage) was used for this task. Additionally, the signal coming from a fabric sensor was extremely noisy, so a 3-pole active lowpass filter was used to filter frequencies above 15hz.
Signal Chain
1. AC→DC-converter→LM7805 5V regulator (‘wall’ voltage to 5V)
2. Stretch Conductive Fabric from Less EMF (resistive stretch sensor)
3. Wheatstone bridge (resistance→differential voltage)
4. AD623 instrumentation amplifier (amplify decivolt range to 5-volt range)
5. 3-pole active Butterworth filter with LM741 or OP27 op amp (filter frequencies above 15hz)
6. A/D conversion with Arduino (AVR ATmega168) [this will be replaced by the Teensy board which offers better USB serial support]
7. USB→Macbook Pro
8. Max serial object + SensorBox decoder
9. Smoothing, scaling, and mapping in Max using [slide], [function], and a custom auto-scaler [minmax].
10. Mapping to pitch-tracking chorus from IRCAM and phase-vocoder- – -part-i/ based time-stretching effects in Max/MSP
Pictures and Diagrams
Breadboard with lowpass filters, voltage dividers, and Arduino (A/D).
3-pole active lowpass filter circuit.
Sensor data filtering, scaling, and mapping in Max.
IRCAM’s [psychoirtrist] pitch-tracking chorus module (part of IMTR-Transformation bundle)
realtime phase-vocoder based time-stretcher implemented in pure Max/MSP
Video
Finn Upham and Morgan Sutherland performing with the Vox Pluralis system at McGill using EMG biosensors on the hands rather than the Vox Pluralis glove documented here.
EMG biosensors measure whether the hands are open or closed. EMG replaced the Vox Pluralis glove during the presentation of the sound processing features of the system for MUMT307 at McGill University.
Performance setup at McGill. EMG sensor processing was performed in Pure Data on a Macbook pro. Sensor data was sent to a Mac Pro which performed all DSP (audio feature extraction and effects processing). An RME Fireface firewire audio interface was used for A/D and D/A conversion.
Inspiration
- Pamela Z, Voci + BodySynth
- Michael Waisvisz, The Hands
- Elena Jessop, Vocal Augmentation and Manipulation Prosthesis
- Laetitia Sonami, Lady’s Glove
- Bart Hess, Future Fur
- CyberGlove Systems, CyberGlove II
References
- Sha Xin Wei, et al., “Demonstrations of Expressive Softwear and Ambient Media Topological Media Lab”
- Dvorak, Moving Wearbles into the Mainstream: Taming the Borg_
- Perner-Wilson, “HOW TO GET WHAT YOU WANT”
- Horowitz and Hill, The Art of Electronics
- Scherz, Practical Electronics for Inventors
- Dudas and Lippe, “The Phase Vocoder”
- Dodge and Jerse, Computer Music: Synthesis, Composition, and Performance
Credits
Glove completed for DART339, “Second-Skin and Softwear” with Joey Berzowska at Concordia University. Vox Pluralis vocal processing completed for MUMT307, Music & Audio Computing II with Gary Scavone at McGill University.
Thanks to Joey Berzowska, Gary Scavone, Sha Xin Wei and the Topological Media Lab, Elio Bidinost, and Navid Navab for invaluable guidance.
Thanks to Laura Boyd-Clowes for making the glove with me and for being wonderful.
Thanks to Finn Upham for collaborating with me on the vocal processing system. Additional documentation can be found at her blog (note: she chose a different name for the project).