Media Medium. Two of my classes, CART363 “The Advanced Languages of Programming” and CART454D “Enchantment, Matter, and Topological Media” require a final techno-art project. When I began brainstorming, immediately I wanted to extend what I was doing at the TML in January and February: that is, a dance film with camera-tracking controlled sound and Jitter visuals projecting on a semi-translucent screen stretched over a pair of dancers. Part of the idea was for the ‘media’ (visuals) to form a ‘medium’ (as in air, water) in which the dancers would be immersed. The sound was then meant to reflect the notion of ‘touch’ (a crude example would be the bzzt that signifies a hit in fencing). While the visuals came together to adequate effect, the sound served as more of a soundtrack than anything. Having spent a few weeks experimenting with camera tracking techniques and brainstorming mappings, along with Tim Sutton who did the sound, the project felt unfinished as it came time to film what we had for the EMPAC submission deadline.
So, I had this format in mind: four speakers facing inward with camera-controlled sound. The question, ‘how can I map the movement of players in a small space to sound tangibly and meaningfully‘. On the bus home a few nights ago, a concept and a technology synthesized: sound as a medium + the State Engine.
[What is the State Engine? From Yon Visell’s project page at Zero-th:
The software provides a general-purpose choreography system for responsive design, based on a continuous dynamics within a topologically general state space. This approach allows designers of responsive media to specify complex media dynamics in a way that is idomatically concise and expressive, encoding the delicate set of relations between the interactions of the users of such systems and the character of the range of responses that are elicited.
(I will describe a basic configuration.) The State Engine consists of a ‘token’ (particle) in a topological space. The token is given coordinates, distances from the vertices of the space (simplex). The token is pulled around, by ‘forces’ ‘exerted’ by the vertices, in a smooth and continuous manner. It’s behavior is designed to mimic physical dynamics, either those of a spring or gravity. You can then pump in sensor data, define generally how the token should respond to that data, and you’re given in return a vector of the coordinates of the token. What’s exciting about this system is how physical and life-like the output is. It yields ideal data with which to drive a ‘media medium’.]
And that’s exactly what I plan to do. I will place four speakers in a square facing inward. Each one will play a sine-tone from an independent oscillator. The frequency of each oscillator will then be mapped to a vertex of a tetrahedron in the State Engine. The range of frequency modulation will be quite low, the idea being that as the token navigates the space, the oscillators will subtly detune, introducing frequency beating. To obtain data to disturb the token, I will use camera motion analysis, specifically a Jitter patch that sees only moving objects and leaves a decaying trace after things have calmed down. The Engine will be configured so that the token will snap back to the center of the space in the absence of motion.
Thus, a user in the space (framed by the speakers and the camera’s scope) will be able to throw the oscillators out of tune by moving. He will have the sense of disturbing a water-like, or elastic medium. A sharp movement will introduce a chorus of high-frequency beating patterns. As the ‘trace’ of their movement fades and the token returns to the center of the tetrahedron, the oscillators will reach equilibrium. Additional parameters to determine in what manner the token will move are yet to be defined, but I expect to work this out as I’m building.
I’m thinking of this as a ‘phenomenological experiment’. My goal will be to make the media as “thick” and “tangible” as possible. Once the system is constructed, I expect I will be spending considerable time ‘tuning’ it. Code-wise, it’s a fairly simple project (snapping together pre-written chunks), but the State Engine is fussy and offers myriad configurations. I suspect that a range of distinct behaviors will be possible. Also, it’s likely that a bit of random, ‘irrelevant’ / peripheral data will make the behavior more interesting. This I expect to discover in the course of experimenting, and that’s the point!
