Interactive Electronics
Don’t just sit there.
Movement and physical action have always been essential elements of live performance. Since I began to work with electronics in music, I had imagined somehow translating movement into sound and image in the digital domain, but the possibility seemed elusive, and practically remote. In recent years, with the widespread availability of inexpensive yet sophisticated new technologies, the capacity of interactive electronics to capture physical actions and reflect them in sound and image has never been greater.
The circuits and devices described below–a new family of instruments–chart a chronological map of my work, over the past 20 years or so, with a succession of experiments, designs, and refinements of a series of sensor-based MIDI (Musical Instrument Digital Interface) controllers and instruments.
LaserHarp Instrument
Connected to STM-3 Controller
STM: Sensors to MIDI
STM (Sensors-to-MIDI) systems employ sensors and microcontrollers to digitally sample various aspects of a performer’s position, orientation and movement, and then use that information for the live control of sound and image in a computer-mediated environment. Sensors can be either worn or carried by a performer, or placed around the performance space in order to turn the space itself into an instrument responsive to the motions of the occupants. The signals from sensors connected to a STM Controller, such as those shown below, are transmitted as MIDI messages to a host laptop or desktop computer running SensorPlay, an application that turns the signals into actions such as playing a musical note, moving a sound in the stereo field, or manipulating a video projection.
Early STM System
4 Hooded PhotoCells
16 Buttons
STM-2 Series (1998)
The STM-2 series controllers were the first designed as custom manufactured printed circuit boards (PCBs), Based on the Basic Stamp processor from Parallax, the STM-2 series was the first to incorporate the basic STM design criteria.
Details
STM-2
- Parallax Basic Stamp 2 processor (hybrid device with onboard EEPROM and voltage regulation)
- Programmed in BASIC
- 16 switch inputs via RJ45 connector
- 4 analog inputs via RJ11 connector to Linear Technology 2 x LTC1298 2-channel ADC (SPI)
- MIDI out interface
- Programming via DB9 serial connector
STM-2P
Includes the features of STM-2 plus:
- Basic Stamp 2SX processor
- 8 program banks (selectable by rotary switch)
- 2 x 4 analog inputs via dual RJ11 connectors to Texas Instruments TLC2543 8-channel ADC (SPI)
STM-3 Series (2001)
While providing the same basic functionality as STM-2, the STM-3 replaces the hybrid Basic Stamp processor with individual PCB components, including a fast RISC processor, to deliver high performance sampling of sensor inputs.
Details
- Scenix SX28AC 8-bit RISC Processor @50MHz, 2048 12-bit words of program memory, 136 bytes of RAM
- Programmed entirely in assembler, including math libraries
- 16 switch inputs via RJ45 connector
- 8 analog inputs via RJ11 connector to Texas Instruments TLC2543 8-channel, 12-bit ADC (SPI)
- MIDI in and out interfaces
- Programming via 4-pin interface to Vcc/Vdd and resonator/clock pins
STM-4 Series (2005)
The STM-4 expands on the STM-3 design to create a slightly larger and more open physical format, with full solder mask and silkscreen, a 50% increase in clock speed, and the ability to network processors via an additional RJ45 connector.
Details
- Scenix SX28AC 8-bit RISC Processor @75MHz (crystal), 2048 12-bit words of program memory, 136 bytes of RAM
- Programmed entirely in assembler, including math libraries
- 16 switch inputs via RJ45 connector
- 8 analog inputs via RJ11 connector to Texas Instruments TLC2543 8-channel, 12-bit ADC (SPI)
- MIDI in and out interfaces
- Programming via 4-pin interface to Vcc/Vdd and resonator/clock pins
- Processor networking via RJ45 connection
STM-5: Invisible Beams
STM-5 devices were the first to contain both the sensors and the controller in the same device, eliminating the need for flat cables and their connectors, and replacing the general purpose design of previous STM controllers with one adapted to the specific behaviors and attributes of a single class of sensor, such as the distance sensors below, which are based on reflected infrared light and ultrasound waves. A signature aspect of the sensors in this series is that they are considered active sensors, meaning that, rather than passively monitoring their environment, they take an active role by sending a signal into the environment and learning about that environment through analysis of the signal’s reflection.
STM-5 Mira
4 Infrared Distance Sensors
Mira (2004/2018)
Mira infrared sensors operate in a non-visible light range, solving a problem that affects visible-light photocells: sensitivity to ambient light conditions, making it difficult to operate reliably in live performance conditions.
Details
- Parallax Propellor 8 Core RISC Processor @80 MHz, each core running in its own block 512 32-bit words, all interchangeable as registers, code, and data. 32 kB of shared RAM
- Programmed in mix of assembler and Parallax SPIN scripting. Individual cores devoted to MIDI In, MIDI Out, sensor data collection, and other tasks
- 4 x Sharp GP2Y0A60SZLF Infrared Distance Sensor, 10-150cm, vertically mounted
- Microchip MC3204 4-channel, 12-bit ADC (SPI)
- MIDI in and out interfaces
- Programming via 4-pin header interface
Ute (2007/2018)
The ultrasonic sensors used by Mira are also very popular in robotics, usually used to prevent mobile devices from running into objects in their path. In live performance, they are quite a versatile source of media control.
Details
- Arduino Pro Mini hybrid device with ATmega328 8-bit processor, onboard regulation, EEPROM, and bootload support, 32 kB Flash, 2 kB RAM, 5V, 16 MHz
- Programmed in C++
- 2 x Devantech SRF04 Ultrasonic (40 kHz) Distance Sensors, 0-10 ft, horizontally mounted
- MIDI out interface
- Programming via miniUSB connector
STM-6 Mimn II
9 Degree of Freedom (9DOF)
Wireless Motion Sensor
STM-6: Wireless and Wearable
STM-6 devices realize a long-standing ambition to free the performer from wires and other physical connections, employing wireless signals to transmit sensor data. Recent advances in the miniaturization of sensors and transmitters, coupled with greatly reduced cost, have finally brought this technology well within reach of the artistic community. Mimn I and II gather sensor data — accelerometer, gyroscope (and magnetometer in Mimn II) — in 3 dimensions and transmit the information, over Bluetooth and WiFi, to a host computer for further processing. Unlike their STM predecessors, they transmit raw sensor data rather than MIDI messages, which may represent a condensation of many sample points, and which could be lost in transmission dropouts (in real-time applications, messages are generally not retransmitted on timeout failures).
Mimn I (2013)
Mimn I employs a MEMS-based accelerometer and gyroscope to monitor motion and orientation. Mimn is worn on the back of the hand, to take advantage of its relatively greater range of motion compared to other body parts.
Details
- Arduino Pro Mini hybrid device with ATmega328 8-bit processor, onboard regulation, EEPROM, and bootload support, 32 kB Flash, 2 kB RAM, 3.3V, 8 MHz
- Programmed in C++
- Analog Devices ADXL345 3-axis accelerometer, 13-bit, +/- 16g range (I2C)
- InvenSense ITG3200 3-axis gyroscope (I2C)
- Roving Networks RN-42 Bluetooth Module
- Programming 6-pin header to Pro Mini using
- Powered by LIR2450 rechargeable 3.6V
Mimn II (2016)
Mimn II represents a large step up from Mimn I, in both speed and capability, providing additional sensors, a much faster processor, longer battery life, and the improved reliability of WiFi communication.
Details
- Espressif ESP8266 WiFi SOC 32-bit RISC processor, 80 MHz, 32 kB program RAM, 80 kB user data RAM
- Programmed in C++
- ST Microelectronics LSM9DS1 3-axis accelerometer, gyroscope, magnetometer (I2C)
- Programming 6-pin header
- Powered by 3.6V 800 mAh rechargeable lithium ion battery pack (onboard charger via USB mini connector)
