Patchwork: Multi-User Network Control of a MassiveModular Synthesizer

Brian MaytonMIT Media Lab75 Amherst St.

Cambridge, MA 02139bmayton@media.mit.edu

Gershon DublonMIT Media Lab75 Amherst St.

Cambridge, MA 02139gershon@media.mit.edu

Nicholas JoliatMIT Media Lab75 Amherst St.

Cambridge, MA 02139njoliat@media.mit.edu

Joseph A. ParadisoMIT Media Lab75 Amherst St.

Cambridge, MA 02139joep@media.mit.edu

ABSTRACTWe present Patchwork, a networked synthesizer module withtightly coupled web browser and tangible interfaces. Patch-work connects to a pre-existing modular synthesizer usingthe emerging cross-platform HTML5 WebSocket standardto enable low-latency, high-bandwidth, concurrent controlof analog signals by multiple users. Online users controlphysical outputs on a custom-designed cabinet that reflectstheir activity through a combination of motorized knobsand LEDs, and streams the resultant audio. In a typicalinstallation, a composer creates a complex physical patchon the modular synth that exposes a set of analog anddigital parameters (knobs, buttons, toggles, and triggers)to the web-enabled cabinet. Both physically present andonline audiences can control those parameters, simultane-ously seeing and hearing the results of each other’s actions.By enabling collaborative interaction with a massive analogsynthesizer, Patchwork brings a broad audience closer to arare and historically important instrument. Patchwork isavailable online at http://synth.media.mit.edu.

KeywordsModular synthesizer, HTML5, tangible interface, collabora-tive musical instrument

1. INTRODUCTIONNearly every computer musician has interacted with someform of modular software synthesizer, but relatively fewhave had the opportunity to patch on a large, tangible ana-log synth. Modular synths are bulky, expensive, and exceed-ingly rare, confined for the most part to music departmentsand museums, or in the hands of a small number of enthusi-asts and composers. Yet these instruments are increasinglysought after by musicians looking for tangible controls anda classic, gritty analog sound, and continue to fascinate au-diences when displayed publicly. At the same time, it isdifficult to create installations that allow the general publicto directly manipulate or patch these instruments; a steep

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.NIME’12, May 21 – 23, 2012, University of Michigan, Ann Arbor.Copyright remains with the author(s).

Figure 1: Paradiso’s modular analog synthesizer.

learning curve and delicate controls mean that visitors canwatch and listen, but they can’t touch. We encounteredthat particular issue when we were invited to install ourmassive modular synth [5], one of the world’s largest, at alocal museum.

Built by Dr. Joseph A. Paradiso throughout the 1970sand 80s, our synth consists of five large cabinets that to-gether contain nearly 200 homemade modules, a number ofhome-built and custom-modified keyboard interfaces, and apair of substantially augmented and fully integrated Moogsynths. To date, Paradiso has been its only serious com-poser and player. In a typical installation, he assembles acomplex patch over many hours, creating an evolving andoften unpredictable, autonomous soundscape.

The synth has attracted a fair amount of public interestover its lifetime, and has recently enjoyed renewed atten-tion. Recently, we created an audio stream that allows anonline audience to listen to a stereo mix of the synth’s out-put in real-time. As an online community formed aroundthe stream, we created a Twitter feed that would provideregular listeners with news and updates, announcing newpatches and live performances. Still, despite all the inter-est from onsite and online audiences, the synth remained anon-interactive instrument.

To address these concerns, and to bring the analog syn-thesizer closer to a broad worldwide audience, we developed

LFO DEPTH NEUTRONMODULATORS

TRIGGER BASS SEQUENCE

FILTER CUTOFF

PROBABILISTICSEQUENCER

SURPRISINGBEEP

JANGLE PROBABILITY

CHAOS TRIGGER BASS SEQUENCE

ANALOG OUT DIGITAL I/O

VOICE PITCH CHORUS TRIGGER SPACETIME

JoePCONTROL

OBSERVE

12 Listeners 7 Controllers

GershonJoePNickJBrianNan-Wei

CONTROLLERS

Figure 2: The Patchwork web client: the knob high-lighted in red is being controlled by a remote user,while a toggle switch is highlighted in blue as thelocal user hovers over it.

Patchwork, a networked synth module with tightly coupledweb browser and tangible interfaces. Patchwork allows on-line and physical audiences to concurrently control motor-ized knobs, buttons, and switches to produce analog signaloutputs from a physical synthesizer module. Designed tomatch the aesthetics of Paradiso’s original synth cabinets,our new, web-enabled cabinet uses an embedded computerand custom hardware to drive the analog and digital out-puts. The cabinet also contains a custom-designed mixerand audio interface to produce a live audio stream thatis fed back to online users, who can also see each other’sactivity in the browser interface. Visitors to the physicalinstallation see the moving knobs and flashing LEDs, hearthe effects of remote users’ actions, and can use their smart-phones to simultaneously control the module.

A web server keeps track of each client, allowing users(both physical and remote) to take momentary control ofindividual knobs and buttons. To support this kind ofcross-platform, multi-user interaction, we use the emergingHTML5 WebSocket standard and the jQuery JavaScript li-brary, enabling low-latency, high-bandwidth control fromnearly any modern computing device.

1.1 Related WorkAs far back as the 1970s, composers and researchers weredeveloping networked music systems that allowed multipleplayers on separate instruments to collaborate in perfor-mance [1]. The Reactable*, for example, allows physicallypresent and remote users to collaborate on one or moretangible tabletop instruments [3, 4]. In [6], Weinberg de-velops a rigorous framework for the field, which he callsinterconnected musical networks. Patchwork extends mul-tiple virtual interfaces into the physical world with actuatedcontrols, positioning it in a small category of single instru-ments that support multiple remote players. In some ways,our system is similar to performing robots like Hoffman andWeinberg’s marimba player [2], in that it brings an existing,physical instrument onto the internet; Patchwork is uniquein that it allows remote audiences to interact directly witha one-of-a-kind instrument and its creator-composer. Ourinstrument also evolves over time, with new patches andadditions, and Patchwork adapts to those changing condi-tions.

The Patchwork module is not the first digital controllerfor an analog synth. Traditionally, Control Voltage/Gate(CV/Gate) and MIDI-to-Control-Voltage devices have al-lowed performers to bring older analog synthesizers into

Figure 3: The Patchwork module

the digital domain, but these devices are designed for single-user, local control, and don’t integrate aesthetically into thesynth or connect to the web like Patchwork does. MOTU’sVolta software plugin also allows users to drive control volt-ages through their digital audio workstation (DAW), butit too represents only one part of the Patchwork system.More than anything, its recent release is an indication thatconsumer interest in analog synthesis is still in resurgence.Patchwork capitalizes on that interest, bringing users to-gether to collaboratively control live-generated music througha shared physical resource.

2. SYSTEM DESIGNThe Patchwork system consists of 3 parts: an HTML5 in-terface for the browser, a web server, and a wooden synthcabinet containing a low-power embedded computer and acustom printed circuit board (PCB). The components aredesigned to be modular and easily configurable; for eachnew patch, the composer edits a configuration file that setsthe numbers and types of each input and output, as well asthe analog voltage ranges on the outputs. Each componentreads that file from the server and sets itself up accord-ingly. Because the physical knob positions are read by themicrocontroller in the cabinet, they can be decoupled fromthe analog output; their ranges are scaled in software tomaximize the control resolution over the range given in theconfiguration file. The system can also be switched intopatching mode, in which the web interface becomes read-only and the composer is given the full voltage range.

2.1 A Synth Interface for the WebPatchwork provides a web browser-based interface to re-mote users, shown in Figure 2. The web client is built us-ing HTML5 and the new WebSocket standard and is there-fore available via any modern browser, including for mo-bile phone and tablet users. The sockets are used for low-latency interaction, so that controlling users can achievenear-seamless control over the synth, and so that other userscan see the synth module updating in real time.

The web client is configurable such that the composerusing the modular synth can simply specify in a configura-tion file which, and how many, control components shouldbe exposed on the web interface, selecting among knobs,keyboards, buttons, switches, and triggers.

The web client interaction allows for fine-grained concur-rent control. Our server mediates between clients to assigncontrol of components like knobs and buttons, which canonly be directly controlled by one user at a time. Whena user tries to control a knob, the client requests owner-ship from the server. Once the user is explicitly handed

control of a component, other users can see the compo-nent updating in real time, and are not able to control it.When multiple users try to take control of a single compo-nent at the same time, the web client does not update theview until the server has acknowledged one user’s owner-ship; in this way we avoid giving misleading visual feedbackto the other user. Once the server has acknowledged owner-ship, the client can update the component without waitingfor more server acknowledgments, thereby maintaining low-latency UI responsiveness.

2.2 The Patchwork ServerThe Patchwork server mediates between online clients, whoconnect through their browsers using the HTML5 Web-Socket standard, and the Patchwork synth module, whichdrives the physical hardware and I/O. The server systemcan be configured in two ways: for small numbers of clients,the low-power computer in the cabinet can act as the soleserver and controller; to support large numbers of clients,an off-site middleware server acts as a buffer between thoseusers and the cabinet controller, managing the client list andassignments, and relaying control messages to the cabinetover a socket.

Our web server design is configurable and extensible, inorder to support different synthesizer configurations andback-end implementations. This means that while workingon a patch, the composer can, without editing any code,choose how the I/O maps to different behaviors (e.g. tog-gles or triggers) and front-ends (e.g. knobs or keyboardinterfaces in the browser), and how the voltages scale withcontrol parameters. Switching to a new backend (e.g. fromthe embedded server to the larger web server with a networkbackend) can be done by implementing a new backend sub-class and without editing the existing framework.

The Patchwork web server mediates between concurrentcontrolling users. To facilitate effective collaborative con-trol, we allow a small set of users to control the synth ata given time, while maintaining a queue of users who haverequested to control it. The web server maintains this infor-mation in shared data structures, and we use the Autobahnand Twisted Matrix Python web frameworks to ensure se-rial accesses of these structures. Users who request controlmay enter their names, so that non-controlling observerscan see who is controlling the instrument.

2.3 The Patchwork Synth ModuleThe Patchwork module (Figure 3) is built around a low-power embedded Intel Atom board running Linux and ourserver software. Input and output functionality is handledby a custom I/O board (Figure 4) that interfaces with theAtom over USB. It is housed in a wooden cabinet with afront panel that aesthetically matches the existing synthe-sizer.

Four analog outputs allow control of continuous parame-ters in the patch. Each one is controlled both by a physicalknob on the front panel of the module and a virtual knob onthe web interface; turning either adjusts the output. Thepotentiometers on the physical panel are motorized to re-flect changes from web users. The output voltage range canbe mapped anywhere within ±10 volt rails, enabling thecomposer to constrain the parameter to appropriate values.Each output can be separately configured; for example, onecould be driven from a virtual keyboard in the web interfaceand output 1 volt/octave to drive a voltage-controlled os-cillator (VCO), while another might output a linear voltageto control the gain of a voltage-controlled amplifier (VCA).Voltage steps can be arbitrarily quantized via lookup tables,enabling an output to play specific notes on a given scale,

Figure 4: The custom I/O board.

for example.Eight digital ports can be configured as either inputs or

outputs. As inputs, control signals in the synthesizer patchcan be visualized on the web interface. As outputs, theycan be configured as triggers (generating a short pulse whenpressed), momentary pushbuttons (on as long as the buttonis held down), or toggles (switched between on/off each timethe button is pressed), with an appropriate representationshown in the web interface for each mode.

The module also accepts quad-channel audio input whichis mixed down to stereo for the live audio stream to theinternet. The levels of each channel can be individuallyadjusted by the composer to create the stereo mix. A stereoVU meter on the front panel both aids in setting the levelsand provides an eye-catching visual representation of theaudio to exhibit visitors. The audio is digitized by a USBaudio codec on the I/O board and encoded to MP3 and OggVorbis by the Atom for live streaming to the internet.

3. FUTURE WORKPatchwork is new to us and the synth community, and wehave only begun to explore the space of applications for it.Still, there are a number of concrete steps and novel appli-cations we are planning for the near future. First, we aredeveloping a user study to investigate the effect of Patch-work on both the online and onsite audiences’ understand-ing of and engagement with the synth. Does an online userfeel differently about controlling a remote instrument overa software synth, even if they can’t see the physical device?Audiences tend to have trouble understanding how com-plex, evolving behaviors can be produced by the interplayof signals generated by parallel, independent modules. Doesindividual control over a set of signal parameters improveaudiences’ understanding of those analog mechanics?

We are also developing a number of new applications forthe Patchwork server. In live performance, for example,Patchwork can be used in conjunction with the synth to al-low audiences or band members to patch outboard instru-ments through live effects, using mobile devices to controlthe assignments. In our experience performing as a bandwith the synth, it can be challenging to coordinate param-eters like key and tempo with the keyboardist-patcher; asa complex system of parallel modules, the synth tends tohave a lot of inertia. In this context, Patchwork could beused by band members to tune parameters from across thestage without interrupting the performance.

Patchwork’s open-ended design allows it to interface toalmost any device. We envision smartphone applicationsthat would map internal sensors to synth parameters, orturn touchscreens into portable dynamic-phrase synth con-

Figure 5: The Patchwork module integrated into the synth patch.

trollers, mimicking Korg’s Kaosillator on an analog instru-ment. Patchwork could serve as a bridge to outboard sen-sors of all kinds, or feed data from the web, like weatherpatterns, into the physical patch.

On the client side, we are developing a JavaScript key-board interface that allows multiple online users to playsynth voices. We are also interested in using Patchworkto further develop the online community around the synth,using Twitter and other means to broaden audiences’ un-derstanding and engagement. Finally, for the module itself,we imagine a family of USB peripherals that would supportinteractive installations. A camera would allow us to trackaudiences and performers using the embedded computer.A new, smaller cabinet could perform crosspoint switchingcontrolled over the web, exposing a simple patch bay on thefront and its own USB interface on the back.

4. CONCLUSIONSWe created Patchwork, an extension to a one-of-a-kind mas-sive modular synth that seamlessly brings the analog instru-ment into the digital, networked domain, while integrat-ing functionally and aesthetically into the original. Oursystem engages a global community of synth enthusiasts,many of whom have never seen, much less interacted withanything like Paradiso’s instrument. It also allows phys-ical audiences to engage with the synthesizer in a muchmore direct way than they typically can, and composersto patch in dialogue with audiences. The Patchwork webinterface serves as a hub for the growing online commu-nity that has formed around the synth’s live stream, andits cross-platform HTML5 design means that it can run onnearly any modern computing device. As a system of easilyconfigurable components, Patchwork opens up a world ofapplications that extend the reach of the massive modularsynthesizer without compromising its unique sound, tangi-ble interface, or aesthetics, bringing the historically impor-tant instrument to an eager, global audience.

Patchwork is available online at http://synth.media.

mit.edu.

5. ACKNOWLEDGMENTSThanks to the MIT Museum for presenting the synthesizerand providing a platform for engaging the public with thiswork.

6. REFERENCES[1] A. Barbosa. Displaced soundscapes: A survey of

network systems for music and sonic art creation.Leonardo Music Journal, pages 53–59, 2003.

[2] G. Hoffman and G. Weinberg. Shimon: An interactiveimprovisational robotic marimba player. Proceedings ofthe 28th of the international conference extendedabstracts on Human factors in computing systems,pages 3097–3102, 2010.

[3] S. Jorda, G. Geiger, M. Alonso, and M. Kaltenbrunner.The reacTable: exploring the synergy between livemusic performance and tabletop tangible interfaces. InProceedings of the 1st international conference onTangible and embedded interaction, pages 139–146.ACM, 2007.

[4] M. Kaltenbrunner, S. Jorda, G. Geiger, and M. Alonso.The reactable*: A collaborative musical instrument. InEnabling Technologies: Infrastructure for CollaborativeEnterprises, 2006. WETICE’06. 15th IEEEInternational Workshops on, pages 406–411. IEEE,2006.

[5] J. Paradiso. Modular Synthesizer. Timeshift:Proceedings of Ars Electronica, pages 364–370, 2004.

[6] G. Weinberg. Interconnected musical networks:Toward a theoretical framework. Computer MusicJournal, 29(2):23–39, 2005.