A Protocol for Audiovisual Cutting

Nick Collins, Fredrik Olofssonsicklincoln.org, fredrikolofsson.com

e-mail: nick@sicklincoln.org, f@fredrikolofsson.com

Abstract

We explore the extension of an algorithmiccomposition system for live audio cutting to the realmof video, through a protocol for message passingbetween separate audio and video applications. Theprotocol enables fruitful musician to video artistcollaboration with multiple new applications in liveperformance: The crowd at a gig can be cutup asvideo in synchrony with audio cutting, a musician canbe filmed live and both the footage and output audiostream segmented locked together. More abstractmappings are perfectly possible, but we emphasisethe ability to reveal the nature of underlying audiocutting algorithms that would otherwise remainconcealed from an audience. There are parallel MIDI and OSC realtimeimplementations and text file generation for non-realtime rendering. A propitious side effect of theprotocol is that capabilities in audio cutting can becheaply brought to bear for video processing.

Keywords: Realtime audio cutting, video processing,VJ techniques

1 An Introduction to VJing

A laptop musician at a contemporary club gig is mostlikely accompanied by a VJ, whose videomanipulations underscore or contrast the auralinterest. It’s recent fashion and accessibility is shownby dedicated software like VJamm, Isadora, Aestesisand Arkaos VJ. Flexible coding environments arealso available, the GEM extensions for PD, the Jitterand nato.0+55 extensions for MAX/MSP, CsoundAVand the DIPS library. There is an online community,evidenced by audiovisualizers.com, or the lev (liveexperimental video) mailing list to name but two. The past decades have seen the rise of scratch videoas an artform (Snider 2000), with visual artistsemulating the liberating cut and dice games of hiphop producers. Witness Max Headroom’s stuttering,Coldcut’s 1997 Let Us Play album, Qbert’s WaveTwisters (2000), and Addictive TV. A new product(www.cycling74.com/products/mspinky.html) makescommercially available a sync scratching tool. Toquote Hart Snider (Snider 2000) ‘people appear todance to the beat of the music when a VJ isscratching the motion of samples’. It is this capabilitythat intrigues us for automated breakbeat cutting. Though VJs can be found working all evening inthe background over many musician’s sets, new acts

are forming as audiovisual collectives from theoutset. Naturally, one of the prime media foraudiovisual experimentation and interaction iscomputer games. The first author recently witnessed awonderful presentation by Frankie the Robot DJ(frankietherobot.com), an animated DJ character builtusing a 3D games engine. The team behind Frankie iscommitted to accessible work for a club audience,dancing with PlayStation controllers in hand ratherthan bending over laptop screens. Film music is the dominant critical language, but aninteresting alternative is given by Nicholas Cook’smodel of multimedia (Cook 1998). Indeed, for VJingwe might consider Cook’s music film rather than filmmusic as an appropriate model. Snider notes ‘themusic always takes the lead in the relationshipbetween visuals and music at a show’. Cookcategorises three basic interactions, termedconformance, complementation and contest. He takesthe view that 1-1 mappings (conformance) are nottrue multimedia, but other authors and manycontemporary music videos highlight the potency ofmickey mousing (Birtwistle 2002: particularly p25,Mick Grierson’s audiovisual cutup machine Ariel). A word of caution: VJing for electronica might betaken as an admission that the visual side of laptopmusical performance is woefully boring. Yet thedistraction of the audience by the strong visualmedium is evident at some gigs. The VJ can findthemselves in the reverse situation to the film musiccomposer, trying not to overpower the audio.

2 Linking Audio and Video

In standard VJ practise, tempo tracking and spectralband energy following of the output audio allows thevideo artist to utilise gestures synchronised to themusic. Visualisers like Gforce, Whitecap or eveniTunes tend to work as fancy three dimensionalgraphic equaliser displays. Inner compositionaldetails of the music are not accessible, since thetracking acts upon the finished product. Further detailmust be explicitly passed to the video renderer. Sincevisual material provides a useful accompaniment tocomplex compositional systems in terms of markingthe algorithmic composition routines underlyingthem, we wanted to explore the possibility for liverevelation of automatic audio cutting routinespossible through visual stimuli and correspondence.Further, the techniques of audio cutting can becheaply implemented in parallel for the videoprocessing domain through an appropriate map.

This paper primarily considers the club situationwhere the feed of information is from audio artist tovisual artist, but the message passing could of coursebe two way, with auditory algorithms utilisinginformation about the video processing. We focus onthe live performance situation. SuperCollider 2 (McCartney 1998) is aprogramming language for real-time audio, perfectfor live audio synthesis and algorithmic composition.An extension set called the BBCut Library (Collins2002) is dedicated to real-time rhythmic audiosplicing and the investigation of cutting algorithms. Itcontains many procedures emulating styles of drumand bass and jungle, Aphex Twin style fast cutrepetitions and stranger routines. In a new project, incollaboration with Fredrik Olofsson, the potential ofthis library for both visual and combined audiovisualcutting is being explored. This paper describes themessaging protocol from SuperCollider to externalvideo applications developed for that task, and thecreative possibilities that result. Since BBCutsupports research into audio cutting, and the aim ofscratch video is to pass on audio cutting tricks intothe visual domain, this project is within the remit ofscratch video art. BBCut sends messages following a definedprotocol via MIDI or OSC (Open Sound Control,Wright and Freed 1997) to a video application in thebackground on the same computer or on a dedicatedmachine. The authors have investigated the use ofMAX/MSP with jitter and nato on the Mac, as well ascustom C++ OpenGL software on a PC, but thereceiver could be implemented in many otherapplications like director, flash or custom videosoftware. In the remaining sections we look at the protocol’stechnical specification and the sender realisationwithin SuperCollider and BBCut, the various receiverimplementations across a variety of realtime videoprocessing software, before going on to discuss someof the creative possibilities of audiovisual cutting andthe results of live experiments with the system.

3 The Protocol

We are not aiming to cover general informationpassing between any SuperCollider patch and a videoapplication, believing this to be too general a task andthat further context is important in refining designs.Instead the protocol is set up with the restricted goalof communicating the internal states of BBCutobjects to promote their use in video manipulation.We do not attempt to pass the hidden variables ofindividual cutting routines, but the rendering state ofthe BBCut hierarchy during performance. This issufficient however to support a number of interestingcompositional applications, detailed in a later section. The protocol is illustrated here with a bias to theMIDI output version. The messages match BBCutrendering states, as explained in the next section.

Message Values MIDI ccnumbers

Description

tempo float,0.25-10

2/34 beats per second

phrase length float,0.0–60.0

4/36 in seconds

source length float,0.0-60.0

6/38 in seconds

block length float,0.0-16.0

8/40 in seconds

phraseprop float,0.0-1.0

9/41 proportion of phrasecompleted at this time

offset float,0.0-1.0

10/42 offset (read) positioninto source. Vital formedia correspondencein live cutting

isroll boolean,0/1

11 a flag set by some cutprocedures

repeat length float,0.0-16.0

12/44 current atomic cutlength in seconds

repeatnumber

integer 13 up to 128 repeats perblock

block prop float,0.0-1.0

14 rough measure ofposition

amplitude float,0.0-1.0

16 amplitude of currentrepeat

synthparam float,0.0-1.0

any spare arbitrary messages withvalues already mappedto 0.0 to 1.0

A number of wasteful, redundant or aestheticallyunnecessary messages like phrase number, blocknumber, source index and an on/off flag wereremoved from a first draft. The receiver can infersome parameters from others, keep it’s own counters,and a cutter being on or off is indicated pragmaticallyby whether it sends messages or not! Decisions were made about using only a single 7 bitcontroller message where practical in order to cut thetransmission bandwidth. It was found useful toconcentrate on the core messages above, rather thanpush too hard to pass many messages that wouldinevitably be ignored in favour of manageablemappings. Different implementations can pass the informationin different ways. The table here reflects the choicesof the MIDI implementation, which is the mostpractical one for live work, since OSC in SC2exhibits too much network jitter and delay under MacOS 9. File writing and OSC can cope with writingany parameter value within the type specification. Itwas felt that at least a draft of the OSC version shouldbe made since this will prove critical for porting toOS X and SC Server, and allow one to escape fromthe vagaries of MIDI. All the MIDI messages are control change, beingeasy to implement and cheap to send. The protocolhas been revised after testing in live performances,and to avoid MIDI overloads and demands on thereceiver, kept to a relatively restricted form sufficientfor the artistic ends discussed below. The volume ofMIDI messages sent is around half the maximumMIDI bandwidth, for three simultaneous cuttersrunning at 180 bpm with virtuosic rolls and stutters.

4 Passing the Rendering Stateof the BBCut Library

The current performance implementation takesadvantage of the internal structure of the BBCutLibrary (Collins 2002); indeed, piggy-backing theinternal message passing allowed us to set up aneffective protocol remarkably cheaply. Furtherrevelation rests on an understanding of thephrase/blocks/repeats paradigm of the BBCutLibrary. This is a simple hierarchy of cutting, withthe phrase at the top level, subdivided into blocks thatutilise a particular offset into the source. The blocksare in turn subdivided into atomic repeats (stutters,cuts) at that common offset. BBCut messaging forrendering follows this breakdown closely,synthesising by the repeat, and updating with eachnew block or phrase. This is a live protocol, so thepertinent message is sent at the point the informationis available, that is, it depends on the time decisionmaking is communicating internally in BBCut, at theonset of new phrases, blocks or repeats, plus anexternal transmission time. It is not practical to go into full details of thestructure of BBCut here but a portion of the class treewill be helpful to see how the existing renderingengine of BBCut, as represented below by theBBCutSynthInternal class, is adapted:

BBCutSynthInternal | BBCSMessenger

/ | \BBCSFileOutLL BBCSMIDIOut BBCSOSCOut |BBCSPoster

These BBCutSynth classes allow one to build in theexternal message passing of the protocol into anyexisting BBCut code, with simultaneous use ofposting, file out, MIDI and OSC as desired. Eachindependent cutter can be tagged with an instancenumber so that independent streams of cuttinginformation are passed to the external application.The instance 0 is reserved for global messaging, likeglobal tempo and sectional markers. As a brief sideline, the file out class stores data intoa linked list while the Synth is running, so as to avoidfile writing conflicts when recording the Synthoutput. The data file is written once synthesis stops. The following SuperCollider client code shouldgive an idea of how the protocol works in practise.Adapting existing BBCut code is just a question ofnesting a BBCSMessenger derived object into thecutsynth chain.

//demonstration of client code for a single cutter that//transmits protocol messages via MIDI and to a file

Synth.record({var cutsynth, midiout,ll;midiout= MIDIOut(7); ll= LinkedList.new;

cutsynth= BBCutSynthParam.easyain(AudioIn.ar(1));cutsynth=BBCSMIDIOut(cutsynth, 1, midiout );cutsynth= BBCSFileOutLL(cutsynth, 1, ll);

BBCut.ar(cutsynth)})

Simultaneous cutters share a midi out, linked list orOSCPort object rather than wastefully create theirown instances, hence a MIDIOut, LinkedList orOSCPort object has to be passed into an argument ofthe *new class method. Since the protocol was developed, it has also beenused to pass messages from non BBCutSuperCollider code. This is easy to do simply bycreating a dummy BBCSMIDIOut object or similar,and explicitly calling the instance methods of theclass, rather than letting the BBCut structural codecall it for you.

5 Video Implementations

Prototype systems for audiovisual cutting weredeveloped independently by the two authors. Collinsused MIDI messaging from a Mac to a PC runningcustom OpenGL + AVI video code. Olofssonimplemented on a single Mac an internal MIDIstream from an early breakbeat cutting algorithm bythe first author to MAX/MSP plus nato. The current live implementation uses SuperCollider2 sending MIDI to a MAX/MSP engine with bothnato and jitter video processing. Receiving data inthe video program is a decoder subpatch that convertsback 14 bit midi into the 0.0-1.0 floating point range.The data is then outputted both via standard Maxsubpatch outlets and as send/receive pair variables foruse in remote parts of the main video program. Forcontrolling OpenGL objects parameters are usuallyscaled to 0-360 degrees and for other purposes, likethe offset parameter mapped to number of frames in acaptured video buffer, individual scaling is easilyaccomplished. With a special patch, text file outputfrom SuperCollider can be imported and decoded innon-realtime. This allows for high rendering qualitywriting of movies to disk.

Figure 1 The main video processing patch

The main control page of the live video program isshown in Figure 1. Alongside other controllers likeLFOs, audio analysers and a step sequencer, it allowsfor very interesting ways of syncing control of videosource and effect parameters to music. The syncingfeatures can be muted to avoid overuse tiring the eye. The first author has demonstrated the control of verylarge flocks of OpenGL objects by writing C++OpenGL applications as receivers on a home desktopPC, but has yet to take this live.

6 Live Rendering Possibilities

The motivations for a mapping supportingsynchronised audio and visual cutting primarilyincluded the cutting of live video streams insynchrony with audio stream cutting. A dancingaudience on a live video feed can be cut and projectedsynced to the rhythmic events they hear (see Figure 2for an example). A live drummer or other musicianplaying to a click track can have their musicautomatically cut whilst video of their performance issimilarly split. Since BBCut supports multiplesimultaneous tempi, this extends to acting upon aninstrumental ensemble even in the case of complexindependent parts. Rather than cutting live streams, where one onlyhas access to events which have already occurred, itis possible to use random access in the synchronisedcutting of any prepared audiovisual material. TheBBCut Library is already set up to deal with both thesituation of live streams and finite buffers in memory. We are not restricted to video processing but mayutilise the protocol as a guide to any live imagerendering. For instance, OpenGL 3D graphics can betriggered from the cutting protocol. Any visualdisplay can react to timing and other renderinginformation reflecting audio cutting algorithms. Inpractise, more than three independent objects trackingcutters was found visually confusing, so mostdemonstrations tended to use a limited number ofmessage passing instances. In live tests, the most effective displays ofsynchronisation were found to be for slower pacedcut sequences. The frame rate had to be sufficientlyhigh to track fast audio rhythms without aliasing. Fora typical 30fps rate, only rhythmic events up to 15Hz,corresponding to cut sizes of just larger thandemisemiquavers (32nd notes) at 120bpm, wereuseful. In practical terms then, the VJ may berequired to filter some of the faster rhythmic eventsinto sensible video rate triggers. In control of OpenGL objects, useful settings werefound by equating brightness and object size to therepeat number parameter, so that swarms of objectsexploded alongside the aurally exciting fast rolls.

7 Conclusions

Music software locked laptop performance is rifewith the difficulties of conveying interaction andgenerative composition, though sometimes a laptopperformer might project their screen display to givesome clues to the audience of the profundity of theiractions. We believe though the potential fordemonstrating tangible realtime linkage betweensimultaneous audio and video allows far moreaudience comprehension of the new automatic cuttingtechnology. The authors have undertaken live performancesusing this technology with an inevitable learningcurve but some encouraging success. A liveperformance sees laptop performers Sick Lincoln andf0 collaborate in a fusion of algorithmic breakbeatcutting, SuperColliding electronica and complexsynced visuals. All of the situations under the liverendering possibilities section above have beendemonstrated and are being further explored inpractical contexts. Rather than refining the technicalside, the authors now believe themselves to be in theposition of refining their mappings aesthetically. There is a selection of quicktime movie demos athttp://www.fredrikolofsson.com/video.

References

Birtwistle, A. 2002. “Insects, Urine and Flatulence: theradical potential of mickey mousing.” Proceedings ofCybersonica, Institute of Contemporary Arts, London, June5-7, 2002. pp 23-35.

Collins, N. 2002. “The BBCut Library.” Proceedings of theInternational Computer Music Conference, Goteborg,Sweden, September 2002. pp 313-316.

Cook, Nicholas. 1998. Analysing Musical Multimedia.Oxford University Press, New York.

McCartney, J. 1998. “Continued Evolution of theSuperCollider Real Time Synthesis Environment.”Proceedings of the International Computer MusicConference, Ann Arbor, Michigan, 1998.

Snider, H. www.artengine.ca/scratchvideo/indextoc.html.Online master’s thesis. Checked 28/2/03.

Wright,M. and Freed,A.1997. “Open Sound Control: A newProtocol for Communicating with Sound Synthesiers.”Proceedings of the International Computer MusicConference, Thessaloniki, Hellas, pp 101-104.

Figure 2 A sequence of 30 frames showing breakbeatstyle offsetting within a captured video sequence