LinnStrument: the ultimate open-source hacker instrument

Geert Bevin

Who am I?• Geert Bevin, Twitter @gbevin

• XRebel Product Manager at ZeroTurnaround

• Java Champion, Musician, Composer, Arranger, Producer, Singer, Guitarist, Gamer, Kung-Fu

• Many open-source projects including Gentoo Linux, OpenLaszlo, RIFE, Juce, …

Musical instrument hacker

Software for Eigenharp

GECO for Leap Motion

LinnStrument firmware and tools

What is the LinnStrument?

Revolutionary Music Performance Controller with 3D Note Expression

DEMO

What’s inside LinnStrument?

Final prototype before production - actual units are slightly different

Translucent silicone sheet

Chassis + circuit boards

Front-panel

Final prototype before production - actual units are slightly different

Metal chassis with wooden sides

Sensor board

Final prototype before production - actual units are slightly different

LEDs board

Final prototype before production - actual units are slightly different

Connection between circuit boards

Final prototype before production - actual units are slightly different

Arduino Due’sARM chip Serial <-> USB MIDI Shield

What you need to know about the Arduino Due

ARM Cortex-M332-bit 84MHz

CPU

SPI signals shared byLED/Sensor

Digital 33-37

Footpedals DIN <-> USBLED control

MIDI <-> Serial

Arduino Due and LinnStrument• 32-bit core, 4 bytes wide operations within single CPU clock

• CPU Clock at 84Mhz

• 96 KBytes of SRAM

• 512 KBytes of Flash memory for code

• Digital I/O pins

• Serial Peripheral Interface (SPI) pins with Slave Select

Very simple Arduino program// the setup function runs once when you press reset or power the board void setup() { // initialize digital pin 13 as an output. pinMode(13, OUTPUT); }   // the loop function runs over and over again forever void loop() { digitalWrite(13, HIGH); // turn the LED on (HIGH is the voltage level) delay(1000); // wait for a second digitalWrite(13, LOW); // turn the LED off by making the voltage LOW delay(1000); // wait for a second }

Arduino code• language based on C/C++

• concise reference with all language structures, values and functions

• Arduino IDE to get started

• the ‘setup’ function runs once, when the board starts up

• the ‘loop’ function runs at 100% CPU, over and over again

Only one execution thread• what happens in the ‘loop’ function is all that’s happening

• if something takes too long, something else isn’t happening

• guerrilla coding tactics: powerful, short strikes and get out of there

• design your algorithms to be sub-dividable

• avoid dynamic memory allocation, it’s very slow

How to use the Arduino IDE

Source code in editor

Select target board

Select serial port

Compile and Upload

I don’t like the Arduino IDE

• non-HDPI fonts

• extremely basic editor

• bad OS integration

• I use Sublime Text 3 with the Stino plugin

How the LinnStrument hardware is accessed through software

Easy LED functions• Change the color and brightness of a single LED void setLed(byte col, // Column of LED to be changed byte row, // Row of LED to be changed byte color, // Color of LED to be changed byte brightness) // Brightness of LED (0, 1, 2 or 3)

• Light up a single LED with the default color and brightness void lightLed(byte col, // Column of LED to be changed byte row ) // Row of LED to be changed

• Clear a single LED void clearLed(byte col, // Column of LED to be changed byte row ) // Row of LED to be changed

Details of LED control• LED control is done through SPI using pin 10, with mode 0, running at 21MHz SPI.begin(10); SPI.setDataMode(10, SPI_MODE0); SPI.setClockDivider(10, 4); // 84MHz divided by 4

• Digital pin 37 is an output and connected to the LED driver chips pinMode(37, OUTPUT);

• Write 32-bit data structure to SPI to control the LEDs, refreshed every 100μs digitalWrite(37, HIGH); // enable the outputs of the LED driver chips SPI.transfer(10, column, SPI_CONTINUE); // send column (left-shifted 2 bits + special bit 7) SPI.transfer(10, blue, SPI_CONTINUE); // send blue byte SPI.transfer(10, green, SPI_CONTINUE); // send green byte SPI.transfer(10, red); // send red byte digitalWrite(37, LOW); // disable the outputs of the LED driver chips

Easy sensor functions• Send 16-bit word over SPI to touch sensor to set the analog switches void selectSensorCell(byte col, // Column used by analog switches byte row, // Row used byte switch) // Switch to read X (0), Y (1) or Z (2)

• Read stable raw X value at the current col and row (returns 0-4095) int readX()

• Read stable raw Y value at the current col and row (returns 0-127) int readY()

• Read stable raw Z value at the current col and row (returns 0-127, 255 feather) byte readZ()

Details of touch sensor control• Touch sensor control is done through SPI using pin 4,

with mode 0, running at 21MHz SPI.begin(4); SPI.setDataMode(4, SPI_MODE0); SPI.setClockDivider(4, 4); // 84MHz divided by 4

• Write 16-bit data to SPI to set analog switches (see ls_sensor.ino) SPI.transfer(4, lsb, SPI_CONTINUE); // first byte of data structure SPI.transfer(4, msb); // second byte of data structure

Read touch sensor data• Touch sensor A/D input is using SPI through pin 52,

with mode 0, running at 21MHz SPI.begin(52); SPI.setDataMode(52, SPI_MODE0); SPI.setClockDivider(52, 4); // 84MHz divided by 4

• Read sensor data delayUsec(7); // wait for stable current after sensor // control changes the analog switches // delay different for each analog switch byte msb = SPI.transfer(4, 0, SPI_CONTINUE); // first byte of sensor data byte lsb = SPI.transfer(4, 0); // second byte of sensor dataint raw = (int(msb) << 8 | lsb) >> 2; // pack into int, shift from 16 to 14 bit

Reading the foot pedals• Done in ls_switches.ino, modify this method to add custom behavior void handleFootSwitchState(byte whichSwitch, boolean state)

• Digital pin 33 and 34, respectively for left and right foot switches, configured as pull-up inputs (inverted inputs: high is off, low is on)

pinMode(33, INPUT_PULLUP); pinMode(34, INPUT_PULLUP);

• Read the digital value of foot pedal states (typically every 20ms) boolean leftPedalState = digitalRead(33); boolean rightPedalState = digitalRead(34);

Details of MIDI / Serial - USB / DIN• Setting digital switches changes the communication methods from the LinnStrument

to the outside world

• Digital pin 35 switches between Serial and MIDI pinMode(35, OUTPUT); digitalWrite(35, HIGH); // high switches to Serial input/output digitalWrite(35, LOW); // low switches to MIDI input/output

• Digital pin 36 switches between USB and DIN connectors pinMode(36, OUTPUT); digitalWrite(36, HIGH); // high switches to USB input/output digitalWrite(36, LOW); // low switches to DIN input/output

That’s all the hardware stuff!

Serial Monitor for debugging

Uncomment line in ls_debug.hRecompile and upload

Enable Update OS modeWhich essentially turns on Serial instead of MIDI

Click on the Looking Glass iconTo enable Serial Monitor and reset LinnStrument

Column 17 selects the debug levelBy default it’s -1, meaning that no debug info is printed

Row 2 selects debug level 0

Row 3 selects debug level 1

Row 4 selects debug level 2

This is printed in the Serial ConsoleAlong with the actual debug messages of corresponding levels

The relevant debug code if (sensorCol == 17 && sensorRow < 4) { debugLevel = sensorRow - 1; DEBUGPRINT((-1,"debugLevel = ")); DEBUGPRINT((-1,debugLevel)); DEBUGPRINT((-1,"\n")); }

void handleTouchRelease() { DEBUGPRINT((1,"handleTouchRelease")); DEBUGPRINT((1," col="));DEBUGPRINT((1,(int)sensorCol)); DEBUGPRINT((1," row="));DEBUGPRINT((1,(int)sensorRow)); DEBUGPRINT((1,"\n"));

void handleNewTouch(byte z) { DEBUGPRINT((1,"handleNewTouch")); DEBUGPRINT((1," col="));DEBUGPRINT((1,(int)sensorCol)); DEBUGPRINT((1," row="));DEBUGPRINT((1,(int)sensorRow)); DEBUGPRINT((1," z="));DEBUGPRINT((1,(int)z)); DEBUGPRINT((1,"\n"));

Overview of firmware files

Core Files• linnstrument-firmware.ino: global data structures, setup and main loop

• ls_displayModes.ino: illuminate LEDs for the different display modes

• ls_handleTouches.ino: driven by main loop, handles touch tracking

• ls_rtos.ino: primitive scheduler calling continuous tasks during delay

• ls_settings.ino: switch behavior from UI, store and recall settings

• ls_touchInfo.ino: encapsulate sensor data into touched cells

Low-level Files• ls_calibration.ino: calibration procedure and data conversion

• ls_leds.ino: low-level communication with the LEDs

• ls_midi.ino: MIDI input, NRPN control, clock, output and queue

• ls_sensor.ino: low-level touch sensing with bias and curve

• ls_test.ino: debug functions and low-level reports

Auxilliary Features Files• ls_arpeggiator.ino: arpeggiator logic, tied to internal MIDI clock

• ls_faders.ino: MIDI CC faders touch handling and data sending

• ls_font.ino: tiny, small, and big font display, including scrolling

• ls_lowRow.ino: continuous cell evaluation for low-row features, driven by the main touch tracking

• ls_noteTouchMapping.ino: track MIDI notes to touched cells, mainly used by arpeggiator

• ls_switches.ino: handles control switches and foot pedals

Support Header Files• ls_bytebuffer.h: circular byte buffer with independent push and pop

locations, used by ls_midi.ino output queue

• ls_channelbucket.h: hands out MIDI channels from a bucket of allowed channel numbers.

• ls_debug.h: debug macros and defines

• ls_midi.h: standard MIDI status codes, used by ls_midi.ino

Useful global functionalities

Global variables

• byte sensorCol: the column number of the current sensor cell

• byte sensorRow: the row number of the current sensor cell

• byte sensorSplit: the split of the current sensor cell (0: left, 1: right)

• DisplayMode displayMode: the active display mode (see DisplayMode enum in linnstrument-firmware.ino for the values)

Global functions

• TouchInfo &cell(): the last touch data for the current cell

• TouchInfo &cell(byte col, byte row): the last touch data for a cell

• FocusCell &focus(byte split, byte channel): the current cell that has the focus for a particular split and channel (FocusCell is a structure that just contains col and row)

Some specific firmware examples

Per-finger touch-tracking• With a general purpose CPU, you’d model this as touch ‘sessions’

that are dynamically created and have a life of their own

• Too much memory churn for Arduino, too much book-keeping also

• Instead, have a touch state for each cell

• Transfer data between cells since we don’t support two fingers touching the same cell

Inside ls_handleTouches.inovoid handleNewTouch(byte z) { // ... snip ...   // check if the new touch could be an ongoing slide to the right if (potentialSlideTransferCandidate(sensorCol-1)) { // if the pressure gets higher than adjacent cell, // the slide is transitioning over if (isReadyForSlideTransfer(sensorCol-1)) { transferFromSameRowCell(sensorCol-1); handleXYZupdate(z); } // otherwise act as if this new touch never happened else { cellTouched(transferCell); } } // similar for slide to the left

Check potential slide transferboolean potentialSlideTransferCandidate(int col) { if (col < 1) return false; if (sensorSplit != getSplitOf(col)) return false; if (!isLowRow() && (!Split[sensorSplit].sendX || !isFocusedCell(col, sensorRow))) return false; if (isLowRow() && !lowRowRequiresSlideTracking()) return false; if (isStrummingSplit(sensorSplit)) return false;   // the sibling cell has an active touch return cell(col, sensorRow).touched != untouchedCell && // either a release is pending to be performed, or (cell(col, sensorRow).pendingReleaseCount || // both cells are touched simultaneously on the edges abs(cell().calibratedX() - cell(col, sensorRow).calibratedX()) < TRANSFER_SLIDE_PROXIMITY); }

Is the sibling cell ready for the transfer?

boolean isReadyForSlideTransfer(int col) { // there's a pending release waiting return cell(col, sensorRow).pendingReleaseCount || // the cell pressure is higher cell().rawZ > cell(col, sensorRow).rawZ; }

Perform the data transfer 1/3void transferFromSameRowCell(byte col) { cell().initialX = cell(col, sensorRow).initialX; cell().initialReferenceX = cell(col, sensorRow).initialReferenceX; cell().lastMovedX = cell(col, sensorRow).lastMovedX; cell().fxdRateX = cell(col, sensorRow).fxdRateX; cell().rateCountX = cell(col, sensorRow).rateCountX; cell().initialY = cell(col, sensorRow).initialY; cell().note = cell(col, sensorRow).note; cell().channel = cell(col, sensorRow).channel; cell().fxdPrevPressure = cell(col, sensorRow).fxdPrevPressure; cell().fxdPrevTimbre = cell(col, sensorRow).fxdPrevTimbre; cell().velocity = cell(col, sensorRow).velocity; cell().vcount = cell(col, sensorRow).vcount; noteTouchMapping[sensorSplit] .changeCell(cell().note, cell().channel, sensorCol, sensorRow);  

Perform the data transfer 2/3 if (cell(col, sensorRow).touched != untouchedCell) { cell(col, sensorRow).touched = transferCell; } cell(col, sensorRow).initialX = -1; cell(col, sensorRow).initialReferenceX = 0; cell(col, sensorRow).lastMovedX = 0; cell(col, sensorRow).fxdRateX = 0; cell(col, sensorRow).rateCountX = 0; cell(col, sensorRow).initialY = -1; cell(col, sensorRow).note = -1; cell(col, sensorRow).channel = -1; cell(col, sensorRow).fxdPrevPressure = 0; cell(col, sensorRow).fxdPrevTimbre = 0; cell(col, sensorRow).velocity = 0; cell(col, sensorRow).pendingReleaseCount = 0; // do not reset vcount!  

Perform the data transfer 3/3

// transfer the focus if this was the focused cell byte channel = cell().channel; if (channel != -1 && col == focus(sensorSplit, channel).col && sensorRow == focus(sensorSplit, channel).row) { focus(sensorSplit, channel).col = sensorCol; focus(sensorSplit, channel).row = sensorRow; } }

Finding the right MIDI channels• Note-per-channel distributes up to 16 MIDI channels across notes

• Notes should reuse same channel as late as possible (release trails)

• Intuitively you’d scan all the active notes and determine which channel is available for a new note, which is again too much overhead

• We use a bucket of available channels, channels bubble up or sink down

• A new note merely has to take the next channel from the top

• Fully encapsulated inside ls_channelbucket.h

Low-row functionalities• Intuitively you’d detect a touch on low-row cells when it’s active

• Then evaluate state of every other cell and trigger behavior

• This is again too much overhead

• Instead keep track of low-row start/stop in a state machine

• Piggy-back when processing each cell in the main loop to evaluate appropriate low-row behavior

Inside ls_handleTouches.ino

void handleXYZupdate(byte z) { // ... snip ... handleLowRowState(z);   if (isLowRow()) { if (newVelocity) { lowRowStart(); } return; } // ... snip ... }

void handleTouchRelease() { // ... snip ... if (isLowRow()) { lowRowStop(); } // ... snip ... }

Inside ls_lowRow.ino 1/2void lowRowStart() { switch (Split[sensorSplit].lowRowMode) { case lowRowStrum: lowRowState[sensorCol] = pressed; break; // ... snip, different for each low-row mode } }   void lowRowStop() { switch (Split[sensorSplit].lowRowMode) { case lowRowStrum: lowRowState[sensorCol] = inactive; break; // ... snip, different for each low-row mode } }

Inside ls_lowRow.ino 2/2void handleLowRowState(byte z) { // this is a low-row cell if (isLowRow()) { // send out the continuous data for low-row cells if (cell().velocity) { // ... snip, different for each low-row mode } } // this is a non low-row cell else { switch (Split[sensorSplit].lowRowMode) { case lowRowStrum: // uses lowRowState to correlate with column handleLowRowStrum(); break; // ... snip, other cases } } }

Sending MIDI bytes• MIDI was causing the LEDs to flicker

• Too much time was spent at once (need more guerrilla!)

• Created a MIDI queue to continuously send byte-by-byte from our RTOS

• Arduino UART classes still caused problems: synchronous wait for readiness when sending

Patched UARTClass--- UARTClass.cpp 2014-11-10 14:55:10.000000000 +0100 +++ UARTClass.cpp 2014-10-10 19:39:43.000000000 +0200 @@ -109,9 +109,15 @@   size_t UARTClass::write( const uint8_t uc_data ) { + return write(uc_data, true); +} + +size_t UARTClass::write( const uint8_t uc_data, const bool wait ) +{ // Check if the transmitter is ready - while ((_pUart->UART_SR & UART_SR_TXRDY) != UART_SR_TXRDY) - ; + while ((_pUart->UART_SR & UART_SR_TXRDY) != UART_SR_TXRDY) { + if ( !wait ) return 0; + }   // Send character _pUart->UART_THR = uc_data;

Queuing of messagesByteBuffer<4096> midiOutQueue; // called for each MIDI message that is sent void queueMidiMessage(MIDIStatus type, byte param1, byte param2, byte channel) { param1 &= 0x7F; param2 &= 0x7F; midiOutQueue.push((byte)type | (channel & 0x0F)); midiOutQueue.push(param1); if (type != MIDIProgramChange && type != MIDIChannelPressure) { midiOutQueue.push(param2); } }

// continuously called by our RTOS void handlePendingMidi() { if (!midiOutQueue.empty()) { if (Serial.write(midiOutQueue.peek(), false) > 0) { // patched UART method midiOutQueue.pop(); } } }

More information at http://www.rogerlinndesign.com

@gbevin

Questions?