KEYed User Interface An HCI Theoretic Design of a Novel Music Composition Interface by Farhan...
-
Upload
della-whitehead -
Category
Documents
-
view
212 -
download
0
Transcript of KEYed User Interface An HCI Theoretic Design of a Novel Music Composition Interface by Farhan...
KEYKEYeded User Interface User InterfaceAn HCI Theoretic Design of a Novel An HCI Theoretic Design of a Novel
Music Composition InterfaceMusic Composition Interface
byby
Farhan Mohamed
supervised by
Dr. Sidney Fels Human Communication Technologies Laboratory
The University of British ColumbiaDepartment of Electrical and Computer Engineering Vancouver, British ColumbiaCanada
MotivationMotivation
“While composing with my music studio computer, I find it stressful to use my piano keyboard, computer keyboard and mouse repetitively, especially when I feel creative and work over long hours”
- Composer/Producer Kevin Kowal
Computer Keyboard
Piano KeyboardComputer Mouse
Kevin’s Studio Computer Setup
GoalGoal
MomentaryFootswitch
Electronic Piano Keyboard Electronic Piano Keyboard for for
Performance and EditingPerformance and Editing
Single-Point TouchpadSingle-Point Touchpad
The KEYed User Interface project was created to bind music performance and editing tasks by using the piano keyboard as the sole input device for all tasks
Kevin’s Setup using KEYed User Interface
Overview of PresentationOverview of Presentation • Current music composition workstation setup
• Alternative approaches and their shortcomings
• Applying findings in HCI
• KEYed User Interface system
• Experiments and results
• Conclusions
Switching between multiple input devices• Long device acquisition time• Not comfortable to use• Lots of physical space occupied by the input devices
Music Composition Workstation Music Composition Workstation
Computer Keyboard
Piano KeyboardComputer Mouse
Composition Workstation Setup
Alternative ApproachesAlternative Approaches
Mackie’s ® Human User Interface ® Meredith Brooks’ Studio Layout
• Interface separates performance and editing
- Mackie HUI hardware
• Interface integrates performance and editing
- Logic® Audio Music Software
Applying HCI FindingsApplying HCI Findings
• We apply HCI studies to composition, integrating the two
tasks
• HCI studies have been applied to performance and editing
HCI FindingsHCI Findings
• Mode Errors and their Prevention
• Human Manual Studies
- Bimanual
• Tacit Knowledge and
Metaphors
• Space - Time Multiplexing Tradeoffs in Input Devices
HCI FindingsHCI Findings
MomentaryFootswitch
Electronic Piano Electronic Piano KeyboardKeyboard
Single-Point TouchpadSingle-Point Touchpad
Overview ofOverview of KEYKEYeded User User InterfaceInterface
HCI Principles AppliedHCI Principles Appliedtoto KEYKEYeded User Interface User Interface
MomentaryFootswitch
Electronic Piano Electronic Piano KeyboardKeyboard
Single-Point TouchpadSingle-Point Touchpad
Mode Error Prevention
Bimanual Action
Space Multiplexed
E2
Arrange Octave (C4-B4)Arrange Octave (C4-B4)
Mapping Example:Mapping Example:
Transport Octave (C3-Transport Octave (C3-B3)B3)
Mapping Example:Mapping Example:C3 Open/Close transport window
C5 Open/Close key edit window
F#6 Undo
Tracks Parts
Sequence
C2
B2
C3
B3
C4
B4
C5
B5
C6
B
6
Touch Pad Octave (C2-B2)Touch Pad Octave (C2-B2)
Mapping Example:Mapping Example:
General Edit Octave (C6-B6)General Edit Octave (C6-B6)
Mapping Example:Mapping Example:
Key Edit Octave (C5-B5)Key Edit Octave (C5-B5)
Mapping Example:Mapping Example:
F4 Move down the track
E2 Selecting volume
G3 Click On/Off (B3)
Note
Piano MappingsPiano Mappings
KEYKEYeded UI Prototype 1 UI Prototype 1
• Prototype I - Steinberg ® Cubase ® 5.0 VST 32 on Windows ® 98
• KEYed Pedal for Mode Switching• Octaves and the Notes are labeled
KEYKEYeded UI Prototype I StudiesUI Prototype I Studies
• The interaction methods compared are:
a) Computer Keyboard + Piano Keyboard
b) KEYed User Interface Prototype I
• Testing for Response Time with 6 Composers
• Composing Tasks A&B, Repeated Once
Note: Task A and Task B involve repetitive performing and editing, thereby requiring the composers to switch back and forth between modes on their own.
KEYKEYeded UI Prototype I ResultsUI Prototype I Results
0
20
40
60
80
100
120
140
seco
nds
Computer keyboard
Piano controller
Mean Time with 95% Confidence error bars
Test 1 Test 2
• Learning Time: 5 minutes• Results using paired t-test:
Test 1 - Piano controller faster, significant (p<0.05)
Test 2 - Piano controller faster, not significant (p<0.06)• No mode errors detected
KEYed UI Prototype IIKEYed UI Prototype II
• Prototype II platform− Steinberg ® Nuendo ® 1.5 on Windows ® 2000
• KEYed Pedal and Sustain Pedal• A single-point touchpad for 1D and 2D tasks
Single-Point Touchpad
KEYed Pedal
KEYKEYeded UI Prototype II StudiesUI Prototype II Studies1. The interaction methods compared are:
a) Computer Keyboard / Mouse + Piano Keyboard
b) KEYb) KEYeded UI II with Audio
c) KEYc) KEYeded UI II without Audio • Testing for Response Time with 10 Composers• Simple, Moderate and Complex editing tasks
Note: All tasks involve repetitive composing and editing, thereby requiring the composers
to switch back and forth between modes on their own.
2. Case study for a one week period with 1 Composer
KEYKEYeded UI Prototype II StudiesUI Prototype II Studies
Experiment Booth
KEYed user interface Layout
Mean Completion times
KEYKEYeded UI Prototype II ResultsUI Prototype II Results
• Learning Time: 16 minutes• KEYed Prototype II with Audio vs. Computer Keyboard / Mouse :
Response time faster with the KEYed UI with no significance • KEYed Prototype II without Audio vs. Computer Keyboard / Mouse :
Response time faster with the KEYed UI with no significance
0
10
20
30
40
50
60
70
80
90
se
co
nd
s
Computer Keyboard /Mouse
KEYed (Audio)
KEYed (No Audio)
Simple Moderate Complex
KEYKEYeded UI Prototype II ResultsUI Prototype II Results
Case study results• Positive feedback• Very natural to use• Requested more mappings• Suggested the use of musical chords• Single-point touchpad too small
• Piano keyboard can be used as an alternative input device in music composition workstations
• A momentary foot pedal like the KEYed foot pedal is easy to acquire, and is an ideal mode switch for switching between the performing and editing modes
• Though small, the single-point touchpad is found to be natural to use for bimanual tasks
ContributionsContributions
ContributionsContributions
• KEYed user interface illustrates how an appropriate mapping of the layout, feedback, and context is important in the design of user interfaces.
• Results published at CHI’2002 and ICMC’2002.
• Overlaying composition functions on other controllers
• Exploring musical structures and mappings, for example, scales and chords
• Providing user customizable mappings
Future DirectionsFuture Directions
ConclusionsConclusions
• Current approaches to music composition workstations design divide the performance and editing tasks
• In pursuing the KEYed project, we have found that findings in HCI can be applied to the problems in current music composition workstation design
• Gives rise to faster response time in devices with minimal learning
• Ultimately, greater comfort leads to greater creative flow
and better music
QuestionsQuestions
KEYed UI System DesignKEYed UI System Design
MIDI OUT MIDI IN MIDI THRU
SB PCI 128
MIDI OUT MIDI IN MIDITHRU
SB Live
MIDI OUT MIDI IN MIDITHRU
KEYed Messaging
System
Cubase 5.0 VSTOr
Nuendo 1.5
KEYed DynamicLink Library
(Window Handles)
EventWindows
Queue
KEYed Sustain Pedal Pedal
Serial Port
Single-point Touchpad
Win
do
ws A
pp
lication
s
Measured Mean Task Completion Time vs Predicted KLM Times Measured Means
(Seconds)Predicted Time A
(Seconds)Predicted Time B
(Seconds)
Task A
Computer Keyboard 116.2218333 67.4 + R1 73.4 + R1
KEYed Prototype I 101.3736667 58.6 + R1 58.6 + R1
Task B
Computer Keyboard 96.71233333 56.2 + R2 62.2 + R2
KEYed Prototype I 86.05816667 42.2 + R2 42.2 + R2
Task C
Computer Keyboard (CK) / Mouse (M)
40.74 CK: 44.4 + R3 CK: 46 + R3
M: 39.8 + R3 M: 41.4 + R3
KEYed Prototype II (with audio)
40.7399 32.2 + R3 32.2 + R3
KEYed Prototype II (without audio)
42.1412 32.2 + R3 32.2 + R3
Touchpad 52.1371 39.8 + R3 41.4 + R3
Task D
Computer Keyboard (CK) / Mouse (M)
62.6374 CK: Not measurable CK: Not measurable
M: 41.3 + R4 M: 43.3 + R4
KEYed Prototype II(with audio)
58.245 22.6 + R4 22.8 + R4
KEYed Prototype II(without audio)
50.0287 22.6 + R4 22.8 + R4
Touchpad 83.6633 41.3 + R4 43.3 + R4
Task E
Computer Keyboard (CK) / Mouse (M)
64.923 CK: Not measurable CK: Not measurable
M: 44.2 + R5 M: 46.4 + R5
KEYed Prototype II(with audio)
62.5232 28.8 + R5 29.2 + R5
KEYed Prototype II(without audio)
59.9132 28.8 + R5 29.2 + R5
Touchpad 99.1176 44.2 + R5 46.4 + R5
Measured Mean Task Completion Time vs Predicted KLM Times
0
20
40
60
80
100
120
se
co
nd
s
Measured Mean Times
Predicted Time A
Predicted Time B
0
20
40
60
80
100
120
140s
ec
on
ds
Measured Mean Times
Predicted Time A
Predicted Time B
CK KP1 CK KP1
TASK A TASK B
CK/M KP2A KP2NA TP CK/M KP2A KP2NA TP CK/M KP2A KP2NA TP TASK C (Simple) TASK D (Moderate) TASK E (Complex)
Where,
CK = Computer Keyboard
KP1 = KEYed Prototype I
KP2 = KEYed Prototype I
M = Mouse
KP2A = KEYed Prototype II with Audio
KP2NA = KEYed Prototype II without Audio
TP = Touchpad
Single point touchpad Ex. Drawing graphs with the touchpad
TASK A
1. Open the Transport window 2. On the current MIDI track, perform a recording by playing on the piano for the
pre-set number of bars with the sustain pedal pressed continuously during the recording
3. After the recording stops, stop playing and move the play cursor to the 0 position
4. Play the bars 0 to 10 of the pre-recorded piece once
5. Stop playing and move the play cursor to the 0 position 6. Close the Transport window
TASK B 1. Open the Transport window
2. On the current MIDI track, perform a recording by playing on the piano for the
pre-set number of bars with the sustain pedal pressed continuously during the recording
3. After the recording stops, stop playing and move the play cursor to the 0 position 4. Close the Transport window 5. Copy the part you just recorded 6. Forward to bar 20
7. Paste the selected track at bar 20 8. Open the VST mixer
9. Move the master fader from the position 0 to position -30.6, or from -30.6 to 0 on
the VST mixer 10. Close the VST mixer
TASK C
1. Select MIDI track 2 2. Select the MIDI part on track 2
3. Open the Key Edit window
4. Draw a note for 3 bars
5. Draw a fade-in volume graph on the volume graph area for 2 bars
6. Close the Key edit window 7. Open the Transport window
8. Perform a Recording by playing on the piano for the preset bars with the sustain
pedal pressed continuously during the recording 9. After the recording stops, stop playing and move the play cursor to the 0 position 10. Close the Transport window
11. Open the EQ window by opening the mixer
12. Turn the Hi-Mid Gain knob to +24
13. Close the EQ and Mixer windows
14. Open the EQ window by opening the mixer
15. Turn the Low-Mid frequency knob to 20000 Hz
16. Close the EQ and Mixer window
REACION TIME FOR TASKS USING DIFFERENT INPUT DEVICES
MEANS ComplexityID STAT TASK 1 TASK 2 TASK 3 ID OVLL
Mean 40.74 62.6374 64.923 56.1BL s 3.92 21.46 20.62
N 10 10 10 30
Mean 52.1371 83.6633 99.1176 78.31TP s 5.92 39.94 25.68
N 10 10 10 30
Mean 40.7399 58.245 62.5232 53.84KA s 6.46 31.63 29.05
N 10 10 10 30
Mean 42.1412 50.0287 59.9132 50.69KW s 6.46 12.45 20.6
N 10 10 10 30
Mean 43.94 63.64 71.62TASK sOVLL N 40 40 40