Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf ·...
Transcript of Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf ·...
![Page 1: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/1.jpg)
Hands On: Multimedia Methods for Large Scale Video Analysis (Project Meeting)
Dr. Gerald Friedland, [email protected]
1
![Page 2: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/2.jpg)
Today
2
![Page 3: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/3.jpg)
Today
2
• Quick Update
![Page 4: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/4.jpg)
Today
2
• Quick Update• Answers to Questions
![Page 5: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/5.jpg)
Today
2
• Quick Update• Answers to Questions• More on ICSI
![Page 6: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/6.jpg)
Today
2
• Quick Update• Answers to Questions• More on ICSI• Some frameworks and tools to work
with sound
![Page 7: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/7.jpg)
Today
2
• Quick Update• Answers to Questions• More on ICSI• Some frameworks and tools to work
with sound• Project discussion
![Page 8: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/8.jpg)
Update: Amazon Glazier
3
More information:http://aws.amazon.com/glacier/
![Page 9: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/9.jpg)
Q&A
4
![Page 10: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/10.jpg)
Q&A
4
Why are we learning about audio in a VIDEO analysis class?
![Page 11: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/11.jpg)
Q&A
4
Why are we learning about audio in a VIDEO analysis class?
- Video = Acoustic + Visual + Temporal Information
![Page 12: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/12.jpg)
Q&A
4
Why are we learning about audio in a VIDEO analysis class?
- Video = Acoustic + Visual + Temporal InformationWhat to use depends on task, see example
![Page 13: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/13.jpg)
Video Navigation
• Record• Store• Play• Random Seek• Annotate Manually
5
Navigation Capabilities: 1985
![Page 14: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/14.jpg)
Video Navigation
• Record• Store• Stream• Play• Random Seek• Annotate Manually
6
Navigation Capabilities: 2010
![Page 15: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/15.jpg)
Motivation
7
Yahoo! Video Segmentation Task of ACM Multimedia Grand Challenge 2009:
“Create navigation based on narrative elements”
![Page 16: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/16.jpg)
Joke-O-Mat: Demo
8
![Page 17: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/17.jpg)
More on ICSI
9
![Page 18: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/18.jpg)
More on ICSI
9
DEMO
![Page 19: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/19.jpg)
Audacity
10
Open Source wave editor and filter:http://audacity.sourceforge.net/
![Page 20: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/20.jpg)
Wavesurfer
11
Open Source wave editor and filter, specialized for speech:http://www.speech.kth.se/wavesurfer/
![Page 21: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/21.jpg)
Praat
12
Open Source speech experimental toolkithttp://www.fon.hum.uva.nl/praat/
![Page 22: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/22.jpg)
Other useful tools
13
•sox -- command-line based audio filter and converter: http://sox.sourceforge.net/
•ffmpeg -- command-line based audio and videoconverter http://ffmpeg.org/
•MPlayer (mencoder) -- open source video player and converter: http://www.mplayerhq.hu/
![Page 23: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/23.jpg)
More Tools: HTK
14
Open Source Speech Recognition Toolkit: http://htk.eng.cam.ac.uk/
![Page 24: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/24.jpg)
More Tools: A nice collection of them
15
Princeton SoundLab:http://soundlab.cs.princeton.edu/software/
![Page 25: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/25.jpg)
More Tools: BeamformIT
16
Open Source Beamformer: http://beamformit.sourceforge.net/
![Page 26: Hands On: Multimedia Methods for Large Scale Video Analysis …fractor/fall2012/cs294-6-2012.pdf · 2012. 10. 5. · Hands On: Multimedia Methods for Large Scale Video Analysis (Project](https://reader036.fdocuments.us/reader036/viewer/2022071111/5fe751c0e6ec0d1bbd493945/html5/thumbnails/26.jpg)
Project Discussion
17