Visualization for Multimedia Contents
Transcript of Visualization for Multimedia Contents
Visualization for Multimedia Contents
Takayuki ItohOchanomizu University, Japan
Shonan NII Meeting 2014/3/10
Itoh Laboratory, Ochanomizu University
This talk introduces our works on…
4
Itoh Laboratory, Ochanomizu University
Music browsingPhotograph browsing
[AVI 10, ACM SAC 11 etc][ACM SAC 11, IV 11 etc]
5
Itoh Laboratory, Ochanomizu University
Part I:Photograph browsing
Photograph browsing tools
• To browse large photo collection intuitively
6
Itoh Laboratory, Ochanomizu University
PhotoMesa Design Gallery
Photograph browsing tools
• To browse large photo collection intuitively• Personal or professional• Structured (tree, graph, time-sequence…) or scattered• Image processing algorithms
– Meta-data based: (time, place, person …)– Low-level feature based: (color, texture, shape …)– High-level recognition based: (human face, object, scene …)
7
Itoh Laboratory, Ochanomizu University
PhotoMesa Design Gallery
Our zooming interface for photo exploration
• Overview the photo groups first, then zoom in– [Zoom-out] display representative photos of groups– [Zoom-in] display each photos in the zoomed groups
8
Itoh Laboratory, Ochanomizu University
[IV08]
Zoom Out
Zoom In
Tree visualization for photo browsing
• Hierarchy representation by nested rectangles• Mapping photos onto icons (= leaves) (for zoom in)
• Mapping photos into rectangles (= branches) (for zoom out)
9
[TVCG04, CG&A06]
Itoh Laboratory, Ochanomizu University
Photo on icon
Photo in rectangle
Hierarchy representation
Photo collection as “life log”
• Many photos by travel, events, ...• Often we reflect our life from “When, Where, Who”
↓• Enjoyable life reflection by looking at photos
from the viewpoints of “When, Where, Who”
10
Visit to Kyoto every year
Hokkaido with family in 2007
Hawaii with friends in 2009
High school travel to Okinawa
Itoh Laboratory, Ochanomizu University
Itoh Laboratory, Ochanomizu University
Photo browser by “When, Where, Who”
• Photo display based on time and place
• Smooth interaction(Rotation, zoom)
• All-in-one display of faces in the photos
• Photo discovery by interaction among time, place, and face
11
Extracted facesY: Latitude
X: Longitude
Z: Time
Photo display
Rotation
Zoom in
Zoom in
[AVI 10, ACM SAC 11 etc]
Life log of a student (1/2)
12
Europe with family
Guam with friends
Domestic trip in lab.
Europe with family
Itoh Laboratory, Ochanomizu University
Lab.
Friends
FamilyFamily
Photograph owner
[AVI 10, ACM SAC 11 etc]
Life log of a student (2/2)
13
Mid. America with family
Japan & Korea trips in lab.
Europe with friends USA trip
in lab.Japan with
friends
Australia
Itoh Laboratory, Ochanomizu University
Friends
FamilyFamily
Lab.
[AVI 10, ACM SAC 11 etc]
14
Itoh Laboratory, Ochanomizu University
Part II:Music browsing
Music browsing tools in my lab
15
Itoh Laboratory, Ochanomizu University
Overview & summarization
Recommendation & preference analysis
Visual user interface
Music browsing tools
• Quick finding & understanding before listening• Goal & purpose
– [Daily use] search, recommendation, …– [Expert use] education, practice, performance, …
• Input data– Audio (WAV, MP3 …) or Note (MIDI …)
• Musical genre– Pops or Classics– With vocals or Without vocals
16
Itoh Laboratory, Ochanomizu University
Music browsing tools in my lab
17
Itoh Laboratory, Ochanomizu University
Overview & summarization
Recommendation & preference analysis
Visual user interface
Visualizing structure of orchestra music
• Summarization of classical music score which has tens of note tracks
• Coloring of note blocks based on melodies
18
Woodwind
Brasswind
Strings
Former Latter
Itoh Laboratory, Ochanomizu University
[IV 11]
Example: “Valse des fleurs”
1919
Melody Harmony Bass Others
Remark to Purple melody
Itoh Laboratory, Ochanomizu University
[IV 11]
Condensation (= Vertical compression)
• Reduce the number of tracks while preserving important melodies
20
Itoh Laboratory, Ochanomizu University
[IV 11]
Condensation (= Vertical compression)
• Reduce the number of tracks while preserving important melodies
21
Itoh Laboratory, Ochanomizu University
[IV 11]
Visual Music Recommendation
• Input evaluation→ Learn & recommend preferable tunes
22
Itoh Laboratory, Ochanomizu University
Plot = tune
Automaticallygenerated
playlist
Not yet Playing Negative Positive
Buttons (play, stop,positive, negative)
[VINCI11]
Recommendation patterns
• Users A&B have similar preferences– Analog sounds– Simple arrange
• User C has much different preference– Low inharmonicity
23
Itoh Laboratory, Ochanomizu University
Loudness Loudness Loudness
High/low ratio High/low ratio Inharmonicity
User A User B User C
[VINCI11]
Conclusion
• Photograph browsing– Zooming interface [Tree vis.]– Meta-data based structuring
• Music browsing– Summarizing scores [Time-varying vis.]– Recommendation with scatterplots [Multi-dimensional vis.]
24
Itoh Laboratory, Ochanomizu University
References
Hierarchical Data Visualization Using a Fast Rectangle-Packing Algorithm, TVCG, 10(3), 302-313, 2004.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRJPE13.pdfHierarchical Visualization of Network Intrusion Detection Data in the IP Address Space, CG&A, 26(2), 40-47, 2006.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRJPE14.pdfCAT: A Hierarchical Image Browser Using a Rectangle Packing Technique, International Conference on Information Visualisation (IV08), 82-87, 2008.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE37.pdfA Personal Photograph Browser for Life Log Analysis based on Location, Time, and Person, ACM Symposium on Applied Computing, 1250-1257, 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE58.pdfColorscore - Visualization and Condensation of Structure of Classical Music, International Conference on Information Visualisation (IV2011), 420-425, 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE65.pdfMusiCube: A Visual Music Recommendation System featuring Interactive Evolutionary Computing, Visual Information Communication - Information Symposium (VINCI'11), 2011.http://itolab.is.ocha.ac.jp/~itot/paper/ItotRICPE67.pdf
25
Itoh Laboratory, Ochanomizu University