Data Formats ( HPC Visualization with ParaView Workshop) Shuaib Arshad April 23, 2014.
Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization...
Transcript of Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization...
![Page 1: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/1.jpg)
Big Visual Data, Deep Learning,
and Open Source
Dr. Anthony Hoogs anthony. [email protected]
Senior Director of Computer Vision http://www.kitware.com/company/team/hoogs.html
Kitware, Inc.
Clifton Park, NY
1 Approved for public release. Distribution unlimited.
![Page 2: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/2.jpg)
The Deep Learning Revolution
2
Fortune magazine Sep. 28, 2016
“Neural nets aren’t new. What’s changed is that today computer scientists have finally harnessed both the vast computational power and the enormous storehouses of data—images, video, audio, and text files strewn across the Internet—that, it turns out, are essential to making neural nets work well.”
“Google had two deep-learning projects underway in 2012. Today it is pursuing more than 1,000.”
“[Google, Amazon, Microsoft, Apple] all have features that let you search or automatically organize collections of photos with no identifying tags. You can ask to be shown, say, all the ones that have dogs in them, or snow, or even something fairly abstract like hugs.”
![Page 3: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/3.jpg)
The (Re-)Birth of Convolutional Neural Networks
3
A. Krizhevsky, I. Sutskever, and G. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” Neural Information Processing Symposium, 2012.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. IEEE Computer Vision and Pattern Recognition, 2009.
![Page 4: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/4.jpg)
A Perfect Storm
4
GPU
Open Source
Software
Deep Learning
Revolution
![Page 5: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/5.jpg)
Kitware Open Source Platforms
ParaView
Resonant
CMake
CDash
• KWIVER Kitware Imagery and Video Exploitation and Retrieval
• VTK the visualization toolkit • ParaView large data analysis &
visualization application • ITK insight image analysis toolkit • CMake cross-platform build system
– CDash, CTest, CPack, software process tools
• Resonant/Girder informatics and information visualization
• Kiwi & VES mobile visualization • IGSTK, CTK, vxl, Open Chemistry Project, VolView, tubeTk, and more… • MIDAS for computational scientific
research, testing, and visualization
![Page 6: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/6.jpg)
CNN Open Source Platforms
6
UC Berkeley http://caffe.berkeleyvision.org/
Google https://www.tensorflow.org/
![Page 7: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/7.jpg)
CNN Open Source Models
7
Caffe 41 models
![Page 8: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/8.jpg)
AlexNet
8
65M parameters
A. Krizhevsky, I. Sutskever, and G. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” Neural Information Processing Symposium, 2012.
![Page 9: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/9.jpg)
Layer 1 Filters
Slide credit: Yann LeCun
![Page 10: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/10.jpg)
Layer 1: Top-9 Patches
![Page 11: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/11.jpg)
Layer 2: Top-9 Patches
• Patches from validation images that give maximal activation of a given feature map
![Page 12: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/12.jpg)
Layer 2: Top-9 Patches
![Page 13: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/13.jpg)
Layer 3: Top-9 Patches Layer 3: Top-9 Patches
![Page 14: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/14.jpg)
Layer 3: Top-9 Patches
![Page 15: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/15.jpg)
Layer 4: Top-9 Patches
![Page 16: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/16.jpg)
AlexNet Results
16
Top 5 Classes Training Images
![Page 17: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/17.jpg)
Training Deep Networks: ImageNet • Many deep networks for
image recognition are trained on ImageNet
• ImageNet contains a large number of training images with wide diversity
• Using 1M+ images for training is typical
• Days worth of training time • What can you do if your
dataset is different from ImageNet content? – Similar datasets do not
exist for aerial / overhead / ISR data
17
14M+ Images 21K+ Categories
![Page 18: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/18.jpg)
Training Alternatives • Train a network from scratch
– Requires large training volume, significant ground truth, computation time (days), experimental iteration
• Refine an existing network – Still requires potentially large training volume with significant ground
truth – Relies on visual features being similar across datasets – open
question • Simulation
– Simulated scenes can provide both training data and labels (known from underlying model)
– Required level of fidelity is unknown • Generative models
– Training process tries to reproduce the input imagery – Hopefully produces features useful for discrimination
18 Training deep networks is still an art
![Page 19: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/19.jpg)
Deep Learning Image Descriptors • AlexNet or any CNN can be used as a generic image
descriptor
19
Layer 7 Fully connected 4096 dimensions
![Page 20: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/20.jpg)
Image Query
20
Start IQR with a single positive examplar
Dataset contains 832 images with 55-100 images per type.
Use CAFFE AlexNet Layer 7 as an image descriptor
![Page 21: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/21.jpg)
Random Selections from Leeds Butterfly Dataset
21
Josiah Wang, Katja Markert, and Mark Everingham Learning Models for Object Recognition from Natural Language Descriptions In Proceedings of the 20th British Machine Vision Conference (BMVC2009)
![Page 22: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/22.jpg)
Results from Single Exemplar
22
![Page 23: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/23.jpg)
Interactive Query Refinement
Imagery from Social Media or other sources
SMQTK Application
Feature Computation Data Store
Indexing/Searching/Ranking Engine
Text Query or Exemplar
X X X X
X X X X
X X X X
Feedback Feedback Feedback
IQR Round 1
IQR Round 3
IQR Round 2
Final Query Result Set
![Page 24: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/24.jpg)
One Refinement Based on Adjudications from Previous Slide
24
![Page 25: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/25.jpg)
Visualization of Image-Induced Networks
25
![Page 26: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/26.jpg)
Social Multimedia Query ToolKit (SMQTK) • Indexing, Searching and Query Refinement on any
images • Rapid query times from ITQ-based indexing • Plugin based architecture allows rapid prototyping and
experimentation • Open source at KWIVER.org • Web Based Sample Apps
![Page 27: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/27.jpg)
FMV vs. AlexNet data
27
ImageNet “automobile” A.P. Hill “vehicle”
AlexNet: 224x224 chips FMVNet: 96x96 chips
http://image-net.org/synset?wnid=n02958343#
![Page 28: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/28.jpg)
28
Neovision data w/ VIRAT indexing & IQR
Query
(no IQR)
(w/ IQR)
Neovision data ingested into VIRAT framework, using FMVNet FC7 descriptors and motion & saliency detectors.
Query
Data: Neovision CNN: FMVNet Detector: Motion & Saliency Descriptors: CNN FC7 Indexing: VIRAT
![Page 29: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/29.jpg)
29
Neovision in SMQTK
Data: Neovision CNN: AlexNet Detector: Windowing Descriptors: AlexNet Indexing: SMQTK
Query
Results after a few rounds of IQR and a re-query of the database
![Page 30: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/30.jpg)
Neovision in SMQTK
Data: Neovision CNN: AlexNet Detector: Windowing Descriptors: AlexNet Indexing: SMQTK
- Results obtained with IQR but no re-querying of the database. - Several rounds were required to train the model away from cars and windows.
Query
![Page 31: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/31.jpg)
KWIVER.org Kitware Image and Video Exploitation and Retrieval Toolkit An Open Source, production-quality video analytics toolkit
Motion-imagery Aerial Photogrammetry Toolkit
Hierarchical SBA
Homography-Driven Loop-Closure
SBA with frame-to-frame tracking only
SBA with loop-closure
Homography sequence with loop detected
91 Frames 4494 Frames
Social Multimedia Query ToolKit
Streaming FMV
Archive Query
VIBRANT: Video and Image-
Based Retrieval and Analysis Toolkit
31
![Page 32: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/32.jpg)
Summary • Dramatic, disruptive advances in deep learning for
computer vision are fueled by: – Big Data – GPU computation – Open source software
• Various tricks can greatly reduce training data requirements
• The ISR community should rapidly adopt deep learning for sensor exploitation problems
• To learn more, come to Honolulu on July 21-26 for the IEEE/CVF Conference on Computer Vision and Pattern Recognition
32
![Page 33: Big Visual Data, Deep Learning, and Open Source · ParaView. large data analysis & visualization application ... – Big Data – GPU computation – Open source software • Various](https://reader034.fdocuments.us/reader034/viewer/2022042803/5f46aea2f5a0de2cac18077d/html5/thumbnails/33.jpg)
Images &
Video
Recognition by Function
Object Recognition & Matching
Content-based
Retrieval
Event & Activity
Recognition Anomaly Detection
3D Extraction, Super-
resolution & Compression
Detection & Tracking
Human Activity Detection (OSD, CTTSO) and Tracking in Wide-Area Video (AFRL)
Object and Building Recognition by Function (DARPA)
Normalcy Modeling and Anomaly Detection (DARPA PANDA and PerSEAS)
Football Play Recognition (DARPA CARVE)
Complex Event Recognition in Internet Videos (GENIE)
Content-based Video Retrieval
by Actions (DARPA VIRAT)
3D model-based video compression (DARPA) and super-resolved 3D reconstruction (DARPA)
Threat Detection in Video (DARPA)
Wide-Area Motion Imagery Event, Anomaly and Activity Detection (OSD Data to Decisions, DARPA PerSEAS)
33