CX4242: Data & Visual Analytics - Visualization€¦ · Shashank Pandit, Duen Horng (Polo) Chau,...
Transcript of CX4242: Data & Visual Analytics - Visualization€¦ · Shashank Pandit, Duen Horng (Polo) Chau,...
CX4242:
Data & Visual Analytics
Mahdi Roozbahani
Lecturer, Computational Science and
Engineering, Georgia Tech
Assignments Overview(Tentative and subject to change)
C X 4242
Assignment 1
Platforms, Languages & TechnologiesPython, Gephi, SQLite, D3, OpenRefine
QuestionsQ1: Collecting and visualizing data (Python & Gephi)
Q2: Analysing data using SQLite
Q3: D3 Warmup
Q4: Analysing data through OpenRefine
Assignment 2
Platforms, Languages & TechnologiesD3, Tableau
QuestionsQ1: Designing a good table and visualizing data with Tableau Q2: Force
directed graph using D3
Q3: Scatter plots using D3 Q4: Heatmap using D3
Q5: Interactive visualization using D3 Q6: Choropleth map
using D3
Q7: Pros and cons of various visualization tools
Assignment 3
Platforms, Languages & TechnologiesJava, Hadoop, Spark, Pig, Azure
QuestionsQ1: Analyzing a graph with Hadoop/Java
Q2: Analyzing a graph with Spark/Scala on Databricks Q3: Analyzing
data with Pig on AWS
Q4: Analyzing a graph using Hadoop on Microsoft Azure Q5:
Regression using Azure ML Studio
Assignment 4
Platforms, Languages & TechnologiesPypy, PageRank, Random Forest, SciKit Learn
QuestionsQ1: Scalable single-machine PageRank
Q2: Implementing a random forest classifier
Q3: Using Scikit-Learn for running various classifiers
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination
Building blocks. Not Rigid “Steps”.
Can skip some
Can go back (two-way street)
• Data types inform visualization design
• Data size informs choice of algorithms
• Visualization motivates more data cleaning
• Visualization challenges algorithm
assumptions
e.g., user finds that results don’t make sense
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination
How “big data” affects the
process?(Hint: almost everything is harder!)
The Vs of big data (3Vs originally, then 7, now 42)
Volume: “billions”, “petabytes” are common
Velocity: think Twitter, fraud detection, etc.
Variety: text (webpages), video (youtube)…
Veracity: uncertainty of data
Variability
Visualization
Value
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination
http://www.ibmbigdatahub.com/infographic/four-vs-big-data
http://dataconomy.com/seven-vs-big-data/
https://tdwi.org/articles/2017/02/08/10-vs-of-big-data.aspx
Three Example Projects from Polo and Mahdi Research group
Apolo Graph Exploration:
Machine Learning + Visualization
18
Apolo: Making Sense of Large Network Data by Combining Rich User Interaction and Machine Learning.
Duen Horng (Polo) Chau, Aniket Kittur, Jason I. Hong, Christos Faloutsos. CHI 2011.
19
Beautiful Hairball
Death Star
Spaghetti
Finding More Relevant Nodes
Apolo uses guilt-by-association
(Belief Propagation)
HCIPaper
Data MiningPaper
Citation network
20
Demo: Mapping the Sensemaking Literature
22
Nodes: 80k papers from Google Scholar (node size: #citation)Edges: 150k citations
Key Ideas (Recap)
Specify exemplars
Find other relevant nodes (BP)
24
What did Apolo go through?
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination
Scrape Google Scholar. No API. 😩
Design inference algorithm (Which nodes to show next?)
Paper, talks, lectures
Interactive visualization you just saw
You will a new Apolo prototype (called Argo)
26
Apolo: Making Sense of Large Network Data by Combining Rich User Interaction and
Machine Learning. Duen Horng (Polo) Chau, Aniket Kittur, Jason I. Hong, Christos Faloutsos.
ACM Conference on Human Factors in Computing Systems (CHI) 2011. May 7-12, 2011.
NetProbe:
Fraud Detection in Online Auction
NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks. Shashank Pandit, Duen Horng (Polo)
Chau, Samuel Wang, Christos Faloutsos. WWW 2007
Find bad sellers (fraudsters) on eBay
who don’t deliver their items
NetProbe: The Problem
Buyer
$$$
Seller
28
Non-delivery fraud is a common auction fraud
source: https://www.fbi.gov/contact-us/field-offices/portland/news/press-releases/fbi-tech-tuesday---building-a-digital-defense-against-auction-fraud
29
NetProbe: Key Ideas
Fraudsters fabricate their reputation by
“trading” with their accomplices
Fake transactions form near bipartite cores
How to detect them?
30
NetProbe: Key Ideas
Use Belief Propagation
31
F A H
Fraudster
Accomplice
Honest
Darker means
more likely
NetProbe: Main Results
33
34
“Belgian Police”
35
What did NetProbe go through?
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination
Scraping (built a “scraper”/“crawler”)
Design detection algorithm
Not released
Paper, talks, lectures
37
NetProbe: A Fast and Scalable System for Fraud Detection in Online Auction Networks. Shashank
Pandit, Duen Horng (Polo) Chau, Samuel Wang, Christos Faloutsos. International Conference on World Wide
Web (WWW) 2007. May 8-12, 2007. Banff, Alberta, Canada. Pages 201-210.
FONT TELLER
Homework 1 (out next week; tasks subject to change)
• Simple “End-to-end” analysis
• Collect data using API
• Store in SQLite database
• Create graph from data
• Analyze, using SQL queries (e.g.,
create graph’s degree distribution)
• Visualize graph using Gephi
• Describe your discoveries
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination