Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... ·...

29
Data to Insight to Change Chris Johnson Scientific Computing and Imaging Institute University of Utah

Transcript of Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... ·...

Page 1: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Data to Insight to Change

Chris JohnsonScientific Computing and Imaging Institute

University of Utah

Page 2: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Sources: Lesk, Berkeley SIMS, Landauer, EMC, TechCrunch, Smart Planet

0.1

1

10

100

1000

1999 2001 2003 2005 2007 2011

all disk storage

all digital info

new digital info/yr

all human documents in 40k Yrs

all spoken words in all lives

amount human minds can store in 1yr

Exab

ytes

(10

)

18

2737

2014

Every two days we create as much data as we did from the beginning of mankind until 2003!

Page 3: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it...Dan Ariely

Big Data

Page 4: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

New Visual Analysis Techniques

Page 5: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging
Page 6: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging
Page 7: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

341 Sections90nm thick sections

~32GB/Section~1000 tiles/section

4096x4096 pixels/tile2.18 nm/Pixel

16.5 TB after processing

Page 8: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Scientific Computing and Imaging Institute, University of Utah

Antony van Leeuwenhoek (1632-1723)

. . . my work, which I've done for a long time, was not pursued in order to gain the praise I now enjoy, but chiefly from a craving after knowledge, which I notice resides in me more than in most other men. And therewithal, whenever I found out anything remarkable, I have thought it my duty to put down my discovery on paper, so that all ingenious people might be informed thereof.

Antony van Leeuwenhoek. Letter of June 12, 1716

Page 9: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Scientific Computing and Imaging Institute, University of Utah

Page 10: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Connectome

Page 11: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

PROBLEM-DRIVEN VISUALIZATION RESEARCH for biological data

- target specific biological problems- close collaboration with biologists- rapid, iterative prototyping- focus on genomic and molecular data

Page 12: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

!"

!#

!$%!$&!$#!$'!()!($!((!(*

+$

+$#

,$

+$#,(

+(

+**

,*

+$*

+*)

,"

-

.

+&

+$)

/

0

+(

+$)

1$

!*)

+(#

)2)) $2))

)2)) $2))

!"# !"#"$3456/5. 5787597 0:.:57-;:4</=

%"&'()*+&"$ %"&,+-$,7/594<>??

$%

@A2)

B(2#

$&

@%2"

B*2'

$'(

@"2%

B*2*

$')

@"2#

B&2A

$'&

@A2"

BA2)

$'*

@A2(

BA2)

$+,

@%2'

B"2*

$+'

@%2*

B*2#

$++

@A2)

B#2$

$+-

@%2(

B#2$

@%2'

B&2)

@"2'

B&2A

@$2&

B(2"

@(2*

B*2'

@"2$

B"2%

@)2*

BA2)

@%2"

B$2'

@A2#

B(2A

@(2#

B*2)

@"2%

B*2"

@A2A

BA2)

@*2&

B$2&

@*2$

B$2(

@$2'

B#2$

.'

.+

.-

.%

./

.(

.)

.&

.*

.',

.''

.'+

.'-

.'%

0123456" 172884798:7;8<01:125="79>6;2586147?"68<5;@0123A1# <"2:5;78=":=5"A @0";5"@ ;>:="<10 8=":41#@

M. Meyer et al., EuroVis 2010. Pathlinesource: Humandestination: Lizard chr1

chr2

chr3

chr4

chr5

chr6

chr7

chr8chr9

chr10

chr11

chr12

chr13

chr14chr15

chr1

6ch

r17

chr1

8ch

r19

chr2

0

chr21

chr22

chrXchrY

chr3

chr1

chr2

chr3

chr4

chr5

chr6

chrachrb

chrcchrdchrfchrgchrh

saturationline

- +

10Mb

chr3

go to:

chr3 chr3

237164 146709664

386455 146850969

orientation:matchinversion

invert

out in

M. Meyer et al., InfoVis 2009.MizBee

!"#$%&'()*+$,-

(.+/

!"#$%&'(.+/

()*+$,-

!"#$%&'(.+/

(012$3

!"#$%&'(.+/

(456$3

.5789+-'(.+/

()*+$,-

!"##$%&'(

)#*%+,-.$/

&:& &:% 3:&&:;

&:<

3:&#=>=?@=(A?=>>BA,C;3="D!EFBA(.+/

[email protected]"=@!I?BA!"#$%&'()*+$,-

&:L,

3:%,3:%,

&:L,

0"%1'#$/GHH!=HG@IFJAH!FMN ?!=G@=(AH!FMN

5//

2+.O0+

85- +0+ P69 P7Q R7 9S 96S 6K1 T2 O--!"#$%&'(

()*+$,-,3U;

()*+$,-,;&,

()*+$,-,;&;

()*+$,-,3<V

()*+$,-,;&L

()*+$,-,3,C

()*+$,-,;,C

()*+$,-,3U,

85- +0+ P69 P7Q R7 9S 96S 6K1 T2 O--

3:&

&:&

&:;V &:,V&:<U

&:UU ;:U%

3:3&

.+7218A05/W+;:C

3:3

M. Meyer et al., InfoVis 2010.MulteeSum

Alignment

clear

class1class2class3class4class5class6class7class8

Site types

100 bp

region1

region2

region3

region4

region5

region1

region2

region3

region4

region5

Regions

InSite

Page 13: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging
Page 14: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

!"#$%&'()*+%,-&./%,0&!*+1/*+"1'(2&*$/"0"1 %3$4

%3$44

%3$444

%3$45

%3$46

%3$5

%3$54

%3$544

%3$5444

%3$6

%3$64%3$644

%3$6444

%3$645

%3$646

%3$65

%3$654

%3$6544

%3$65444

%3$66

%3$664

%3$71

%3$45

%3$8

%3$9

%3$:

%3$;

%3$<

%3$=

%3$>%3$?%3$@

%3$8A

%3$88

%3$89

%3$8:

%3$8;

%3$8<%3$8=

%3$8>

%3$8?

%3$8@

%3$9A

%3$98

!/*#$/*+"1-+1&

B C

8AD.

%3$45

E"(*"'(

%3$45 %3$8?

=9?8; ==A?===

?=>@9 ==9<?=<

"$+&1*/*+"1'F/*%3+1G&$!+"1

(+1G&$*

Page 15: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

!"#$%&'()*+%,-&./%,0&!*+1/*+"1'(2&*$/"0"1 %3$4

%3$44

%3$444

%3$45

%3$46

%3$5

%3$54

%3$544

%3$5444

%3$6

%3$64%3$644

%3$6444

%3$645

%3$646

%3$65

%3$654

%3$6544

%3$65444

%3$66

%3$664

%3$71

%3$45

%3$8

%3$9

%3$:

%3$;

%3$<

%3$=

%3$>%3$?%3$@

%3$8A

%3$88

%3$89

%3$8:

%3$8;

%3$8<%3$8=

%3$8>

%3$8?

%3$8@

%3$9A

%3$98

!/*#$/*+"1-+1&

B C

8AD.

%3$45

E"(*"'(

%3$45 %3$8?

=9?8; ==A?===

?=>@9 ==9<?=<

"$+&1*/*+"1'F/*%3+1G&$!+"1

(+1G&$*

Page 16: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

!"#$%&'()*+%,-&./%,0&!*+1/*+"1'(2&*$/"0"1 %3$4

%3$44

%3$444

%3$45

%3$46

%3$5

%3$54

%3$544

%3$5444

%3$6

%3$64%3$644

%3$6444

%3$645

%3$646

%3$65

%3$654

%3$6544

%3$65444

%3$66

%3$664

%3$71

%3$45

%3$8

%3$9

%3$:

%3$;

%3$<

%3$=

%3$>%3$?%3$@

%3$8A

%3$88

%3$89

%3$8:

%3$8;

%3$8<%3$8=

%3$8>

%3$8?

%3$8@

%3$9A

%3$98

!/*#$/*+"1-+1&

B C

8AD.

%3$45

E"(*"'(

%3$45 %3$8

@AAA9= <:>8?A9

88?A=?= <=8?A=9

"$+&1*/*+"1'F/*%3+1G&$!+"1

(+1G&$*

"#* +1

Genome-wide synteny through highly sensitive sequence alignment: SatsumaM. Grabherr, et al. Bioinformatics (2010) 26 (9): 1145-1151.

Page 17: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Data Science Programs

• http://analytics.ncsu.edu/?page_id=4184

• 19 MS programs in Data Analytics

• 8 MS programs in Data Science

• 28 MS programs in Business Analytics

• Several additional “tracks” or “concentration” programs

Page 18: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Big Data Curriculum

Analytics Electives:

• Data Mining (required)

• Machine Learning (required)

• Visualization (required)

• Artificial Intelligence.

Decision making under uncertainty.

• Natural Language Processing.

Understanding textual data and language.

• Probabilistic Modeling.

Advanced statistical techniques and tools (using R).

• Image Processing.

Analysis and learning on image data.

Page 19: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Big Data Curriculum

Algorithmics Electives:

• Advanced Algorithms (required)

• Models of Computation for Big Data.How algorithmic bottlenecks change as data becomes very large;Relation to modern big data systems (e.g. MapReduce).

• Computational Geometry.Geometric interpretation of big data analysis and computation.

• Computational Topology.Topological data analysis and algorithms.

Page 20: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Big Data Curriculum

Management Electives:

• Database Systems (required)

• Parallel Programming for Many-Core Architectures.Parallel Computing and High Performance Computing.Scalable programming on GPUs, many-cores, and HPC clusters.

• Advanced Computer Networks.Large-scale network protocols, architectures, and applications.

• Network Security.Message integrity, access control, authentication, confidentiality.

Page 21: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Piloting in Adobe (Lehi)

• Starting Fall 2014.

• Live 2-way streaming. Interaction across video.

Fall 2014: Visualization: T-Th 9:10 - 10:30am

Fall 2014: Adv. Algorithms: T-Th 10:45 - 12:05am

(plan for early evening, e.g. Data Mining M-W 5:15-6:35pm)

• Potential for Instructor on site in future.

Adobe lecture room open to others.

Page 22: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

The SCI Institute

Page 23: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Productivity Machines

Page 24: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Acknowledgments NIH/NIGMS Center for IntegrativeBiomedical Computing

Center&for&Extreme&Data&Management,Analysis,&and&Visualiza:on

CEDMAV

Utah&Center&for&Neuroimage&Analysis

Scalable&Data&Management,&Analysisand&Visualiza:on

NIH NAMIC

IAMCSInstitute for Applied Mathematicsand Computational Science

Page 25: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Scientific Computing and Imaging Institute, University of Utah

More Information

www.sci.utah.edu

[email protected]

Page 26: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Ecosystem Challenges Around Data Use

Chris JohnsonScientific Computing and Imaging Institute

University of Utah

Page 27: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Sources: Lesk, Berkeley SIMS, Landauer, EMC, TechCrunch, Smart Planet

0.1

1

10

100

1000

1999 2001 2003 2005 2007 2011

all disk storage

all digital info

new digital info/yr

all human documents in 40k Yrs

all spoken words in all lives

amount human minds can store in 1yr

Exab

ytes

(10

)

18

2737

2014

Every two days we create as much data as we did from the beginning of mankind until 2003!

Page 28: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it...Dan Ariely

Big Data

Page 29: Data to Insight to Change - CRA Homearchive2.cra.org/uploads/documents/events/snowbird/2014... · 2014-07-23 · Data to Insight to Change Chris Johnson ScientiÞc Computing and Imaging

Leonid Zhukov - Director of Data Science, Ancestry.com

Vance Checketts - VP and GM of EMC

Edison Ting, Solutions Architect, Pivotal

Panelists