1© Copyright 2013 Pivotal. All rights reserved. 1© Copyright 2013 Pivotal. All rights reserved.

Climate Data Lake: Empowering Citizen Scientists in Acadia National Park

Srivatsan RamanujamPrincipal Data Scientist, Pivotal

@being_bayesian

John CardenteDistinguished Engineer/Lead Data Scientist, EMC

@johncardente

19-Feb-2015

2© Copyright 2013 Pivotal. All rights reserved. 2© Copyright 2013 Pivotal. All rights reserved.

Climate Change is happening

Source: NASA

3© Copyright 2013 Pivotal. All rights reserved.Source: National Geographic

At the Ice Caps…

From Chasing Ice

by James Balog

4© Copyright 2013 Pivotal. All rights reserved.

And in the

Oceans…

5© Copyright 2013 Pivotal. All rights reserved. 5© Copyright 2013 Pivotal. All rights reserved.

Weather is Nature’s clock

6© Copyright 2013 Pivotal. All rights reserved. 6© Copyright 2013 Pivotal. All rights reserved.

And the clock is shifting

Source: EPA

7© Copyright 2013 Pivotal. All rights reserved. 7© Copyright 2013 Pivotal. All rights reserved.

English Oak Winter Moth Pied Flycatcher

Both et al. 2006 Nature

EARLIER SAME TIME

EACH YEAR

Disrupting species interactions

8© Copyright 2013 Pivotal. All rights reserved. 8© Copyright 2013 Pivotal. All rights reserved.

…the study of the timing of recurring plant and animal life-

cycle stages, or phenophases, and their relationship to

environmental conditions.

Phenology

Source: USA National Phenology Network

9© Copyright 2013 Pivotal. All rights reserved. 9© Copyright 2013 Pivotal. All rights reserved.

Acadia National Park

Source: Wikipedia

A key phenology

observation point

10© Copyright 2013 Pivotal. All rights reserved. 10© Copyright 2013 Pivotal. All rights reserved.

Citizen Science, the “Hadoop” of Phenology

11© Copyright 2013 Pivotal. All rights reserved.

Challenges impeding scientific progress

A multitude of disparate data sources

Different approaches to data conditioning

Different analysis platforms and scripts

Inconsistent data treatment distracts the

scientific debate from the findings!

12© Copyright 2013 Pivotal. All rights reserved.

Challenges to encouraging citizen science

Different infrastructures for data collection

Unable to see how contributions help science

Inability to analyze the data and collaborate with scientists

Missing a feedback loop to encourage and

engage citizen scientists

13© Copyright 2013 Pivotal. All rights reserved.

EMC Federation teams up with Earthwatch

14© Copyright 2013 Pivotal. All rights reserved.

Objectives

• Authoritative, curated data repository

• Shared analysis platform and software libraries

• Web portals to

– Show citizen scientists how contributions benefit science

– Present data visualization and exploration tools

– Provide access to data sets for volunteer analysis

– Offer common data collection APIs and interfaces

– Host forums to interact with leading scientists

15© Copyright 2013 Pivotal. All rights reserved.

Climate Data Lake

PL/X

Hadoop++

Data

Data Lake Foundation

Explore

CloudFoundry

Collect

Isilon ECS

16© Copyright 2013 Pivotal. All rights reserved.

The kick-off expedition at Acadia

17© Copyright 2013 Pivotal. All rights reserved.

Post kick-off activities

• Established partnerships with data providers

• Stood up initial Data Lake platform

• Active collaboration across all participants on data

conditioning, analysis, feature selection, and visualization

• Launching an initial web presence in March

18© Copyright 2013 Pivotal. All rights reserved.

What’s next

• Pursuing the “full vision” using Federation Technologies

• Talking with other potential corporate sponsors

• Exploring ways to expand support to other climate and

academic projects

19© Copyright 2013 Pivotal. All rights reserved. 19© Copyright 2013 Pivotal. All rights reserved.

20© Copyright 2013 Pivotal. All rights reserved.

Learn More

• White House Press Release:

Harnessing Climate Data to Boost

Ecosystem & Water Resilience

• Pivotal Blog: Pivotal Data Science

Looks at Climate Change @ Acadia

NP

• EMC Reflections Blog: Big Data vs.

Climate Change

BlogsEMC TV video

21© Copyright 2013 Pivotal. All rights reserved.

The Team

@alyssacaddle1

@stevendozen

@johncardente

@lilewite

@being_bayesian

@maryanngarwood

@emctv

@TheMatthewNolan

@johnnysuire

@sriauggie31

@earthwatch_org

@schoodicinst

@hwschoodic

+ many more (not on twitter ) !