From Data to Knowledge

M. Weiss, R. Schaefer, L. Paxton, C. Pikas, S. Babin,

S. Simpkins, D. Morrison, J. Holm*, B. Fortner*

•NASA/JPL, Pasadena, CA•NC State University, Raleigh NC

Outline Current Problems: Climate Change & Space Weather – critical Earth

and Space Science areas Need more than data (through e.g. VxOs) to address problems Data must be translated into knowledge -> “actionable” information

for decision makers Elements of the solution: Data & model access, Collaborative

workspace, subject matter expertise, knowledge management, active subgroups

=> The Process can be efficiently aided through the establishment of Virtual Organizations

APL work toward VOs

Note: In this talk VxO = Virtual “x” Observatory; VO = Virtual Organization

The Problem The world is facing problems of a global scale that are very

challenging: Space Weather events can disrupt our increasingly high tech

dependent society Climate Change effects threaten lives, societies, and political

stability These problems require coordinated action from a variety of

agencies and institutions by people who are not experts in space weather or climate change.

A wealth of data and models exist that can be analyzed by experts to translate the data and model results into knowledge.

Bringing together data and models through a unified interface is not enough; we must bring together the community: data providers, subject matter experts; scientists, policy analysts, etc. into a virtual organization to get the appropriate knowledge to the people who need it.

Scientific research

Homeland Security

Military

Agencies: NASA,

NOAA, FAACommercial Air, Ground

UtilitiesInternet,Power

Education

Public

Space Weather: Common Needs

Disparate Communities – Wide Ranging consequences

Space Weather Crosses Scientific Disciplines

Solar Effects Magnetospheric Effects

Solar Energetic Particles

Lower Atmospheric Effects

Understanding Climate Change Similarly mixes Earth Science Disciplines

Climate Change Consequences Are Wide Ranging, Serious, and Urgent

Current CO2 emission rate higher than IPCC “worst case” scenario

Billions of People would experience serious consequences from climate change

Relative vulnerability of coastal deltas as shown by the indicative population potentially displaced by current sea-level trends to 2050 (Extreme = >1 million; High = 1 million to 50,000; Medium = 50,000 to 5,000; following Ericson et al., 2006). Source: IPCC

Governments and Organizations need to re-orient policies and procedures to prepare for this eventuality

UNEP Climate Change Science Compendium

Water quality measurements

Economic planning

Agricultural Productivity

Fisheries Productivity and Management

Traffic management

Wastewater management

Air Quality

Climate Models

Weather Research

Recreational Use

Public Health

Legal Requirements

NGOs

Public Sector

Education

Watershed Models

Economic Models

Commercial Sector

But Tools for decision makers are fractionated and deciders are isolated within their communities

How to Work toward Solving These Problems Goal: Bring together stakeholders, data providers, researchers,

scientists, and policy analysts together in a virtual organization to transform data and models results into knowledge

Two virtual organization concepts being developed at JHU/APL GAIA – Global Assimilation of Information for Action (Climate

Change VO) SWIFTER - Space Weather Informatics, Forecasting, Technology,

and Enabling Research (Space Weather VO). Bring together the elements and people necessary for the VOs

Collaboration and Data Discovery Tools Social Organization of Communities

Ingredients for SWIFTER & GAIA VOs

Social Collaboration Tools Data Access – satellite data, model results, climate records,

ground data (a diverse set including economic and agricultural data for climate change)

Data Manipulation tools – to enable discovery, need general search and visualization capability

Social Collaboration Tools – for knowledge sharing and management (wikis, blogs, workflow sharing, user rating of data sources, etc.)

Social Organization Organize active subgroups on focused topics Conduct a series of focused workshops & seminars On-line forums moderated by identified subject matter experts Social scientists to identify governance issues

SWIFTER VO elements Tools for SWIFTER already available at APL that can be

incorporated: VITMO – Virtual Ionosphere Thermosphere Magnetosphere

Observatory – data and graphic visualization of a range of data SuperDARN, SuperMag, Distributed Advanced Radar Network and

global magnetometer data aggregations visualizable with a common framework (rBrowse)

Experience bringing Research tools into Operational use (R2O) Ionospheric Satellite Sensor Teams (TIMED and DMSP UV

sensors) Blackbook3 data fusion backbone E-conferencing capability (http://workshops.jhuapl.edu/s1/index.

html) Subject matter experts (space weather research faculty) Regular Space Weather Meetings (including APL “SEASONS”

Space Weather conference)

GAIA VO APL Team identified: Analysts, Scientists, Information

Technologists, Information Managers, and Social Scientists. APL Establishing Partnerships with other organizations:

Johns Hopkins Environment, Sustainability, and Health Institute JHU Department of Earth and Planetary Sciences Center for Integrative Environmental Research, U. MD NOAA Climate Program Office

In the planning stage for a series of focused workshops on specific climate change issues to identify the highest priority issues.

Build on experience from VITMO data aggregation and other space weather data discovery tools.

GAIA and SWIFTER Focus Bringing together the community from its disparate organizations Enabling the community to collaborate and share knowledge Guiding the community to identify its high priority issues, metrics,

and its own best solutions. SWIFTER – identifies best data/models based information to make

decisions about Space Weather vulnerable technologies GAIA – gathers information about specific climate change issues to

enable policy makers to understand the coming consequences

Summary Difficult problems facing society

Heavy dependence on Space Weather vulnerable technology as we approach the maximum solar activity period after a long quiet period

Climate Change will impact the world in ways that will change societies in potentially catastrophic ways – we need to start making policies that mitigate those effects now.

APL working to create VOs to address these areas: SWIFTER – Space Weather GAIA – Climate Change Impacts

Both will be enabled by Collaborative web based knowledge sharing /management tools.

Backup

How to Enable Decision Makers With Actionable Information Climate Change Needs: “The federal government should undertake

a national initiative for climate-related decision support.... This initiative should include a service element to support and catalyze processes to inform climate-related decisions and a research element to develop the science of climate response to inform climate-related decisions and to promote systematic improvement of decision support processes and products in all relevant sectors of U.S. society and, indeed, around the world.” – NRC, “Informing decisions in a changing climate” NAS Press.

Space Weather – The explosion in the use of Space Weather vulnerable technology (e.g. GPS, satellite communications, unregulated power grids, etc.) requires a better flow of actionable information so

Create VO to bring together data providers, analysis tools, subject matter experts, and policy makers together in a collaborative, discovery enabled environment.

SWIFTER & GAIA

Research to Operations (R2O) Difficulty transitioning a research product to a reliable operational

product “crossing the valley of death” (see http://www.nap.edu/catalog/9948.html) requires: Understanding the importance and risks of the transition Continuous development of the transition plans Adequate resources Continuous feedback (in both directions) between the R&D and

operational activities; feedback between organizations is especially difficult to facilitate

Difficult enough within a single institution – here, we have to cross institutional and “cultural” boundaries (commercial, military, academic, governmental) where users and researchers do their work

Here, cultural refers to the social organization of an institution.

Knowledge Management Needs – Space Weather Raise awareness of all available nowcast & forecast

products/procedures Capturing the best products and methods to meet specific needs

(best practices) Come to consensus on a uniform set of metrics (e. g., skill

scores) to judge quality of forecasting/nowcasting techniques Come to consensus on the type and number of sensors needed

for reliable space weather forecasts Collect dispersed expertise into an easily accessible place Provide help to make space weather related data (e.g. from

VITMO) and models (e.g. from CCMC) more easily accessible to non-expert users

Provide a forum for communicating new developments in Space Weather forecasting

Knowledge Management Needs – Earth Sciences /Climate Change Raise awareness of all available nowcast & forecast

products/procedures Capturing the best products and methods to meet specific

needs (best practices) Come to consensus on a uniform set of metrics (e. g., skill

scores) to judge quality of forecasting/monitoring techniques Come to consensus on the type and number of sensors needed

for reliable space weather forecasts Collect dispersed expertise into an easily accessible place

Outreach Needs Increase public awareness of modern society’s dependence on

space and the need for space weather. Provide a forum for the public to learn more about space weather Educate public and policy makers about issues, events, and

consequences of space weather and space weather information dissemination

Provide a translation between the needs of users and the capabilities and methods of current and future space weather technologies – why should anyone change what they’re doing now?

How to Address These Needs? Follow the internet paradigm for knowledge management! Build a Virtual Organization :

Establish an On-line Community of Users Provide on-line tools to foster information sharing:

On-line forums Bulletin boards Focused tutorials Virtual meeting rooms On-demand data visualization Users ratings of products

Moderator Tools to focus, summarize, and enable discussions Facilitate user communication about products and methods Become a portal for multiple data sources and models Build with the intention of growing rather than relying on a

priori definition

A Variety of Internet Resources will be enlisted. Tools must be

provided to foster cross disciplinary discussions

What is the need that our VOrg fills?

Provides the glue to connect existing resources to create a Virtual Organization that connects users with knowledge and subject matter experts.

Addresses needs that are unmet by current systems by integrating climate and weather models into a decision support framework.

Users and analysts will be connected through GAIA & SWIFTER.

A key technology need for our community is the establishment of a Virtual Organization

Earth Sciences Needs a “Systems Integrator” GAIA is a generalized open architecture designed to support

cross-cutting decision making Vision: “enabling decisions – getting the right information to

the right people right in time” Access to global climate change data is a challenge for US

government, non-government organizations, and researchers alike. Climate change and its impacts are no longer purely a

“science” problem Data are distributed across many government organizations in

a wide variety of formats Not all data are readily accessible to users because of formats Not all data are known to users who should have access Data may be available – but actionable knowledge may not

The data are “designed” to meet the needs of the particular science community that created that data set

GAIA – “One stop shopping for Earth Science Knowledge and Information” In the following slides the functional requirements of GAIA are

described. Analysts can be thought of as members of the basic and applied

research community Users are government and NGO policy makers as well as members

of the research community. GAIA enables the transition of models and information from

research to operations. GAIA enables decision making. GAIA institutes 3 core elements:

A data access function – GAIA VxO An interaction e-connectivity facility – e-GAIA An interactive visualization library – GAIA ACTION

The GAIA VxO unites a variety of data types through a common interface. The data are held remotely – not at the Virtual Observatory (VxO)

We build on existing efforts by others What we provide is the glue to connect the user to the data We provide the service that allows a novice or skilled user to locate data of

interest. There are no Earth Sciences Virtual Observatories – because of this:

Every user has to discover the relevant data for themselves Each user has to determine how to handle the data files Each user has to develop procedures to open the data files Each user has to determine how to visualize the data

GAIA will a solution for a small, high value focus-area – food security with the test case being the Chesapeake Bay GAIA will provide the means to access and preview data as well as to select and

download data for local use GAIA will enable comparisons between data and models

GAIA will answer: Where are the data? How are they accessed? What format are they in? What do they mean? How good are the data?

Data Access Across VxOs

Geophysical Data (topography, geology, land use, surface composition)

Model output and forecasts

Experiment planning tools and coincidence calculators for space and aircraft

Sensor(s) dataSensor(s) information (incl. availability, location, band coverage, resolution, limiting sensitivity, accuracy)

Analyst

Virtual Observatory – the glue that enables connection to the data

One stop shopping through a simple interface

Work flow – capturing the transformation of data into knowledge

Knowledge transfer as well as information content is the goal Once a tool for producing a given data product has been created -

how do you reproduce that? “Workflow” means reproducing the process of achieving a given

result: What data were accessed? What parameters were supplied? How was the data processed? Where are “golden data sets” produced with this flow?

A workflow can be passed on to a user - RtoO The user can then reproduce the processing of a highly skilled

analyst. The workflow can become an algorithm that supports a user

directly.

GAIA/SWIFTER will leverage APL and IT community investments APL has a rudimentary infrastructure that can be pieced together

to support collaboration: (http://workshops.jhuapl.edu/s1/, wikis, sharepoint, meetingplace, etc.) but they are not integrated.  

Collaborative platforms which provide the means to share information: Hubzero, Drupal, and Blackbook.

Blackbook supports tiered, secure access to data For many users this will be an important factor in any

collaborative environment Supported by US National Intelligence Office Blackbook wiki at http://blackbook.jhuapl.edu

These applications support “work flow” extensions Work flows are the means for capturing subject matter expert

knowledge as a “process” that is repeatable and adaptable The e-GAIA facility enables the users to close the loop with the

analysts.

APL Well Positioned to be Home Institution for Space Weather Virtual Organization (SWIFTER) Space Weather data: build on VITMO (Virtual Ionosphere

Thermosphere Mesosphere Observatory), SuperMAG, and a variety of home grown Space Weather products and visualization tools to expand data access and usage

APL already works with a variety of organizations (NASA, Military, Homeland Security, University (JHU), etc.

Leverage Knowledge Management expertise with APL partner NASA/JPL

Already provides virtual meeting facilities (used for CAWSES) APL has facilities for classified meetings to meet DoD needs APL has in-house space weather expertise in solar physics,

magnetospheric dynamics, and aeronomy who are part of teams for ACE, TIMED, AMPERE, STEREO, RBSP and other well as other sensors UV imagers, in-situ particle detectors.

SWIFTER Summary Space Weather:

common needs from disparate communities

Progress can be facilitated with the creation of a Space Weather Virtual Organization (SWIFTER)

Brings together a set of tools to access data, models, and on-line collaboration tools to enable rapid progress

APL can provide a unique approach to addressing Earth Science issues APL plays a key role as a technical resource for all US government

agencies. APL has the vision to support the assessment of climate and

weather impacts on issues of importance to the US and its interests Climate has impacts across the spectrum from the economy to

defense and international and national issues. The problem is that the approaches, to date, to climate impacts have

been fragmented even within a given agency. GAIA will demonstrate a generalizable approach to transforming

data into knowledge GAIA will do this by picking a well defined test case and delineating

an architecture that will address Earth Science issues within that test case. The environment defined by GAIA can be applied to other

problems.

GAIA – enabling decisions through information access GAIA is a generalized open architecture to support cross-cutting

decision making Vision: “enabling decisions – getting the right information to the

right people right in time” Access to global climate change data is a challenge for US

government, non-government organizations, and researchers alike. Data are distributed across many government organizations in a

wide variety of formats Not all data are readily accessible Not all data are know to users who should have access Data may be available – but actionable knowledge may not

GAIA will address a “test case” – food security that addresses needs across APL stakeholders and ties in other areas in JHU School of Public Health; The Paul H. Nitze School of Advanced

International Studies; Carey Business School All APL Business Areas are touched by the test case.

The Nitrogen Cycle Couples Climate, Weather and Human Activity

There is a large modeling community that is decoupled from policy makers.

The models don’t speak to the public. What does

increased dissolved NH4 mean to a fisherman?

Users may be concerned with short to long term timescale effects and responses

Planners (urban, county, road, waste water) intersect with commercial users and government agencies.

Complexity can obscure the inter-relationships between the various communities making informed decisions difficult.

Groundwater issues illustrate some of the concerns Groundwater contributes more than half (54 percent) of the total annual

flow of streams in the Chesapeake Bay watershed. The Groundwater nitrate load contributes about half (48 percent) of the

total annual nitrogen load of streams entering the Bay. The apparent ages (residence times) of water collected from springs

range from modern (0-4 years) to more than 50 years, with 75 percent of the ages less than 10 years.

The discharge, nitrate load, and residence time of Groundwater vary in the watershed due to differences in combinations of rock type and physiographic province (known as hydrogeomorphic regions), and land use.

Quantifying the discharge, nitrate load, and residence time of Groundwater in the Chesapeake Bay watershed assists in developing an understanding of the movement of nutrients from their sources to streams, and in determining the "lag time" between the implementation of management actions and distinguishable improvement in surface-water quality.

There is no predictive capability and little ability to overlay information

Because data collection efforts are fragmented it is hard to both visualize the data and access the data in a timely fashion. Fixed data products are the only ones

available May not meet the users real needs

No capability to couple different types of data (weather, topography, land use, development).

No ability to test scenarios For example: What is the effect on

turbidity of the bay at a particular location after a particular amount of rainfall over a specific area?

GAIA is intended to bring together different existing pieces and provide the glue to put the puzzle together.

GAIA – “One stop shopping for Earth Science Knowledge and Information” In the following slides the functional requirements of GAIA are

described. Analysts can be thought of as members of the basic and applied

research community Users are government and NGO policy makers as well as members

of the research community. GAIA enables the transition of models and information from

research to operations. GAIA enables decision making. GAIA institutes 3 core elements:

A data access function – GAIA VxO An interaction e-conferencing facility – e-GAIA An interactive visualization library – GAIA ACTION

In the following figures the individual functions and responsibilities of the GAIA community members are described.

Data Analyst Results

The principal function of the scientific community is to take in data and produce a result.This result may be a publication.Often these results are not readily accessible or understood by policy/decision makers.

One of the challenges is that the data come from many sources, in many formats.

Data

Analyst ResultsData

Data

Data

Data

Data

Data

One of the challenges is that the data come from many sources, in many formats.

Data

Analyst ResultsData

DataSemi- or Un-structured Data

Cloud Data

RDBMS

Linked Open Data

Cloud Analytics

GAIA will identify high value data for particular use cases and demonstrate the means to extract relevant data

Data

Analyst ResultsData

DataSemi- or Un-structured Data

Cloud Data

RDBMS

Linked Open Data

Extractors

Ingesters

SQL

Cloud Analytics

A “Virtual Observatory” at APL would capture the data access knowledge

Data

Data

DataSemi- or Un-structured Data

Cloud Data

RDBMS

Linked Open Data

Extractors

Ingesters

SQL

Virtual Observatory Data

The virtual observatory knows where the data are and understands the format of those data.The VxO extracts the relevant data from these sources and delivers them to the analyst.

Data Access Across VxOs

Geophysical Data (topography, geology, land use, surface composition)

Model output and forecasts

Experiment planning tools and coincidence calculators for space and aircraft

Sensor(s) dataSensor(s) information (incl. availability, location, band coverage, resolution, limiting sensitivity, accuracy)

Analyst

Virtual Observatory – the glue that enables connection to the data

One stop shopping through a simple interface

GAIA also hosts tools for the analyst.

Data Analyst ResultsWe can’t anticipate all possible uses of the data. GAIA will build an open architecture that enables analysts to access and visualize the data.Users then contribute tools to a common open source library.

The Analyst uses tools and information from models to produce a result.

Data

Models

Results

Algorithms

Work Flow

Anomaly Identification

Feature Extraction

The Results are made available to Users and archived by GAIA – this feedback is essential to building knowledge

Data

Models

Results

Algorithms

Workflow

Anomaly Identification

Feature Extraction

Work flow – capturing the transformation of data into knowledge Knowledge transfer as well as information content is the goal Once a tool for producing a given data product has been created -

how do you reproduce that? “Workflow” means reproducing the process of achieving a given

result: What data were accessed? What parameters were supplied? How was the data processed? Where are “golden data sets” produced with this flow?

A workflow can be passed on to a user - R2O The user can then reproduce the processing of a highly skilled

analyst. The workflow can become an algorithm that supports a user

directly.

Analysts create results from the data and these results are made available to Users

Data Analyst Results

Users take the results and interact with them to create the knowledge required to make an informed decision

Results Users Inter-action

The User requires a set of tools to organize data and results

ResultsInter-action

Maps

Timeline

Visualization

Google

Entity Relationship

Search

iGoogle

IM

Alert

GAIA would provide the architecture for creating new tools – not all the tools. GAIA would “seed” the process by creating an initial set of tools for a particular “use case”

GAIA would support electronic conferencing to promote the virtual interaction of the analyst and user communities

Analyst UserKnowledge

Discovery

Dissemination

The interaction with the analysts enables the user to address their individual requirements while also providing access to subject matter experts

Analyst User

Evaluate

Test

Hypothesize

Integrate

Observe

GAIA will support the feedback from the user to the analyst community

Analyst User

Bayesian Reasoning

Reasoning by Analogy

Reinforcement Learning

Evidence marshalling

Learning and Reasoning

Adaptive Learning

GAIA will support the interaction of the users with the results and provide feedback to the analyst on how well their results meet user needs.

Results Users Inter-action

Supported by e-GAIA

Virtual Sessions

e-GAIA: bringing users and analysts together e-GAIA provides an electronic

conference facility For the IRAD we will determine what

the specific requirements would be to enable an e-conference on the focus problem and how that could be implemented.

Specific conferences would be held to address focus areas. For example, for a focus on the Chesapeake Bay: Impact of global climate change on

area economy Long term planning Impact of economic growth on

ecosystem Data infrastructure

registration

Conference resources

Focused tutorials

Discussion and Synthesis

Economy

Ecology

Global Impacts

Other topics

Data Access ToolsVisualization

Tools

Model library

On Demand Models

Outcomes of e-conferences will shape future activities e-GAIA will provide the means:

for the participants to pose questions, form new ad hoc focus groups, propose new e-conferences

for the user community to assess the value of models and visualization tools

To focus efforts in a particular direction To establish a (leadership) role for APL in Earth Sciences To grow a vigorous Earth Science program

GAIA ACTION - Actionable Content & Timely Information On the Network ACTION is an open source architecture for implementing access to

information in a customizable user interface. GAIA ACTION will develop a set of basic functions to demonstrate

the utility of the interface Typical modules will provide real-time access to news, databases,

model runs, satellite data and monitoring stations GAIA ACTION provides a cross-agency cross-disciplinary

interface to resources that are already available The information, models and tools are available. GAIA ACTION takes advantage of the existing investment and

leverages that to establish a new business area and to provide a nucleus for a future work

Example Interface to Data, Models and Existing Data Display Tools

Example Interface to Data, Models and Existing Data Display Tools

NOAA Data Tool Google Maps

GAIA Tools

Environmental signature data from remote stations

Trend and climatological data

Developing Infrastructure – Build, Buy or Modify

APL has a rudimentary infrastructure that can be pieced together to support collaboration: (http://workshops.jhuapl.edu, wikis, sharepoint, meetingplace, etc.) but they are not integrated.  

Collaborative platforms which provide the means to share information: Hubzero, Drupal, and Blackbook. These are different than social networking tools like

Facebook, Googlewave, Linkedin, etc To paraphrase George Orwell – “all users are created equal – some

are more equal than others” Blackbook supports tiered, secure access to data For many users this will be an important factor in any

collaborative environment Supported by US National Intelligence Office Blackbook wiki at http://blackbook.jhuapl.edu

Collaborative Platforms Must be Open Source All three of these can provide a collaboration platform that we can

use.   Hubzero and Drupal are more oriented toward the science

community and benefit from scientific community involvement in developing extensions.

All of these are either open source or expected to be made open source.  

All provide an API for extensions Why do we want an open source solution?

Features need to be developed as the community evolves such as on-line data visualization, data quality, model quality, and subject matter expert evaluation.  

The Geophysics Community can take Advantage of these Developments HUBzero was created with NSF funding. It provides a set of tools to run batch jobs (and graphical

“workflows”) from remote users – this could be useful for extracting knowledge from VxOs and modelling centers.  

Drupal is an international effort  There are also “workflow” extensions as in HUBzero.

Google Wave is still only in development, but it has the power of Google behind it.   Google already has many useful on-line tools, like Google docs,

that allow people to collaboratively create and edit MS Office compatible documents through your web browser.  

Google Wave also provides collaborative windows where people can type messages in different languages that are translated in realtime as you type!

GAIA will transform the use of climate knowledge

Development can be facilitated with the creation of a Climate Knowledge Virtual Organization (GAIA) The effort is enabled by concurrent efforts from other

communities within and external to APL and the JHU community.

Data Analyst Action