From byte to mind Thinking about Open practices, data sharing and

lifetime, and communication for Climate Scientists

@benlaken www.benlaken.com

What do you think when you hear ‘Open Science’ ?

Community/Network inc. many NPOs to spread technical skills

Responsible data management/archival for long-term data preservation

Data is archived and accessible (not just on request)

Free-to-access journals (no paywall)

Code is open, archived, version controlled, and peer reviewed

Optimised for reproducible work and rapid improvement

Lowers/removes cost barrier of software tools

Accountable peer-review process, and even post-pub review

J.M. Wicherts, M. Bakker, & D. Molenaar: "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results". PLoS ONE, 6(11): e26828, 2011, doi:10.1371/

journal.pone.0026828.

Widespread reluctance to share published research data is often hypothesized to be due to the authors' fear that reanalysis may expose errors in their work or may produce conclusions that contradict their own. However, these hypotheses have not

previously been studied systematically.We related the reluctance to share research data for reanalysis to 1,148 statistically significant results reported in 49

papers published in two major psychology journals. We found the reluctance to share data to be associated with weaker evidence (against the null hypothesis of no effect) and a higher prevalence of apparent errors in the reporting of

statistical results. The unwillingness to share data was particularly clear when reporting errors had a bearing on statistical significance.

Our findings on the basis of psychological papers suggest that statistical results are particularly hard to verify when reanalysis is more likely to lead to contrasting conclusions. This highlights the importance of establishing mandatory data archiving

policies.

“…the unwillingness to share data was most pronounced when the errors concerned statistical significance.”

Immediate value of Open Science: transparency

“…most did not follow proper archiving standards (e.g., by keeping code books and writing meta-data [37]), but simply stored data on their own (current) computer (32%), on CDs/DVDs (18%), or on the shelf (20%). Haphazard data management is documented in a number of scientific fields [37], [38], [39], may result in errors in analyzing and reporting of results, and obviously impedes the sharing of data after results are published.”

Openness tied to the fate of data: Archiving

Archiving strategies are for more than just reliability…

Default academic archival strategy

“Digital information lasts forever—or five years, whichever comes first.” Jeff Rothenberg, 1999, Ensuring the longevity of digital information

Digital information has a life cycle

A baby called Data

…5 yrs…

If your work/data has not ‘reproduced’ itself within a short(ening) timeframe, it will be lost.Due to both ‘lifecycle’ of scientific work

(e.g. see “Attention decay in science” http://arxiv.org/pdf/1503.01881.pdf)and the technological challenges of ensuring reproducibility over the long-term…

A computer program circa 1970

http://celebrating200years.noaa.gov/magazine/tsunami_database/NOAApunchcard.html

1955 Program (and programmer):

62,500 punched cards for SAGE military computer network

(a 5mb program)

http://www.computerhistory.org/revolution/memory-storage/8/326/924

…a larger program…

And what about the data these programs were

using?…

UK Met Office, 1960’s, climate data storage on magnetic tape reels

http://www.sciencephoto.com/media/435868/view

Sizes of ~5 to 140mb

from ~1950s to 1970’s

Causes of death for data: Trapped on obsolete storage media and operating systems. Closed source/propriety file types dependent on unsupported systems.

Or even just long-term degradation on storage media (‘data rot’).

1988

1983

Open Science encourages data preservation/archiving:

“The primary barrier to computational reproducibility in many domains has nothing to do with the technological approaches …, but stems rather from a

reluctance to publish the code used in generating the results in the first place.” Barnes, N. (2010). Publish your computer code: it is good enough. Nature. 467, 7317, 753-753

“Despite extensive evidence to the contrary, many researchers and journals continue to assume that summary descriptions or pseudo-code provide a

sufficient description of computational methods used in data gathering, processing, simulation, visualisation, or analysis.”

Boettiger, C (2014) An introduction to Docker for reproducible research. ArXiV: 1410.0846v1

Like archiving data, coding practices are crucial

NPO to encourage use of reproducible computer skills and Open Science practices among Scientists.

Modern scientists are now essentially software developers

Being asked to review a paper with no code or data attached

Requests for data/code of a paper from corresponding authors partially addressed or just ignored

Failure to reproduce results from a paper due to poor/partial documentation of methods

Published results you don't find credible

Problems reproducing your own past work after some time, due to lost data/no documentation, or obsolescence of hardware software!

Code/models/data are lost with people on short-term contracts moving?

Common symptoms of ‘closed’/traditional science

Fortunately, all these nasty symptoms have a simple cure: adopting Open Science practices!

Wider research communication: philosophy and methodsRelated to OS, the spreading of your ideas to wider audience. This needs some philosophical consideration in Climate Science, as it can get messy…

Pew research centre (2009) http://www.people-press.org/2009/07/09/section-4-scientists-politics-and-religion/

Surveys of both gen. public and Scientists show high approval for Scientists to advocate on ‘political’ issues

Do you think you should be involved in a public discussion?

The public has high levels of ‘trust’ in Scientists

Source IPSOS MORI (2014) https://www.ipsos-mori.com/researchpublications/researcharchive/3504/Politicians-trusted-less-than-estate-agents-bankers-and-journalists.aspx

Consistent question since 1983, asked to UK public, “Would you generally trust them to tell the truth or not?”

You have your own bias, what you consider important. These value judgements influence what you value in life,

pursue scientifically, and how you interpret objective facts. Supporting this unique mix of values is advocacy.

The support of your normative values.Advocacy

What this means:

Connection between Science, Open Science, communication, and advocacy

(Normative value —> a claim about how things ought to be)

Adapted from Gavin Schmidts 2013 Stephen Schneider lecture at AGU

There is an idea that Advocates and Scientists are/ (and should be kept)

distinct.

http://smbc-comics.com/index.php?db=comics&id=2088#comic

See if you agree with the following statements and would publicly support them:

Anthropogenic emissions should be limited by international regulations.

Geoengineering should be seriously considered.

Nuclear and solar power should become the primary source of power generation globally.

Vegetarianism and Entomophagy should be encouraged to ease climate impacts of animal agriculture.

Funding for Scientific Research should be a higher priority.

Developing nations should not be allowed to invest in coal technology.

Scientists should communicate more about what they do and find.

Adapted from Gavin Schmidts 2013 Stephen Schneider lecture at AGU

Public understanding of Science should be encouraged.

A personal case study from your feelings on hot-button issues

(n.b. Entomophagy = practice of eating insects)

Judith Curry (2012), http://judithcurry.com/2012/09/10/too-much-advocacy/

Some typical criticisms“But by advocating policy positions- overtly or by stealth - scientists may be forfeiting their privileged positions as scientists and becoming just ordinary guys with opinions,

and in the process, undercutting the credibility of their scientific work.”

Problems with advocacy

Seen as a threat to scientific objectivity Politicization may damage reputation of scientists/scientific bodies

Potentially seen as abuse of a position of trust Jeopardy to employment/job security

Dichotomy between science telling us what is, and the scientist telling us what to value?

Science doesn’t tell us the answer to our problems. Neither should scientists. Tasmin Edwards, (2013) The Gaurdian, http://blogs.plos.org/models/climate-scientists-must-not-advocate-particular-policies/

Summary of perceived issues:

The previous test should have shown you that not all types of advocacy are viewed equally: most people are happy to

support advocating more funding, but some are not happy to support climate scientists advocating policies.

(Yet, generally, scientists see no problem advocating for: Science, Funding, public understanding, and even themselves…)

Everyone has value positions (consciously and unconsciously), the keys to being a responsible advocate are:

a. Making your value positions conscious b. Making them explicit c. Not letting your value positions distort your subjective priors (we

need a community to do this effectively) d. Defending your value positions separately from arguments of

probabilities and consequences

Some Scientists that that it is impossible to do these things, and instead say they will be neutral, but no one is.

S. Schnider’s rules for responsible advocacy

Steven Schnider (1996)

Gavin Schmidt, (2015), What should climate scientists advocate for?

Bulletin of the Atomic Scientists, doi: 10.1177/0096340214563677

Stephen Schneider (2009) Is the Scientist-Advocate an Oxymoron?.

Diss. Stanford University

“What's the use in having developed science well enough to make predictions if all we are willing to do is sit around and wait for them to come true? ”

More on this topic:

In the Climate Science community many peoples attitudes to these questions are likely summed up by this quote:

Prof. Sherwood Rowland, on his discovery CFCs damage the ozone layer said:

Steps to disseminate your scientific work and values

1. Get a unique researcher ID (many journals now use this)

2. Create profiles on researcher-based social networks

3. Generate some original content from code, presentations, posters, papers and reviews.

4. Spread the word/join conversations, and see many benefits: e.g. feedback, research awareness, citations, wider networks, more

impact, contribute to cultural conversation…