Approaches to Preservation Storage Technologies
-
Upload
micah-altman -
Category
Technology
-
view
762 -
download
1
description
Transcript of Approaches to Preservation Storage Technologies
Prepared for
MIT Libraries Informatics Program Brown Bag Talk
June 2013
Approaches to Preservation Storage Technologies
Dr. Micah Altman<[email protected]>
Director of Research, MIT Libraries
Approaches to Preservation Storage Technologies 2
DISCLAIMERThese opinions are my own, they are not the opinions of MIT, Brookings, any of the project funders, nor (with the exception of co-authored previously published work) my collaborators
Secondary disclaimer:
“It’s tough to make predictions, especially about the future!”
-- Attributed to Woody Allen, Yogi Berra, Niels Bohr, Vint Cerf, Winston Churchill, Confucius, Disreali [sic], Freeman Dyson, Cecil B. Demille, Albert Einstein, Enrico Fermi, Edgar R.
Fiedler, Bob Fourer, Sam Goldwyn, Allan Lamport, Groucho Marx, Dan Quayle, George Bernard Shaw, Casey Stengel, Will Rogers, M. Taub, Mark Twain, Kerr L. White, etc.
Approaches to Preservation Storage Technologies 3
Collaborators & Co-Conspirators
• Jefferson Bailey, Karen Cariani, Jonathan Crabtree, Michelle Gallinger, Jane Mandelbaum, Nancy McGovern Trevor Owens
• NDSA Coordination Committee & Working Group Chairs
• Research SupportThanks to the Library of Congress, & the
Massachusetts Institute of Technology.
Approaches to Preservation Storage Technologies 4
Related Work• Altman, et. al, 2013. “NDSA Storage Report: Reflections on
National Digital Stewardship Alliance Member Approaches to Preservation Storage Technologies”, Dlib 19 (5/6)
• National Digital Stewardship Alliance, 2013 (Forthcoming), 2014 National Agenda for Digital Stewardship.
• Micah Altman, Jonathan Crabtree (2011) Using the SafeArchive System : TRAC-Based Auditing of LOCKSS, 165-170. In Archiving 2011.
Most reprints available from:informatics.mit.edu
Approaches to Preservation Storage Technologies 5
Simple question?
• If you have 1000 files (bitstreams), and you’d like to have 99.99% chance of accessing them in 20 years. How do you store them?
Approaches to Preservation Storage Technologies 6
Simplistic Answer: Put it in AWS
• Amazon Glacier claims a design reliability of 99.999999999%
• Neat-o !!!!!!!!!! – Longer odds than winning Powerball OR– Getting struck by a lightning, three times OR– (Possibly) eventually finding alien civilization
• But …
Approaches to Preservation Storage Technologies 7
Clarifying Requirements
• What are the units of reliability? - Collection? Object? Bit?
• What is the natural unit of risk? • Is value of information uniform across units?• How many of these do you have?
Approaches to Preservation Storage Technologies 8
Hidden Assumptions• Reliability estimates appear entirely theoretical
– (MTBF + Independence)* enough replicas -> as many 9’s as you like… – No details for estimate provided– No historical reliability statistics provided– No service reliability auditing provided
• Empirical Issues– Storage manufacture hardware MTBF (mean time between failures) does not match
observed error rates in real environments…– Failures across hardware replicas are observed to correlated
• Unmodeled failure modes– software failure
(e.g. a bug in the AWS software for its control backplane might result in permanent loss that would go undetected for a substantial time_
– legal threats (leading to account lock-out — such as this, deletion, or content removal);– institutional threats (such as a change in Amazon’s business model)– Process threats (someone hits the delete button by mistake; forgets to pay the bill; or
AWS rejects the payment)
Approaches to Preservation Storage Technologies 9
Business Risks?• Amazon SLA’s do not incorporate or reflect
“design” reliability claims:– No claim to reliability in SLA’s– Sole recover for breach limited to refund of fees for
periods the service was unavailable– No right to audit logs, or other evidence of reliability
Approaches to Preservation Storage Technologies 10
What practices are leading stewardhip
organizations using?
Approaches to Preservation Storage Technologies 11
Results from the NDSA Bi-Annual Preservation Storage Survey
• 74 institutions surveyed. 58 met selection criteria.– Follow up on non-responders: 100% response rate.– Low rolloff on individual questions– Next round will be > 2x bigger
• Survey Methods– Close ended, with open ended extensions– Selected qualitative followup
• Survey Data– Instrument and data available as open data
Approaches to Preservation Storage Technologies 12
About the NDSA
Approaches to Preservation Storage Technologies 13
Key Findings: What are Current Institutional Practices?
• 90% of respondents are distributing copies of at least part of their content geographically
• 88% of respondents are responsible for their content for an indefinite period of time
• 80% of respondents use some form of fixity checking for their content• 75% of respondents report a strong preference to host and control their own
technical infrastructure for preservation storage• 69% of respondents are considering or currently participating in a distributed
storage cooperative or system (ex. LOCKSS alliance, MetaArchive, Data-PASS)• 64% of respondents are planning to make significant changes in the
technologies in their preservation storage architecture in the next three years• 51% of respondents are considering or already using a cloud storage provider
to keep one or more copies of their content• 48% of respondents are considering, or currently contracting out storage
services to be managed by another organization or company
Approaches to Preservation Storage Technologies 14
How Many Copies
Approaches to Preservation Storage Technologies 15
How Many Copies? …by Role
Approaches to Preservation Storage Technologies 16
Preservation Storage -- New Approaches
Approaches to Preservation Storage Technologies 17
What do organizations want from their preservation systems?
Approaches to Preservation Storage Technologies 18
What are most memory organizations not doing yet?
• Formal cost and valuation models• Auditing&evaluation• Certification• Comprehensive content review
Approaches to Preservation Storage Technologies 19
Plans for future
Approaches to Preservation Storage Technologies 20
Emerging State of the Practice
Approaches to Preservation Storage Technologies 21
Methods for Mitigating Bit-Level Risk
Physical:Media,
Hardware,Environment
Number of copies
Diversification of copies
Formats FileTransforms:compressio
n,encoding, encryption
Fixity Repair
Loca
l St
orag
e
FileSystems:
transforms,deduplicatio
n, redundancy
Repl
icati
on
Verifi
catio
n
Audit
Approaches to Preservation Storage Technologies 22
Emerging State of Practice
• Organizational – Multi Institutional Stewardship– Institutions hold digital assets they wish to preserve,
many unique– Many of these assets are not replicated at all– Even when institutions keep multiple backups offsite,
many single points of failure remain, because replicas are managed by single institution
– Approaches: LOCKSS, Digital Preservation Network, MetaArchive, Data-PASS, Datanet Federation Consortium, Data-ONE
• Technical: Fixity, verification and auditing• Legal: Secession planning, Confidentiality, …
Approaches to Preservation Storage Technologies 23
Future research: What do we need
to know?
Approaches to Preservation Storage Technologies 24
ModelingBit Corruption
Media characteristics
Threat characteristics
Correlations
Logical Scope of Corruption
File Characteristics
File system Characteristics
Probability of Successful Repair
Auditing Frequency
Number of copies
Repair frequency
Corruption
Detection
Repair
Approaches to Preservation Storage Technologies 25
The Risk Problem Restated
Keeping risk of object loss fixed -- what choices minimize $?
“Dual problem” Keeping $ fixed, what choices minimize risk?
Extension
For specific cost functions for loss of object:
Loss(object_i), of all lost objects
What choices minimize:
Total cost= preservation cost+ sum(E(Loss))
risk
cost
Are we there yet?
Approaches to Preservation Storage Technologies 26
Research Directions• Growing the evidence base
– Descriptive inference – patterns of use– Descriptive inference – outcomes– Predictive inference – trend analysis– Causal inference – effectiveness of interventions
• Modes of inquiry– probability-based surveys
(e.g. of information management practice and outcomes) – replicable simulation experiments tied to theoretically grounded models of
information management and risk; – creation of testbeds and test-corpuses which can be used to systematically
compare new practices, tools, and methods; – field experiments, in which randomized interventions are applied and evaluated
in real operational environments.
Bibliography (Selected)
• David S.H. Rosenthal, Thomas S. Robertson, Tom Lipkis, Vicky Reich, Seth Morabito. “Requirements for Digital Preservation Systems: A Bottom-Up Approach”, D-Lib Magazine, vol. 11, no. 11, November 2005.
• Pinheiro, E., Weber, W.D., & Barroso, L. A. (2007). Failure trends in a large disk drive population. In Proceedings of 5th USENIX Conference on File and Storage Technologies.
• Rosenthal, David SH. "Bit preservation: a solved problem?." International Journal of Digital Curation 5.1 (2010): 134-148.
Approaches to Preservation Storage Technologies
27
Questions?
E-mail: [email protected]: micahaltman.comTwitter: @drmaltman
Approaches to Preservation Storage Technologies
28