Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State...
-
Upload
janice-hufford -
Category
Documents
-
view
218 -
download
2
Transcript of Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State...
![Page 1: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/1.jpg)
Getting “Digi” with itat the Digital Archives
Adam Jansen
Deputy State Archivist
Washington State Archives
![Page 2: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/2.jpg)
What is ‘Archiving’ in the Electronic Age?
Protecting machine readable records of enduring legal, historical or fiscal value from loss, alteration, deterioration and technological obsolescence in a environment independent from that which produced the record.
![Page 3: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/3.jpg)
Mission of the Digital Archives
• Collect electronic records of enduring legal, historical or fiscal value
• Maintain these records in perpetuity in a useable state for the good of the public
• Make records that are discloseable accessible to the public
![Page 4: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/4.jpg)
Public Records
As defined in RCW 40.14
ANY records that have been made by or received by any agency of the state of Washington in connection with the transaction of public business
![Page 5: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/5.jpg)
Redefining Public• Avg over 650 researchers per day• Avg length of stay over 6 minutes• 6% .gov - 4% .edu - 1% .org• 13% came from Internet Search (Google,
MSN, Yahoo)• Researchers from 131 foreign countriesResearchers from:Canada, US Military, Romania, Germany, France, Australia, Japan, UK,
Netherlands, Russia, Thailand, Portugal, Belgium, Poland, Italy, Indonesia, Singapore, Sweden, Mexico, New Zealand, Czech Republic, Hungary, Brazil, Norway, Columbia, Austria, Greece, Bulgaria, China, Yugoslavia, Philippines, Spain, South Korea, Denmark, Oman, Pakistan, South Africa, Jamaica, Switzerland
![Page 6: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/6.jpg)
Records and Informationor, Why we do what we do
If - Information is power…
And - Records are storage of information
Then – Records must be preserved for future generations
Why?
The foundation of democracy in America is government accountability to the people
![Page 7: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/7.jpg)
What are the challenges (or why is it so hard!?!?!?)
• Socio-political– Resistance to change– Inability to keep pace
• Technology– Ever upwards and onwards– Little thought on looking back
![Page 8: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/8.jpg)
![Page 9: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/9.jpg)
New Federal Mandates to Manage Certain Electronic Records
As electronic records become more integrated into society, producers of those records will be held to higher standards of conduct
• Health Insurance Portability & Accountability Act of 1996 (HIPAA)
• Gramm-Leach-Billey Act of 1999
• Patriot Act of 2001
• Sarbanes-Oxley Act of 2002
• Help America Vote Act of 2002 (HAVA)
• More mandates to come
Records must be managed and destroyed methodically in normal course of business
![Page 10: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/10.jpg)
Shifting Media• Historically records were stored on paper, kept in filing
cabinets– When the cabinet was full, records sent to file room
• Now records stored electronically on computers– When the computer is ‘full’ – add more hard drives
Basic skills to manage and maintain records has been lost, replaced by infinite storage
![Page 11: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/11.jpg)
So the question becomes… who takes care of the
records, and do they have the knowledge?
![Page 12: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/12.jpg)
Why a Digital Archives?• Comply with statutory & regulatory mandates.
– The Law requires preservation of certain public records – it doesn’t
specify whether those records are paper or electronic. All records
must be given the same care.
• Avoid loss of legal & historical records
– As technology changes, the older media (5 ¼” floppy disks, for
instance) become harder to read.
• Preserve rare and ‘at-risk’ paper records
• Centralize Records
– Centralization means uniformity in maintenance
– ‘Trained professionals’ serve as caretakers
• Improved access for citizens
– By centralizing historical electronic records in one location, ‘one-stop
shopping’ will provide the information quicker and easier
![Page 13: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/13.jpg)
What the Digital Archives is not
• Not mass storage for active business applications & data
• Not remote back-up for state & local government networks & data
![Page 14: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/14.jpg)
The Digital Archives will:
• Preserve electronic records with long-term legal, historical and/or fiscal significance
• Assure platform-neutral retrieval 50, 100, or more years from now
• Provide security back-up of certain permanent electronic legal records (courts, vital records, land records, etc.)
![Page 15: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/15.jpg)
8 Requirements for Preservation• Readable• Retrievable• Intelligible• Encapsulated• Reconstructible• Identifiable• Understandable• Authentic
* From Authentic Electronic Records by Charles Dollar
}Hardware
} File Format
} Content Management
![Page 16: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/16.jpg)
Hardware• File Room of the 21st century• Capacity and Speed double every 18 months• Many choices
– Tape– Optical– Hard Drives
First Immutable Law of Digital Archiving
“What hardware you use today will be obsolete within four years”
![Page 17: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/17.jpg)
Archival Software Formats
• Native• ASCII• TIF• PDF/A• XML
Whenever possible seek the open standard solution!
Remember WordStar and DBase II ???
![Page 18: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/18.jpg)
Content Management• Essential to maintain control of the information
explosion• Allows hard coded rules and information
exchange• BUT still requires a strong knowledge,
understanding and implementation of basic records management
Second Immutable Law of Digital Archiving:
“Data is Data, a Record is a Record, It is the content that drives retention, not the media”
![Page 19: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/19.jpg)
The Digital Archives Experience
![Page 20: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/20.jpg)
Standards Driven
• Open Archival Information System – ISO Standard for electronic records archiving
• DOD 5015.2 – ISO Standard for Records Management Applications
• InterPARES – International effort to define requirements for e-archiving
![Page 21: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/21.jpg)
Protection from ObsolescenceDigital Archives Multi-pronged approach:
Stored as BLOBs in DB with metadata:• Maintain native format• Create open file format version• Render XML formatted version, wrapped
• Acquire original hardware and software
![Page 22: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/22.jpg)
Ingestion Process• MUST be flexible
– No Mandate and 3300 agencies
• Microsoft BizTalk 2004
• Transforms, adds metadata based on business rules
• Creates ‘deep storage’ copy wrapping original file in XML, with Hash
• Creates ‘web’ version of original file
![Page 23: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/23.jpg)
Data Ingestion
• How we use it– Design XML/Flat-file schemas for all incoming
data– Use “Maps” to convert from external formats
to internal formats– Build Orchestrations to move the data from
the data files to the database– Image conversion– Generate Deep Storage XML file
![Page 24: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/24.jpg)
fnamefirstname
Fst_namefirst
06/07/05
Jun-07-0507-Jun-05
06/07/2005
Predefined
Pipelines
First_Name
06/07/2005
![Page 25: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/25.jpg)
Deep Storage XML SchemaRecord Common
•Who
•What
•When
•Where
•Original File
•‘web’ file
•Security
•Fixity
Vital Records• Type
Birth• Date of
• Father, Mother
• Hospital
![Page 26: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/26.jpg)
Data Security
• Encrypted SSH FTP transmission
• Issue Digital Certificate
• Verify IP and computer information
• MD5 Hash on all original files
• Copy of FTP on tape prior to ingestion
• DB backups on tape
• Record Level Security for confidential Info
![Page 27: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/27.jpg)
Record Level Security• Restrict records at item, field or series
level
• Restrict to individual, dept, office or global
• Uses authenticated login to reveal fields
• Anonymous users see ‘Restricted’
![Page 28: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/28.jpg)
Digital Archives New Projects
![Page 29: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/29.jpg)
Capturing the Web
• Web pages are how we ‘do business’
• Universally accessible to public, 24x7
• Information repository
• Captures history, business of agency
• Important to ‘archive’ news, forms
![Page 30: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/30.jpg)
Web Archiving• Custom Built Solution• Multiple streams, Assist with Archiving• Stores all web content in database, full text
searchable• Allows predefining of internal fragments, levels,
maximum file size, secure authentication• Web Services allows use of current architecture
for retrieval
• Cannot capture ‘deep web’ content
![Page 31: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/31.jpg)
Email Archiving
• Permanent, executive level correspondence
• Sent as .pst, .msg
• Store ALL email, even the ‘junk’
• Transfer from proprietary into database
• Full text search
• Attachments stored separately, migratable
![Page 32: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/32.jpg)
Maps and Photos
• Stores oversized maps and high resolution photos
• Converts images to compressed format for viewing over the web
• Provides thumbnails for searching
• Uses LoC metadata indexing standards
• Search on title, description
• E-commerce to order photo-reproductions
![Page 33: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/33.jpg)
Third Immutable Law
“Anything that you do today, will need major overhaul in two years”
Technology and industry changing at unprecedented rates… But, more records are ‘lost’ every day!
– Key is to be flexible and attack with forethought
![Page 34: Getting Digi with it at the Digital Archives Adam Jansen Deputy State Archivist Washington State Archives.](https://reader036.fdocuments.us/reader036/viewer/2022062417/5517aa8e5503463e368b5dbc/html5/thumbnails/34.jpg)
Digital Archives @ Eastern Washington University, Cheney, Washington
Questions?Questions?
Adam JansenAdam JansenDeputy State ArchivistDeputy State [email protected]@secstate.wa.gov