Microform or Digitize: Which is Right For You?

Brad HoustonRecords Manager, University of

Wisconsin-Milwaukee

Reformatting

• v. 1. To create a copy with a format or structure different from the original, especially for preservation or access. 2. To migrate information from one carrier to another.• Source: Richard Pearce-Moses, A Glossary of Archival and Records Terminology,

Society of American Archivists, 2005

– Encompasses microfilming, digitization, photocopying, redevelopment, you name it!

Why reformat your records?

• Save space in the office or records center• Improve access to frequently-requested

records• Move records to format more suitable for

long-term preservation– Alternately: Provide duplication for

redundancy/off-site storage

Goals of this presentation

• Discuss general issues to consider before embarking on reformatting project

• Examine the two main forms of reformatting practical for most municipalities – Digital and Microfilming, of course!

• Describe scenarios in which either, both, or neither type of reformatting is appropriate

PART I: GENERAL CONSIDERATIONS

LegalOperationalAdministrative

Before you start reformatting…

Why is my office thinking about this in the first place?– Space concerns?– Vital Records Protection?– Increased Public Access?– Better Discoverability?– Improved Security?– Do these records NEED to be reformatted?

How are these records used?

Preservation Concerns• Vital records

– Deeds, Birth Certificates, etc.

• Private-use records• Archival Documents

– Policies, reports, minutes

• Infrequently accessed• Unlikely to be

subpoenaed/discovered

Access Concerns• Administrative records

– Notarization, budgets, etc.

• Public-use records• Short-retention documents

– Procedures, expense reports

• Frequently accessed• Increased likelihood of

subpoena/discovery

What are the startup costs?

• Time/Labor needed to reformat– Specialized labor/skills needed for project

• Equipment outlays• Public notices (as needed)• Other buy-in costs (appeals to officials?)

What are the maintenance costs?

• Cost of physical equipment maintenance• Time/labor needed to maintain• Time/labor needed to provide access• Media stability offsets• Is the maintenance cost offset by return on

investment?

What are the security implications?

• Does the information in the reformatted records require protection?

• If so:– Reformatting a net increase or decrease in

security?– Initial security measure outlays– Staff information security training– Secure access to public (as applicable)

What are the legal implications?

• Federal Rules of Civil Procedure Sec. 34 (December 2006)– Discovery applies equally to reformatted records!

• Public Records Laws in various states– Example: Wis. Stat. 19.32: Open Records Act

• Other federal/state disclosure laws– Sarbanes-Oxley, Gramm-Leach-Bliley, etc.– May apply if working with private contractors

Should I reformat at all?

• Do reformatting candidates have high retention periods (>10 years)?

• Is my estimated ROI (in either time or cost) more than the time or cost to reformat?

• Do I have the infrastructure (or can I build it) to adequately support and maintain system?

If no to any of the above, consider postponing reformatting program!

PART II: MICROFILM AND MICROFICHE

An Oldie But a Goodie

The least you need to know: Microfilm

• Preservation: Well-suited• Access: Requires time investment• Initial outlay: Medium equipment/low

training costs• Continued Maintenance: Low Cost• Security Impact: Neutral• Legal impact: Potentially high

First steps: Needs planning

• Active vs. Inactive records for reformatting?• Storage and use copies? One or both?• Need for long-term preservation?• Microfilming from source documents or data

from a computer?• Large or small amount of documents to be

filmed?

Film Selection: Physical Medium

• Diazo Film– Best for use copies; less stable for permanent

preservation• Vesicular Film– Flips master image polarity; good for non-printing

films. Non-ideal for preservation• Silver Halide Film– Best for long-term preservation; easily damaged

with repeated use

Film Selection: Format

• Microfilm roll (16mm or 35mm)– Inexpensive and compact, but cumbersome to

access– Best format for storage of vital or archival

documents

• Microfiche (105mm, rows/columns)– Easier to organize/access, but more expensive to

produce and store– Best format for case files, semi-active records

Initial Outlay: Equipment

• Microfilm Camera– Rotary—best for initial

reformatting project– Planetary—better image

quality, diverse sizes

• Microform reader– Different designs for film

and fiche– some do both, for a price

– Built in printer?

Planetary microfilm camera

Initial Outlay: Document Prep

• Remove documents from folders, remove fasteners, etc.

• William Saffady: 1000 pages/hour (2.5 drawers/hour) as a benchmark– This figure increases for non-standard file sizes

• Do you need a target? Find out before you film!

Reduction Parameters

• Reduction: Number of times a page is reduced through microphotography– 24x reduction for standard-sized documents– 30x reduction for large documents– 48x for COM reduction

• Remember to make sure your microform reader can read the specified reduction!– Variable-zoom lens, as appropriate

Quality Control

• Should be done for each slide to check for:– Legibility (naked eye overview)– Resolution: ability of reader to distinguish fine

detail• Specialized resolution chart

– Density: Contrast between info/non-info areas• Densitometer

• For standards, see: ANSI/AIIM MS23-1998

Systems Maintenance

• Low cost of providing access, training staff/public on use

• Cameras may need to be replaced periodically• Readers may last decades, depending on

complexity– If all else fails, records readable with a light table

and magnifying glass!

Environmental/Security

• On-site records=your responsibility to maintain good preservation environment!– Windowless room if possible– No more than 50% relative humidity– No more than 70 degrees Fahrenheit– Acid-free/chemically stable sleeves (microfiche)

• Same security implications as with paper records– Maintain secure room, locked cabinets, etc. as

needed

Access concerns

• Microfiche may be indexed/organized same way as “regular” case files

• Microfilm: roll format means special index should be created– Consider primary access point of records– Entries should include: reel, frame, secondary access– Provide cross-references as needed– Indicate in index when filming took place, and by

whom the records were filmed

Legal Issues and Records Retention

• Will you be able to produce the documents in a timely manner?

• Uniform Photographic Copies Act (28 USC 1732)

• If records have 10 year retention period or less, off-site paper storage may be more cost-effective

Final Verdict:

Microfilm remains useful for its relative lack of expense, easy reproduction, and extreme longevity (>500 years!). However, the issue of accessibility, particularly with microfilm rolls, must be taken into account when deciding to film records for preservation or access.

PART III: DIGITIZATIONNew Paradigms, New Problems

The Least You Need to Know: Digitization

• Preservation: Potentially problematic• Access: Very good, with the right tools• Initial Outlay: Low equipment/medium

software/high staff training• Continued Maintenance: high level of staff

time/effort• Security Impact: Potentially high• Legal Impact: High, but manageable

Digitization is NOT a panacea!

• Potentially very powerful productivity tool and space saver

• At minimum, the following are required for an effective program:– Content Management System– Digital Preservation Plan– Disaster Recovery Plan

Needs Planning: Digitization

• Active vs. Inactive records– For digital records, this is a function of retrieval

speed

• Public vs. Private access to the records• What are your preservation needs?• File-sharing needs?

Initial Outlay: Hardware

• Scanner, preferably with a document feed– Resolution required depends on type of scan

• Digital Camera/Overhead Scanner• Importing/editing computer• Public Access computer (as needed)• Storage Media

Digital Storage Media

• Good: Removable storage media (CD-ROMs, DVD-ROMs, External Hard Drive)– Good for quick access; low longevity

• Better: Disc Array or Server Array– Medium longevity and access speed; better for

redundancy of data

• Best: Distributed Storage Space– Maximum redundancy; slowest retrieval time

Initial Outlay: Software

• Scanning/Import software– Usually comes with scanner

• Image Manipulation software– For document reformatting, include Optical

Character Recognition (OCR) capability

• Content Management System (CMS)– The most important (and most expensive)

component of digitization management

Content Management Systems

• Should include robust metadata at point of entry:– Subject– Author– Recipient– Date– Other keywords/tags

• Some CMSs extract this automatically– ask your vendor!

Content Management, Records Management

• Consider a CMS with integrated Records Management:– Facilitated indexing– Automated or semi-automated disposition– Improved information security– Legal protection under “Safe Harbor” rule (FRCP

37)

DoD 5015.2 RM standard

• Standards for ERMS– often advertised!• A compliant system will include:– Metadata entry and file plans (including retention

scheduling)– Records declaration and classification– Records protection and/or change auditing– Designation of official ERMS administrator(s)

Imaging Parameters

• Pixels per inch/dots per inch (ppi/dpi)• Capture mode• Bit Depth• File Format• Compression• n.b. Higher values increase image quality

AND file size/storage requirements

PPI/Resolution

• Should be as high as possible, but consider display and space needs!

• To adjust resolution of already-imported image, adjust image size (smaller=sharper)Document Type PPI (master) PPI (access)

Text 400 150

Photographs 400-800 200-300

Maps 600 (400 if oversized) 150-300

Graphics (including arch. Drawings)

600-800 200-400

Capture Mode/Bit Depth

• A measure of the amount of colors able to be displayed in the image

• USUALLY an easy call to make– with some exceptions

Document Type Capture Mode Bit Depth

Text Bitonal 1 bit

B/W photos/drawings Grayscale 8 bit

Color photos/drawings RGB 24-48 bit (access v. master)

Maps Grayscale/RGB 24-48 bit (access v. master)

File Format/Compression

• Higher compression=greater degradation of data over time

• Lossy vs. Lossless compression• Proprietary vs. Non-Proprietary formats

Document Type File Format (Preservation) File Format (Access)

Text TIFF or PDF/A JPEG or PDF

Photographs TIFF or JPEG 2000 JPEG

Graphics TIFF or JPEG 2000 JPEG

Maps TIFF or JPEG 2000 JPEG

What’s in a name?

• Don’t rely on the CMS to find files for you!• File names should be easily-browsable:– Date of imaging or use– Type of file (e.g. dept. acronym?)– Unique identifier (e.g. project name, serial #)

• File naming not a substitute for proper metadata in the CMS

Quality Control: Physical Aspects

• Looking for similar defects as with microfilm QC– Legible images– Correct orientation/resolution– Density, color balance, etc.• Targets and color bars assist with this check

• Some QC defects are digitization-specific:– Correct size, name, file format, etc.– Digital artifacts (lines on screen, etc.)

Quality Control: Content Aspects

• Has the OCR read the content properly?– Usually a cursor drag determines this– Most OCR programs allow for manual corrections

• Is the metadata complete and appropriate?– Dublin Core: search and use metadata– PREMIS: Preservation metadata– Administrative metadata (rights, statutes, etc.)– Structural metadata (how records tie together)

Systems Maintenance

• Maintenance of a digitization system often comprises up to two-thirds of total budget!– Cataloging, description, indexing– Ongoing staff training– Physical data storage– Data migration

• Consider cost-sharing with neighboring municipalities

Access to Digitized Records

• The key upside to digitization! • Public Access Terminal/Web Access/Admin

use only?• Rights management?– Will you allow printing?– Watermarks

• Electronic file structure• Searchability?

Preservation Concerns

• IT department usually keeps data storage secure, environmentally stable

• Data itself may have stability problems!– Software obsolescence– Hardware obsolescence– Data integrity loss– File format obsolescence

Preservation Solutions?

• Non-proprietary formats, lossless compression– Usually implies access copy creation

• REGULAR (5 years/3 version) reformatting:– Migration to new systems/file formats– Emulation of programs to read files on old systems– Encapsulation in XML “wrappers”

• Work with IT to see which is right for your organization

Information Security

• Risk of security breach is MUCH higher because of ease of reproducibility

• Potential Solutions:– Versioning/Logging of imaged documents– Declaration of imaged documents as “records” in

DoD 5015.2 system– Visible or metadata-based watermarks– Embargo on portable storage for imaged

documents

Digitization Legal Issues

• Zubulake Case (2004): If you have the records, you must produce them or pay penalties

• Federal Rules of Civil Procedure– Electronic records are discoverable– Safe Harbor rule DOES NOT APPLY if there is

reasonable expectation of litigation (Broccoli)– Plaintiff may request format for production

Legal Issues: Responses

• Make sure your CMS has a robust search feature– Semantic web /fuzzy logic?

• Be prepared to convert formats as needed• Talk to legal team about Litigation Hold

procedures• Formalize records retention/destruction

schedules and procedures

Final Verdict

Digitization offers organizations excellent options for saving space and making reformatted records accessible to staff and to the public. However, great care must be taken to ensure that preservation, security, and legal issues are addressed before a digitization solution is implemented.

PART IV: CONCLUSIONSummation, scenarios, resources, questions

In Summary:

Microfilm and Microfiche• Excellent preservation

medium• Low-to-medium startup

costs• Low maintenance and staff

training requirements• Potential accessibility issues• High security, reduced legal

response ability

Digitization• Superb accessibility if

managed within CMS• Increased ability to respond

to legal requirements• Medium-to-high startup

costs• High maintenance and staff

training requirements• Security and preservation

issues

Scenarios: Office A

• Reformatting target: Birth Records– Considered a “vital record” by statute– Medium-to-high usage– High security needs– Medium-to-low subpoena likelihood

• What would you suggest?– My answer: Microfilm

Scenarios: Office B

• Reformatting Target: Financial Records– Records Schedule states 7-year retention time– Subject to public access under state records law– Very high use– referred to on daily basis– High likelihood of audit/subpoena– Medium security requirements

• What would you suggest?– My answer: Do not reformat OR digitize as

requested

Scenarios: Office C

• Reformatting Target: Child Welfare Case Files– Records Schedule states 20-year retention time– High use while clients active; low use thereafter– Very high security requirements– Medium-to-high potential for subpoena

• What would you suggest?– My answer: Microfiche case files and/or digitize

Scenarios: Office D

• Reformatting Target: Committee Minutes– Permanent Retention/Archival– Subject to public access under state records law– High use by both public and staff users– Low security requirements– Medium likelihood of audit/subpoena

• What would you suggest?– My answer: Digitize with web access