Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

50
MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007 Digitizing Entomology The Biodiversity Heritage Library @ the Smithsonian Martin R. Kalfatovic Smithsonian Institution Libraries 26 November 2007 A Presentation for the Department of Entomology, National Museum of Natural History, Smithsonian Institution

description

Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian. Martin R. Kalfatovic. National Museum of Natural History, Department of Entomology Staff Meeting. Martin R. Kalfatovic. November 26, 2007. Washington, DC.

Transcript of Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

Page 1: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Digitizing Entomology

The Biodiversity Heritage Library @ the Smithsonian

Martin R. KalfatovicSmithsonian Institution Libraries26 November 2007

A Presentation for the Department of Entomology, National Museum of Natural History, Smithsonian Institution

Page 2: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Yet another physical difficulty is the task of assembling the library and indexes which will enable the student to work under proper conditions…. the beginner must now be prepared to spend liberally, or else must establish himself in an institution where a large library exists; if he work by himself with only a few books, he will have to confine himself to a very narrow specialty indeed.

'The Limitations of Taxonomy' by J.M. Aldrich, Science, April 22, 1927, vol. LXV, no. 1686, p.381

Page 3: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

2003, Telluride. Encyclopedia of Life meeting

February 2005. London. Library and Laboratory: the Marriage of Research, Data and Taxonomic Literature

May 2005. Washington. Ground work for the Biodiversity Heritage Library

June 2006. Washington. Organizational and Technical meeting

August 2006. New York Botanical Garden. BHL Director’s Meeting.

October 2006. St. Louis/San Francisco. Technical meetings

February 2007. Museum of Comparative Zoology. Organizational meeting

May 2007. Encyclopedia of Life and BHL Portal Launch. Washington DC..

BHL Timeline

Page 4: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

American Museum of Natural History (New York)Field Museum (Chicago)Natural History Museum (London)Smithsonian Institution (Washington) Missouri Botanical Garden (St. Louis)New York Botanical Garden (New York)

BHL Members

Page 5: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Royal Botanic Garden, KewBotany Libraries, Harvard UniversityErnst Mayr Library of the Museum of Comparative Zoology, Harvard UniversityMarine Biological Laboratory / Woods Hole Oceanographic Institution

BHL Members

Page 6: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Members

University of Illinois, Urbana-Champaign (contributing member)

Scheme for addition of European and Asian partners under consideration

Additional categories of membership under consideration

Page 7: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Page 8: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Structure of the Encyclopedia of Life

Serine Molecule

Page 9: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Serine Molecule

Synthesis CenterField Museum

BiodiversityHeritageLibrary

SecretariatSmithsonian Education &

OutreachSmithsonian/Harvard

InformaticsMarine Biological

Laboratory & MOBOT

Page 10: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

EOL Species Pages

Built from a variety of new and existing sources

Views available for varying levels of expertise from novice to expert

Page 11: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

EOL Species Pages

Legacy literature a key component of the EOL species pages

Page 12: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Focus: Literature

Page 13: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Focus: Literature

Page 14: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

• Core literature pre-1923: 400,000 (80 million pages)

• All pre-1923: 600-750,000 (120-150 million pages)

• All literature: 1.4-1.6 million (280-320 million pages)

Mass. Zoological and Botanical SurveyReports on the fishes, reptiles and birds of Massachusetts, 1839

BHL Focus: Literature

Page 15: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Selection Tools

Combined Serial list for selection of title to scan to avoid duplication of effort

Mongraphic “de-duping” algorithm

OCLC Collection Analysis

Page 16: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

1.3 million catalogue records 73% are monographs

(remainder are serials at title-level)

63% is English language material

The next most popular language (9%) is German

About 30% of material was published before 1923

BHL Collections

Page 17: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Selection

Marine Biological Laboratory/WHOI Marine monographs General Science

Museum of Comparative Zoology MCZ publications Herpetology monographs

and serials Ichthyology monographs and

serials

Page 18: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Selection University of Illinois

Fieldiana Natural history of Illinois

American Museum of Natural History AMNH publications Ornithology

Natural History Museum NHM publications Major natural history general

serials

Page 19: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Selection Botany Collections

Missouri Botanical Garden, New York Botanical Garden, Harvard Botany Libraries, and Royal Botanic Garden, Kew will cooperatively develop a methodology for botanical publications

Page 20: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Selection Smithsonian Institution

Libraries Smithsonian publications Marine mammals Fishes Selected special collections

materials Entomology collection

Page 21: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

The Internet Archive

• 501(c)(3) organization• Dedicated to “Universal Access to

Human Knowledge”• Founder of the Open Content

Alliance• Provides:

– Mass scanning– Archival storage of files– Image processing– Technology development

Page 22: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Scribe Scanner

• Single Scribe Machine– Custom built by the

Internet Archive– Human operated– 3,500 page per shift per

week

Page 23: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Scanning Centers

Northeast Regional Scanning Center 10 Scribe machines MBL/WHO (scanning) Harvard

New York Public Library 10 Scribe machines AMNH NYBG

Page 24: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Scanning Centers

University of Illinois 2 Scribe machines

Natural History Museum, London 1 Scribe machine

Missouri Botanical Garden Non-Scribe operation

Page 25: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Scanning Centers

Washington, DC 1 Scribe machine at

Smithsonian Libraries Planned 10 Scribe

facility in the Washington area (Spring 2008)

Page 26: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Scanning Stats

2 million plus total pages scanned

250,000 plus from the Natural History Museum, London

150,000 from the MBL/WHOI library

Fieldiana, 15,000 plus pages

Page 27: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Scanning Stats

Smithsonian Libraries 250,000 pages (non-Scribe

scanned, 1996-2007) 50,000 Scribe scanned

pages (since August 2007) Other libraries (non-Scribe)

MOBOT: 780,000 AMNH: 150,000

Page 28: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

But what about ...

Page 29: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Page 30: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Difficult (impossible?) to re-purpose much of the material

Quality of images often questionable

“Frankenbooks” Sketchy / inaccurate

bibliographic data

But what about

Page 31: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Persistent Identifiers Stable URL Handle DOI BICI/SICI ISSN ISBN

http://www.biodiversitylibrary.org

Page 32: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Structural Markup<article>  <title>A BRIEF CONSIDERATION OF

CERTAIN POINTS IN THE MORPHOLOGY OFTHE FAMILY CHALCIDID^E.*.</title>

  <author>L. O. HOWARD.</author>   <volume>1</volume>   <issue>2</issue>   <start_page>65</start_page>   <end_page>86</end_page>   <start_count_page>85</start_count_page>   <end_count_page>106</end_count_page>  

<start_page_image_file>3908800908001101smthrich_0085.djvu</start_page_image_file>

  <end_page_image_file>3908800908001101smthrich_0106.djvu</end_page_image_file>

  </article>

Page 33: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Semantic Markup

GoldenGATEThe intention of the GoldenGATE editor is to build a bridge between NLP components and XML markup of natural language text according to arbitrary XML schemas. It allows the deployment of NLP components to marking up the bodies of literature they were designed for. In this way, it enables transforming the texts into XML content according to an XML schema that was designed to gain maximum benefit from the knowledge provided in them.

Integrated Open Taxonomic Access (INOTAXA)

Page 34: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

10.7 million name strings in NameBank

Uses sophisticated algorithm (TaxonGrab) to locate likely name strings in OCR text

Iterative processing of BHL texts will both increase the number of name strings in NameBank and increase the accuracy of name string recognition

Taxonomic Intelligence

Page 35: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL & Publishers

Page 36: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Permissions

• Seek permissions from copyright holders

• Opt in Copyright Model: The BHL will actively work with professional societies and associations to integrate their publications into the BHL in a way that serves the societies’ missions and goals

• BHL will digitize learned society backfiles and mount them through the BHL Portal at no cost.

• Will provide a set of files to the publishers for reuse as they see fit.

Page 37: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Advantages• Use of the articles will increase

as evidenced by citation upsurge• Long-term management of the

digital assets is provided by the BHL at no cost

• Publishers’ content is embedded in the emerging knowledge ecology that is sweeping biology in this century

• Structural markup of backfiles into conformance with NLM DTD (just starting)

Page 38: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Successes

• Entomological News• Journal of Hymenoptera

Research

• Herpetological Review

• Publications of the San Diego Natural History Museum

• University of Kansas natural history publications

• And more ...

Page 39: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

BHL Portal• Library catalog-like interface

to BHL literature• Enhanced structural

analysis to provide volume/issue/article page access to the literature

• Iterative development based on feedback from user community

• Provide access to two key audiences:–Humans–Machines

Page 40: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Page Delivery

Page 41: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Taxonomic Intelligence

Page 42: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Search Browse

Web 2.0 Features

Page 43: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Discovered Bibliographies

Page 44: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

• Initial grant from the MacArthur and Sloan Foundations (as part of the Encylopedia of Life grant)

• Additional support from parent institutions

• Additional grants being actively pursued by BHL and individual members

Funding & the Future

Page 45: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

• Co-evolving bioinformatics resources produce a rich information ecology:– Consortium for the

Barcoding of Life (CBOL) with gene sequences deposited in GenBank.

– GBIF’s Electronic Catalog of Taxonomic Names

– Hebaria and museum specimen databases

Funding & the Future

Page 46: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Financial Sustainability Strategy

• Quick ramp-up high early costs – development, mass scanning, etc. Drive long-term costs down the asymptote toward zero.

• Derive some long-term costs from the operating budgets of the member institutions. (examples under consideration: acquisitions budget, staff positions, etc.)

• Integrate functions/tasks with wider efforts where appropriate, e.g. mass storage.

• Clear roles for staff who wear multiple hats. Two full-time grant funded positions currently but >15 staff who make substantive contributions.

Funding & the Future

Page 47: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

• The Long Now Strategy– Institutions that are creating the

BHL exist to persist through time. That’s an important part of their business. Use them!

– The future is uncertain, the technology landscape changes, people pass on. So create consortial structures that are low-overhead, flexible, and can respond quickly

Funding & the Future

Page 48: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

In any well-appointed Natural History Library there should be found every book and every edition of every book dealing in the remotest way with the subjects concerned. One never knows wherein one edition differs from or supplements the other and unless these are on the same table at the same time it is not possible to collate them properly.

Charles Davies Sherborn, Epilogue to Index Animalium, March 1922

Page 49: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Thank You

Page 50: Digitizing Entomology: The Biodiversity Heritage Library @ the Smithsonian

MARTIN R. KALFATOVIC :: SMITHSONIAN LIBRARIES :: DEPT. of ENTOMOLOGY :: WASHINGTON, D.C. :: 26 NOVEMBER 2007

Thanks to: Chris Freeland, Missouri Botanical

Garden Tom Garnett, The Biodiversity Heritage

Library Project Suzanne C. Pilsk, Smithsonian

Institution Libraries The staff at the Internet Archive

Images from The Galaxy of Images, Smithsonian

Libraries (www.sil.si.edu/imagegalaxy) Martin R. Kalfatovic Keri Thompson Bernard Scaife

CREDITS