BHL Technology Overview

© 2008 Biodiversity Heritage Library www.biodiversitylibrary.org

Biodiversity Heritage Library (BHL):Technology Overview

Chris FreelandDirector, Bioinformatics

Missouri Botanical Garden

Technical DirectorBiodiversity Heritage [email protected]

www.biodiversitylibrary.org

mailto:[email protected]

http://www.biodiversitylibrary.org/


BHL Partners

Museums– American Museum of

Natural History (New York)

– Natural History Museum (London)

– Smithsonian Institution (Washington)

– The Field Museum (Chicago)

Botanical Gardens– Missouri Botanical Garden– New York Botanical Garden– Royal Botanic Garden, Kew

University Libraries– Botany Libraries, Harvard University– Ernst Meyer Library of the Museum

of Comparative Zoology, Harvard University

– University of Illinois

Bioinformatics Institutes – MBL/WHOI– uBio.org


Why have BHL?In any well-appointed Natural History Library there should be found every book and every edition of every book dealing in the remotest way with the subjects concerned. One never knows wherein one edition differs from or supplements the other and unless these are on the same table at the same time it is not possible to collate them properly. Moreover for accurate work it is necessary for the student to verify every reference he may find; it is not enough to copy from a previous author; he must verify each reference itself from the original.

Charles Davies Sherborn, Epilogue to Index Animalium, March 1922

Charles Davies Sherborn (1861-1942)


Unique Components of BHL

• Combining metadata records from multiple libraries (similar, but different) and representing through a shared portal

• Use of JPEG2000• Web 2.0 Mashups• Taxonomic data mining• Services• Rare & novel content


Scanning process

1. Select Book2. Pull from Shelf3. Send to IA scanning center4. Book is scanned & QA5. Page images loaded on IA cluster

1. Derivatives created

6. Book returned to library7. Files harvested from IA portal8. Books available for display within BHL portal

Mushrooms of America, edible and poisonous. Ed. by Julius A. Palmer, Jr. , 1885.

http://www.biodiversitylibrary.org/bibliography/1652


Scan & Store: Internet Archive

Scanning on Scribes

Storage in Petaboxes


Scanning & Derivatives

• XML• JP2

• PDF• JPG• TXT• DJVu

Master Derivatives


Harvest from IA

Extract, Transform, Load (ETL)

• Custom scripts to extract content via IA’s APIs

• Database scripts to transform to relational data structure

• Load into database


Stable URL

Attribution

Name Finding

Page Turning Page TurningZoom/Pan

Download/View

Browse

Search

Filter

Target/Object


JPEG2000 (*.jp2) display

• RAW original => 85% .jp2

• LuraTech encoder– Wavelet compression

• LizardTech decoder– Tiled on the fly,

cached for performance

• GSIV browser-based client viewer– ‘AJAXian’

LizardTech ExpressServer

Browser GSIV.js


.jp2

.jpg

IA

/page/1274907

pageid: 1274907

BHLdb

http://www.archive.org/download/mushroomsofameri00palm/.../mushroomsofameri00palm_0010.jp2

images.mobot.org

A user requests Mushrooms of America, edible and poisonous, Plate X:http://www.biodiversitylibrary.org/page/1274907

locate:

BHL/IA architecture

= 5.0+ sec transfer

Time to deliver image: 8+ sec


Reuse, don’t rebuild

TIF Image from ScannerConverted to text via PrimeOCRName finding via TaxonFinder Extract namesSubmit to NameBankSOAP response

Name Finding in action

with Taxonomic Intelligence…


Names data mining


Tag cloud from LCSHSubject Heading from library catalog

Expressed as MARCXML

Tag Cloud


Geocoding LCSH


RSS Feeds

Specific: Last 25 books published in German from NYBGRSS Feed location: http://www.biodiversitylibrary.org/RecentRss/25/GER/NYBG

1. Allgemeine deutsche Garten-Zeitung, 7, 1829 (added: 04/03/2008 ) 2. Zeitschrift fr wissenschaftliche Mikroskopie und fr mikroskopische

Technik. 2, 1885 (added: 03/28/2008 ) 3. Zeitschrift fr technische Biologie. 7, 1919 (added: 03/27/2008 ) 4. …

General: Last 25 books from all librariesRSS Feed location: http://www.biodiversitylibrary.org/RecentRss/25

1. Summa plantarum : v.1 (added: 05/01/2008 ) 2. Vegetable materia medica of the United States (added: 04/30/2008 ) 3. The family herbal; (added: 04/30/2008 ) 4. …


Services

• Names– v.1 released

http://www.biodiversitylibrary.org/services/name/NameService.asmx

• Stable urls– http://www.biodiversitylibrary.org/bibliography/1652– http://www.biodiversitylibrary.org/name/Carcharodon_carcharias

• Future:– Citation Resolver– Titles Resolver


BHL Name Serviceshttp://www.biodiversitylibrary.org/services/name/NameService.asmx


Provider Integration

• Encyclopedia of Life

• Atrium Andes Biodiversity

• Wikipedia

• EDIT Scratchpads

• More to come…


Hardware Infrastructure

• Distributed

• Partially redundant– Work needed

• Mixed platforms

• Mixed app frameworks


MOBOT

Petabox cluster

Internet Archive


File Storage Estimates

• 4MB per page including derivatives

• 1 million pages = 4TB storage

• Expected output:60 – 100 million pages

240 - 400 TB for files

10 - 20 GB for db


Future Work

• Services– Citation Resolver– Titles Resolver

• Interfaces

• Editing– Authoritative– Community

• Backend


Fedora

• Funded by Gordon and Betty Moore Foundation to adopt Fedora Commons

• Working with Internet Archive to define use and practice

• Project completionDecember 2009


Thank You

Chris Freeland

[email protected]

BHL Portal


BHL Blog

biodiversitylibrary.blogspot.com

BHL collection at Internet Archive

www.archive.org/details/biodiversity

BHL Technology Overview

Technology

Transcript of BHL Technology Overview