OCR and the Transformation of the Humanities

Gregory Crane and David BammanTufts University

Bruce RobertsonMount Allison University

John Darlington and Brian FuchsImperial College London

Three basic changes

Three basic changes

1. Transformation of scale of questions

Three basic changes

1. Transformation of scale of questionsBreadth and Depth

Three basic changes

1. Transformation of scale of questionsBreadth and Depth

2. Student researchers and citizen scholars

Three basic changes

1. Transformation of scale of questionsBreadth and Depth

2. Student researchers and citizen scholarsNot enough professors and library professionals

Three basic changes

1. Transformation of scale of questionsBreadth and Depth

2. Student researchers and citizen scholarsNot enough professors and library professionals

3. Globalization of cultural heritage

Three basic changes

1. Transformation of scale of questionsBreadth and Depth

2. Student researchers and citizen scholarsNot enough professors and library professionals

3. Globalization of cultural heritage Not enough expertise in Europe + North America

Towards Dynamic Variorum Editions

Gregory Crane and David BammanTufts University

Bruce RobertsonMount Allison University

John Darlington and Brian FuchsImperial College London

Thanks to …

• Digging into Data Phase 1• National Endowment for the Humanities• JISC (UK)• SSHRC (Canada)• National Science Foundation• Mellon Foundation• Google Digital Humanities• Cantus Foundation• German Research Foundation

The Dynamic Variorum as grand challenge

The Dynamic Variorum as grand challenge

• How do you build self-organizing collections?

What is a variorum?

• Short for cum notis variorum, “with notes of different people”

New Variorum Shakespeare Series

New Variorum Shakespeare Series

New Variorum Shakespeare Series

“New” = 140 years old

New Variorum Shakespeare Series

“New” = 140 years old

“New” vs. 1821 Shakespeare Variorum

Heinsius’ Claudian

Heinsius’ Claudian

Heinsius’ Claudian

NVS 2011

NVS 2011

What was in the 1873 NVS Macbeth?

What was in the 1873 NVS Macbeth?

• Index

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics• Bibliographies

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics• Bibliographies

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics• Bibliographies• Running Text

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics• Bibliographies• Running Text• Multiple Versions

What was in the 1873 NVS Macbeth?

• Index• [Table of contents]• Sources• Adaptations• General Topics• Bibliographies• Running Text• Multiple Versions• Annotations

Brown’s Intermedia c. 1990

The problem…

• Not feasible to summarize scholarship on any major canonical author by manual means

• An issue in 1665 and in 1905 but much worse now…

• How do we generate a Variorum edition from the very large collections that make this such a challenge? How do we make scale an advantage?

Shakespeare as an easy case…

Shakespeare as an easy case…

Shakespeare as an easy case…

c. 500 years of English ….

Greco-Roman World

Greco-Roman World

From Rabat to Kandahar …

c. 100 CE papyrus from Euclid (c. 300 BCE)

http://www.math.ubc.ca/~cass/Euclid/papyrus/papyrus.html

800-1000 CE: Greek into Arabic

Hunayn Ibn Ishaq (809–873), Arabic version of the Prognosticon from the Hippocratic Corpus

http://www.nlm.nih.gov/exhibition/odysseyofknowledge/

c. 1200-1300: Arabic into Latin

Medieval Translation of the Prognosticon from Arabic into Latin

Return of Greek sources c. 1500

This first edition of Dioscorides' Greek text, printed in Venice in 1499 by Aldo Manuzio (ca. 1447–1515)

Status as of October 2011• What do you do with a billion words?

– 2000 years of Latin• How do you integrate data across languages

– Projecting markup over noisy data• How do you trace ideas?

– Detecting changes within and across languages• How do you get the data you need?

– Customizing OCR for a pre-modern language• How do you scale up your services?

– From workflows to Cloud-based design

Disciplines and Speakers

• David Bamman, Tufts University– Computational Linguistics

• Bruce Robertson, Mount Allison University– Digital Classics

• Brian Fuchs, Imperial College London– Software Engineering

1. Computational Linguistics

David BammanTufts University

(Carnegie Mellon University)United States

Overview: Publications– Bamman, David and Gregory Crane (2011), “Measuring Historical

Word Sense Variation,” Proceedings of the 11th ACM/IEEE Joint Conference on Digital Libraries (JCDL 2011). Nominee, Best Paper Award.

– Bamman, David, Alison Babeu, and Gregory Crane (2010), "Transferring Structural Markup Across Translations Using Multilingual Alignment and Projection," in: Proceedings of the 10th ACM/IEEE Joint Conference on Digital Libraries (JCDL 2010). Winner, Best Paper Award.

– Bamman, David and David Smith (forthcoming), “Extracting Two Thousand Years of Latin from a Million Book Library”, Journal of Computing and Cultural Heritage.

2000+ Years of Latin

Goal: Tracking Language Change

• Lexical change (new vocabulary, shifts in word sense)• Syntactic change (including the influence of the author’s L1 on

the Latin syntax)• Topical change (the rise of new genres)• Identifying the spread of variation across authors.

Corpus development

• Data source– 1.2M books from the Internet Archive (snapshot of

collection from 2009)– 25,886 works catalogued as Latin

• Metadata problems1. Language identification (many of these works are not

Latin.)2. Historical date info (dates of publication != dates of

composition.)

25,886 works catalogued as Latin in the IA, charted by “date.”

Language ID

• Language ID to identify which of these works actually have Latin as a major language.– Trained a language classifier (alias-i Lingpipe) on:

• 24 editions of Wikipedia• Perseus classical corpus• Known badly-OCR’d Greek in the IA.

• Results– 10,263 of 25,886 books catalogued as Latin are not recognizably so

(mostly Greek)– 6,790 books not catalogued as Latin in the 1.2M collection are in fact

so (98% precision).– Net: 22,413 Latin books containing 2.97 billion words.

Composition dating

• With undergraduate students in Classics, established dates of composition for each Latin text. So far, considered 10,398 of them:– 7,055 dated– 3,343 excluded as not representative of language use –

e.g., reference works (dictionaries, catalogues, lists of manuscripts)

• From these 7,055 works, we extract just the Latin to create a dated historical corpus of 389 million words.

25,886 works catalogued as Latin in the IA, charted by “date.”

7,055 Latin works in the IA, charted by date of composition.

“America”

(1066)

“de”

(2,955,462)

“oratio”

“lead” vs. “iron”

Polysemy

Lead Iron

(verb) cause to go (verb) to smooth w. an iron

be in command (noun) element Fe

(noun) position of advantage tool with flat steel base used to smooth clothes

chief part in play golf club

element Pb

graphite in pencil

Oratio

(noun) Speech

Prayer

Words have many senses.

Measuring sense variationMethod: Train broad-coverage word sense disambiguation using aligned

parallel texts

English/French (Diab and Resnik 02), English/Chinese (Chan and Ng 05, Ng et al. 03), English/Portuguese (Specia et al. 05), English/Vietnamese (Dinh 02).

Parallel text alignment1. Identify translations (130 English translations manually identified by students from a

representative range of dates)2. Word align Latin text <-> English text (ca. 1.3M words)3. Induce a sense inventory from the alignment

Word sense disambiguation1. Train a WSD classifier on noisily aligned texts2. Automatically classify remaining 387M words3. Track lexical change

WSD via parallel texts

• SMT based on Brown et al (1990)

• Different senses for a word in one language are translated by different words in another.

• “Bank” (English)– financial institution =

French “banque”– side of a river = French

“rive” (e.g., la rive gauche)

(Dynamic Lexicon)

(Bootstrapping multilingual digital libraries)

+

Projecting XML markup across editions and translations (Bamman and Crane 2010)

1. Alignment of the source document with the target document in a cascading process: document -> sentence -> word

2. Projection of XML tags in the source document to the target document in way that exploits the linguistic similarity of the text pair.

2. Parallel text alignment

• Sentence level: Moore’s Bilingual Sentence Aligner (Moore 2002)

– aligns sentences that are 1-1 translations of each other w/ high precision (98.5% on a corpus of 10K English-Hindi sentences)

• Word level: MGIZA++ (Gao and Vogel 2008)

– parallel version of: GIZA++ (Och and Ney 2003) - implementation of IBM Models 1-5.

3. Sense induction

4. WSD Training

Source word oratione (oratio)

Sense label prayer

Training context ad spem pertinent, quae in … dominica continentur

5. WSD Classification• For all words without an aligned translation, use the surrounding context

to determine the most likely sense.

5a. WSD static evaluation

System villa pastor miles scientia oratio Average

5-gram LM 54.8% 69.2% 90.2% 73.7% 61.4% 69.9%

6-gram LM 58.3% 61.5% 91.2% 65.8% 63.8% 68.1%

Bayes 63.5% 62.3% 92.6% 70.2% 48.0% 67.3%

Token Unigram LM

63.5% 62.4% 92.6% 70.2% 48.0% 67.3%

Token Bigram LM 64.3% 62.4% 92.6% 70.2% 48.8% 67.7%

TF/IDF 64.3% 60.7% 82.8% 70.2% 49.6% 65.5%

KNN 64.3% 73.5% 84.4% 63.2% 40.1% 65.1%

MFS Baseline 60.9% 66.7% 92.6% 79.0% 60.6% 72.0%

• Created held-out test set of 105 instances of 5 Latin nouns with known shifts in meaning sampled uniformly from 21 centuries. Evaluated 7 different WSD classifiers + simple baseline of most frequent sense overall (MFS).

5b. WSD time series evaluation

5b. WSD time series evaluation

5b. WSD time series evaluation• Evaluated via mean square error between gold standard time series and

automatically classified one.

System villa pastor miles scientia oratio Average

5-gram LM .056 .034 .052 .044 .137 .065

6-gram LM .053 .053 .052 .022 .022 .040

Bayes .047 .060 .055 .040 .228 .086

Token Unigram LM

.047 .060 .055 .044 .230 .086

Token Bigram LM .047 .060 .055 .044 .230 .087

TF/IDF .037 .050 .049 .040 .189 .073

KNN .101 .028 .054 .039 .248 .094

MFS Baseline .228 .170 .014 .091 .338 .178

“oratio”

6. Tracking lexical change: “oratio”

Acknowledgments• This work was supported by grants from:

– The Digging into Data Challenge ("Towards Dynamic Variorum Editions”)– The National Science Foundation (IIS-910884, "Mining a Million Scanned

Books: Linguistic and Structure Analysis, Fast Expanded Search, and Improved OCR”)

– The National Endowment for the Humanities (PR-50013-08, "The Dynamic Lexicon: Cyberinfrastructure and the Automated Analysis of Historical Languages”)

• Thanks are also due to research assistants Alison Darling, Elise Goodman-Tuchmayer, Daniel Libatique, Lee Marmor, John Owen and Erin Shanahan.

2. Digital Classics

Bruce RobertsonMount Allison University

Canada

Digitizing and Viewing Difficult Texts:Lessons From Ancient Greek

 19th century provides a vast array of editions of Greek text, many still very useful        - Yet they could not be accessed digitally What tools and workflows might help us digitize diverse texts such as these?

What applications can we create to make the resulting OCR data useful to researchers and students?

Diversity of 19th Century Fonts and Layout

Character Classification and the Modern UndergraduatePerforming optical character recognition requires a great deal of 'training' This is perfectly suited to the undergraduate researcher

Ph.D. student asks: "why isn't this part of the beginning Greek curriculum?"

    It introduces students to the beauty and heritage of the typography of their subject    It immediately engages them in a vital research project(True of all languages where learning a new character set is a preliminary skill)

Results

http://www.youtube.com/watch?v=OIjaq7ds2J8

Lessons Learned

Undergraduates provide excellent middle-tier academic labour Shared dictionary data will be fundamental to a cloud-based approach

Include as many languages as possible from the beginning         

Future Work

Continue to improve Greek OCR engine based on 'Gamera' Integrate visualization tools that aid students of the language Implement many dictionaries: English, French, Latin, etc. Integrate other crowd-sourcing opportunities so interested viewers can:    Verify  or correct dubious OCR results    Identify the grammar or syntax of words               

3. Software Engineering

John DarlingtonBrian Fuchs

Imperial College LondonUnited Kingdom

ICL’s role in DVE

• High-throughput infrastructure for– OCR for Greek and Latin– Text-based Feature extraction

• E-Science utility computing infrastructure

• High-level functional interfaces for e-Science.

DVE: Context at SCG

• E-Science Frameworks – Grid– Cloud– Parallel Processing– Functional / Declarative approaches

• Internet services and economics– Healthcare– Music– Mobile Applications– Transport

OCR parallel challenge

• The key to OCR at scale is miminising the need for eyeballs.

• i.e. “ground-truth”-- manual checking against the original.

Rapid OCR using MapReduce +a Cloud IaaS

Infrastructure: State of play

– 6 node static hadoop testbed– 160 node eucalyptus cluster on old

opteron chips– 20 dual quad-core machines with 16TB

storage on fibre. – Stack assembled and deployed– Initial training sets tested.

Throughput Infrastructure

• Boschetti Aligner– OCR post-processing for Greek/Latin

developed at PDL – multiple sequence alignment dynamic

algorithm ( like BLAST, Clustal, Mr. Bayes)

– bayesian classifier to select the most probable sequence of characters

– spell-checking filtered by ocr evidence 

Throughput Infrastructure• MapReduce

– = functional Map/Fold. – Made famous by Google, but developed by others.– Map, then reduce– Map: apply a function in parallel to a bunch of

key/value pairs. – Reduce: apply a function in parallel to each group

of similar k/v pair outputs from Map.

Throughput Infrastructure• MapReduce

– E.g. count occurrences of words in docs– Map( docname, doc.txt))-> ‘mittitur’:1, ‘cura’:1– Reduce(word:count) ‘mittitur’:23, ‘cura’:10,

…

Throughput Infrastructure• MapReduce

– E.g. count occurrences of words in docs– Map:

• Count the words in 1000 documents (in parallel)• map( docname, doc.txt))-> ‘mittitur’:1, ‘cura’:1

– Reduce• Group the output by word, and add up

occurrences (in parallel)• Reduce(word:count) ‘mittitur’:23, ‘cura’:10,…

Throughput Infrastructure• Eucalyptus

– Open Source Cloud Computing – UC, Santa Barbara Spin-off– compatible with Amazon EC2/ S3 – Supported in Ubuntu as of 10.4.

Throughput Infrastructure• Hadoop

– Apache Distributed File System for MapReduce jobs.

– MapReduce Engine—co-ordinates MapReduce

Cluster provisioning

• Create an image with the whole stack• Deploy the image as many times as

nodes are required• Push required config data to the nodes• Turn on• Keep storage separate (i.e. don’t use

hdfs to store data)

OCR parallel methods

• Run parallel jobs on the same scans• Score results• Use highest score in the next round

OCR parallel methods

• 3 different ocr engines per page• x different filters per page• x different filters per section of page.

= c. 30 runs per scan.

OCR vote and error predictionmethods

Courtesy: Federico Boschetti

Alignment voting

• Map: Run three ocr engines/training sets on each page– Gamera– Tesseract: training set 1– Tesseract: training set 2

• Reduce: – spell check and compare

Training set voting

• Map: Run random pages on all avail. training sets.

• Reduce: Check against dictionary, and score.

Tiling

• Map: Run several filters over different parts of a page to compensate for local minima = blotches

• Reduce: score the output and compare.

Why Eucalyptus?

• Scalable– Amazon/NGS hybrid possibilities

• Reuseable– Very fast start-up/tear-down.

• Configurable– Quickly configure custom throughput

clusters

Why MapReduce?

• “Shared Nothing” architecture• = suited to “dumb” processes like page ocr

Why Hadoop? Easy to integrate with other FS’s, e.g. s3 Excellent customisation options Most flexible implementation of MapReduce (cf.

GridGain)

Why not MapReduce?

• Requires extensive refactoring.• Only a subset of functional possibilities.

Why not Hadoop? Filesystem is slooooowwww…. Resource intensive. Headnode is a bottleneck…

Challenges for the future

• Feature Extraction.e.g.– Named Entities– Part of Speech tagging– Multi-lingual alignment

• Iteration is hard with distributed systems!

Conclusions

Three conclusions

• Increased intellectual range

Three conclusions

• Increased intellectual range– Greco-Roman Antiquity is an enabling subject to

understand cultural tectonic forces at work today

Plato’s Republic and the Guardians

The Islamic Republic of Iran and the Guardianship of Islamic Jurists

Sometimes Greek philosophy does have an impact..

Plato’s Republic and the Guardians

The Islamic Republic of Iran and the Guardianship of Islamic Jurists

Three conclusions

• Increased intellectual range– Greco-Roman Antiquity is an enabling subject to

understand cultural tectonic forces at work today

• Cultural heritage -> network of cultures

Three conclusions

• Increased intellectual range– Greco-Roman Antiquity is an enabling subject to

understand cultural tectonic forces at work today

• Cultural heritage -> network of cultures– We share Greco-Roman Cultural Heritage

Students of Greek and Latin

Students of Greek and Latin

Students of Greek and Latin

How do we work together?

Three conclusions

• Increased intellectual range– Greco-Roman Antiquity is an enabling subject to

understand cultural tectonic forces at work today

• Cultural heritage -> network of cultures– We share Greco-Roman Cultural Heritage

• Decentralized Lab Culture in the Humanities

Three conclusions

• Increased intellectual range– Greco-Roman Antiquity is an enabling subject to

understand cultural tectonic forces at work today

• Cultural heritage -> network of cultures– We share Greco-Roman Cultural Heritage

• Decentralized Lab Culture in the Humanities– Even/esp. hard subjects need contributions from

student researchers and citizen scholars

Student Researchers

Tufts

Student Researchers

Tufts

Holy Cross

Student Researchers

TuftsFurman

Holy Cross

Student Researchers

TuftsFurman

Holy CrossHouston

Student Researchers

TuftsFurman

Holy CrossMount Allison

Houston

Huge Open Collections

• Provide the net public with physical access to unprecedented bodies of cultural heritage

• Researchers and automated systems provide initial intellectual access BUT…

• These alone cannot succeed without student researchers and citizen scholars

Three basic changes

1. Transformation of scale of questionsBreadth and Depth

2. Student researchers and citizen scholarsNot enough professors and library professionals

3. Globalization of cultural heritage Not enough expertise in Europe + North America

We can (if we choose) transform our ability to advance the intellectual life of

society

Thank you!