Audiovisual collections, the spoken word and user needs of scholars in the Humanities

Observations based on related work in The Netherlands 2005-

2012

E-research

• New and/or rapid ways to gain knowledge• Digital resources and information technology• Big data & data mining (social sciences)• Digital Humanities / E-Humanities• Digitization, Infra, Tools, Standards• CLARIN.eu / DARIAH.eu

Emerging focus on audiovisual

• Multi-modal, multi-semiotic: – multiple layers of meaning / interpretation– E.g., “quote + intonation + images + discourse”

• New dimensions for scholarly research• Large investments in digitization:

– Images for the Future: 200k hours of film, video and audio

– Various digitization projects for scientific collections

?METADATARULES

Metadata & Annotations

• Annotations:– General (document level)– Specific (segment level)

• Metadata: typically sparse / document level• Requirements dependent on research field• Annotation generation:

– Manual (Individual, Teams, Crowd) – Automatic: (un/lightly) supervised

Monitoring radio transcriptions

INGEST SUPERVISION // ARCHIVIST SUPPORT:

Quickly assess quality of ASR

Spoken word search 2005-2012

• Wide range of projects in various domains– Radio

• Daily ingest: selection of programs• Woord.nl: public access to radio content

– Historical video collections with sparse data– ``Oral History’’

• Development of an ASR service for cultural heritage institutions

1st experiment on ASR for humanities: access to personal recordings of Dutch novelist WF Hermans

Access to interview collection with camp survivors World War II

FEMINIST MOVEMENTAccess to interview collections

INTERVIEWS ON BOMBARDEMENT OF ROTTERDAM

Alignment of transcripts for indexing

Access to Radio interviews Experiments with various types of access and result presentation: speaker changes, speaking rate, search strategies, word clouds

Access to Historical Speeches:Alignment & Linking

ACCESS TO DISTRIBUTED ORAL HISTORY COLLECTIONS

• Infrastructure for searching collections at various institutes in The Netherlands

• Harvesting of Metadata (OAI-PMH)

• ASR as a service• Evaluated with

Oral Historians

Observations on speech search

• Large variation in ASR performance• Performance (and decisions on use)

should be assessed in context of application: audiovisual search

• Usefulness in audiovisual search should be assessed in context of use scenarios

• Use scenarios require specific presentation/visualization requests

Usefulness of results• Perception of usefulness

– Usefulness in context of search/data exploration– Educate / Expectation management– Guide searching – Show why (errors, confidence, trust-levels, cut-offs)– Focus on research needs

• Improve on ASR quality– Educate: how to record an interview (Oral History)– Use available textual resources (alignment, vocab optimization)

• Improve on search application– Visualization– Result presentation

• documents versus segments• combination of information sources• cross/within-collection linking

Methodology (1)• E-research is an intervention in current practices!• Promise:

– increased efficiency, relevance, novelty• Interest of scholars:

– tools that facilitate or simplify existing practice (RIN report, 2011)• Co-development ICT-researchers & scholars to adjust

expectations. Examples:– Finding more in less time may not be a goal in itself for humanities

researchers– Deep engagement with primary texts versus results on the

segment level

Methodology (2)• 4 stages:

1. Preliminary archival search• Browsing as a general interest• Purpose driven (checking details, complementary resources)• Item-oriented (finding first mentioning of something)• Collection-oriented (thematic, source, person, event)

2. Content analysis• Visualization, compression, aggregation• (optionally) go back to (1)

3. Presentation and dissemination• Enhanced publications (persistent identifiers on segment level)

4. Curation• Trusted digital repository

• (spoken) search scenarios: facilitate these stages

ASR for research• Triple-A: Accessible, Affordable, Accurate• Individual researchers sending files to ASR?• Embedded in suite of research tools?• What about integration in search applications?

– Stagnation due to inadequate local infrastructures• Variation across collections requires ‘tailor-

made’ approaches: e.g., speaker adaptation, vocabulary adaptation, alignment, collection of related resources (information trail)

ASR service

Model of use:• Free test bundle (10h)• Various small/medium/large

bundles• Reduced costs (only

hardware and maintenance)• Management by CH body• Maintenance by industry

partner

Upload: via http, ftp, api

Account information

Dutch Queen Wilhelmina addressing the Dutch people from London during WWII