NYPL Labs 9-10-13 HacksHackers Presentation

The Great Data Migration

or... hackin’ the library with nypl labs

9/10/13

a What is NYPL Labs?

Ben Vershbow | Founder & Manager - NYPL [email protected] | @subsublibrary

@nypl_labs | #HacksHackers | Ben Vershbow | [email protected] | @subsublibrary

mailto:[email protected]


a New York Public Library


existing patron base


existing patron base+ global community of users


free for all to use


free for all to use+ hack / build / improve


books, archives, images, documents, A/V etc.


+ digital material, data & APIsbooks, archives, images, documents, A/V etc.

Map Warpermaps.nypl.org

a

a

What’s on the Menu?menus.nypl.org

a

Stereogranimatorstereo.nypl.org

NYPL NYPLBPL↑ ↑ ↑




+




a

Direct Me NYC: 1940directme.nypl.org

x




NYPL collec-ons

Genealogy community

NYPL collec-ons

Genealogy community

NYPL collec-ons

U.S. Geological Survey

Genealogy community

NYPL collec-ons

U.S. Geological Survey OpenStreetMap (via MapBox)

Genealogy community

NYPL collec-ons


New York Times API

Genealogy community

NYPL collec-ons


New York Times API

NYPL users & staff


Textapi.repo.nypl.org



a

Crowd-sourcing the transcription of historical theater programs

Ensembleensemble.nypl.org

Paul [email protected] | @nonword

FromThePage / Transcribe Bentham




Scripto




T-PEN




Freeform text transcription is not complex entity extraction




Crowd sourcing complex entity extraction of documents with inconsistent layouts

e.g. historical theater programs

Ensemble




NYPL Labs | What’s on the Menu?




Transcribable & DocumentCloud




Zooniverse | Notes From Nature




Zooniverse | Old Weather

NYPL Labs | Ensemble

http://ensemble.nypl.org

Built from Scribehttps://github.com/zooniverse/scribe






FromThePage beta.fromthepage.com

T-PEN t-pen.org

Whats’ on the Menu? menus.nypl.org

Transcribable github.com/propublica/transcribable

Notes from Nature notesfromnature.org

Old Weather oldweather.org

Ensemble ensemble.nypl.org


http://beta.fromthepage.com/

http://beta.fromthepage.com/

http://t-pen.org/TPEN/

http://t-pen.org/TPEN/

http://menus.nypl.org

http://menus.nypl.org

https://github.com/propublica/transcribable




http://www.notesfromnature.org/

http://www.notesfromnature.org/

http://www.oldweather.org/

http://www.oldweather.org/

http://ensemble.nypl.org/

http://ensemble.nypl.org/



a

Archives & Manuscripts

archives.nypl.org

Trevor [email protected] | @trevorthornton

Matt [email protected] | @thisismmiller

a

Archives & Manuscripts

archives.nypl.org

Trevor [email protected] | @trevorthornton

Matt [email protected] | @thisismmiller

or: where to find Timothy Leary’s Powerglove

Unique, unpublished materials: correspondence, personal papers, organizational records, literary manuscripts, AV documentation, electronic records

Typically included within discrete collections, which are often acquired in whole

Finding aids provide researchers with guidance on collection contents

EAD (Encoded Archival Description)XML schema for encoding finding aids

NYPL Archives & Manuscripts

@nypl_labs | #HacksHackers | Archives & Manuscripts | Trevor Thornton | [email protected] | @trevorthorntonMatt Miller | [email protected] | @thisismmiller



The traditional model for presenting EAD-encoded finding

aids




What we did (more or less)




System overview




a

Video Annotation & Synchronization

digitalcollections.nypl.org/tools/video/compose

For NYPL Digital CollectionsJerome Robbins Dance Division

Brian Foo | [email protected] | @beefoo

ScenariosJerome Robbins Dance Division

Enhance & Improve video data• e.g. Sync multiple angles of the same performance• e.g. Annotate a performance

Discovery• e.g. Compare multiple performances

Instruction• e.g. Enhance lecture with multimedia

Probably many more• e.g. Mash-ups

@nypl_labs | #HacksHackers | Video Annotation & Synchronization | Brian Foo | [email protected] | @beefoo



Technology Used

RoR - Backend Framework

Backbone.js - Javascript MVC Framework

Brightcove - Video delivery platform

Popcorn.js - HTML5 media framework by Mozilla• Does not natively support multi-video• Does not natively support Brightcove

@nypl_labs | #HacksHackers | Video Annotation & Synchronization | Brian Foo | [email protected] | @beefoo



ateh vectorizor

github.com/NYPL/map-vectorizer

mauricio giraldo arteagaNYPL Labs

[email protected] | @mgiraldo

building =




not paper

building =




not paper

not black

building =




not paper

not black

> 20m2 (~180ft2)

building =




not paper

not black

> 20m2 (~180ft2)

< 3,000m2 (~27,000ft2)

building =




not paper

not black

> 20m2 (~180ft2)

< 3,000m2 (~27,000ft2)

+ attributes (color, dots, crosses...)

building =




process




https://github.com/NYPL/map-vectorizer

test it! (please)




gdal_polygonize.pygenerates polygons automagically!




$ gdal_polygonize.py test.tif -f "ESRI Shapefile" test.shp test

we need to optimize the input




we need to simplify the output

(for those polygons that we care about)




pts = spsample(polygon, n=1000, type="hexagonal")

pts = spsample(polygon, n=1000, type="hexagonal")pts = spsample(polygon, n=1000, type="regular")

pts = spsample(polygon, n=1000, type="hexagonal")pts = spsample(polygon, n=1000, type="regular")pts = spsample(polygon, n=1000, type="random")

pts = spsample(polygon, n=1000, type="hexagonal")pts = spsample(polygon, n=1000, type="regular")pts = spsample(polygon, n=1000, type="random")pts = spsample(polygon, n=500, type="hexagonal")

x.as = ashape(pts@coords,alpha=2.0)




we need to validate the output

(polygonzo!)




a Old NYCDan Vanderkam

SOME OTHER [email protected] | @danvdk

~40,000 images

Mostly taken from 1920–1950

Many were taken by Percy Loomis Sperr, who was commissioned by the library to take photographs of buildings soon to be demolished

Milstein Collection

@nypl_labs | #HacksHackers | Old NYC | Dan Vanderkam | [email protected] | @danvdk



Images on the NYPL site were small, pictures even smaller.

What’s MrSID?

Challenges




First find the areas that aren’t brown:




Then find the Rectangles:




UI work

Better geocoding for boroughs with complicated streets

Keep your eyes out for an Old NYC launch this fall!

http://www.danvk.org/wp/2013-02-09/finding-pictures-in-pictures/

What’s left?




awhat’s next

(hint: you)

Dave [email protected] | @riordan

this used to be a reservoir of water, now its a reservoir of knowledge


–an anonymous nypl docent



now its a reservoir of data




now its time to use it




there will be more




HackathonsPublishing Hackathon | Maphack




there will be more




NYPL Tech Challenges(coming soon)

like the x-prize but for way lower stakes and civic good




Questions for you:




what kind of things would you want to work on with

nypl labs?




making ebooks easier to borrow?




opening up historical social networks?




we want to know what questions you’re

interested in




how you want to use the library today...




...will be how everyone will use the library very

soon.




help us make that happen




its gonna be awesome




NYPL Labs 9-10-13 HacksHackers Presentation

Documents

Transcript of NYPL Labs 9-10-13 HacksHackers Presentation