Let It Flow: Government e pubs (NYPL Open Book Hack 2014)

download Let It Flow: Government e pubs (NYPL Open Book Hack 2014)

of 20

  • date post

    11-Aug-2014
  • Category

    Design

  • view

    102
  • download

    6

Embed Size (px)

description

NYPL Labs hosted the Open Book Hack Weekend at the New York Public Library featuring digital book open source and content development based on HTML5, EPUB, and the Open Web Platform. Readium Foundation, O’Reilly Media, Perseus Books, Hypothes.is, Google, and Datalogics sponsored the event. I served as a user experience designer for my team, which included Dave Mayo of Harvard University, Julia Pollacks of the Bronx Community College, and Jeremy Baron. We worked on developing a way to convert PDF documents to reflowable EPUB format. We looked at converting government documents, including a sample of slip opinions from the United States Supreme Court. Sample documents and scripts are located at: https://github.com/pobocks/pdf2freedom Below is our group presentation. More information about the Open Book Hack event is available at the OpenBook2014 github site: https://github.com/openbook2014/nypl-hack-weekend/wiki/Hack-ideas

Transcript of Let It Flow: Government e pubs (NYPL Open Book Hack 2014)

  • Let it Flow: Government Documents PDF (or PDF + XML) to reflowable EPUB Open Book Hackathon, NYPL January 11-13, 2014
  • Proposed Tasks: Legal ePUB What would users want? Remove hard line breaks for continuous, reflowable display in multiple formats/modes. Parse metadata from header information, pagination and other semantic content. Parse citation of previous opinions or laws and link to cited legislation. Create stylesheet and page markup for display on devices. It is important to: Ensure that data is formatted so that it flows naturally regardless of device used. Maintain information about pagination for citation and discovery purposes.
  • Proposed Tasks: Legal ePUB What would users want? Remove hard line breaks for continuous, reflowable display in multiple formats/modes. Parse metadata from header information, pagination and other semantic content. Parse citation of previous opinions or laws and link to cited legislation. Create stylesheet and page markup for display on devices. It is important to: Ensure that data is formatted so that it flows naturally regardless of device used. Maintain information about pagination for citation and discovery purposes. We only had time for one thing.
  • Supreme Court Slip Opinions
  • Supreme Court Slip Opinions Supreme Court slip opinion documents are available in PDF format. Dockets may contain multiple document types: Syllabus Opinion of the Court Per Curiam Dissention
  • Metadata: SCOTUS Slip Opinions Citation Document Type Notice Presiding Court Docket Number Case Names/Parties to the Case Date Decided Justice Opinion Front matter for each document contains several centered blocks of content. At first glance, the documents contain minimal structural information.
  • XML Format 1 (Slip Opinion) OCTOBER TERM, 2013 Syllabus NOTE: Where it is feasible, a syllabus (headnote) will be released, as isbeing done in connection with this case, at the time the opinion is issued.The syllabus constitutes no part of the opinion of the Court but has beenprepared by the Reporter of Decisions for the convenience of the reader. See United States v. Detroit Timber & Lumber Co., 200 U. S. 321, 337. SUPREME COURT OF THE UNITED STATES Syllabus KANSAS v. CHEEVER CERTIORARI TO THE SUPREME COURT OF KANSAS No. 12609. Argued October 16, 2013Decided December 11, 2013 We used Poppler to convert Supreme Court slip opinion documents (PDF) to XML. This revealed position and size attributes for lines of text, but no structured semantic information.
  • XML Format: Font ID 1 (Slip Opinion) OCTOBER TERM, 2013 Syllabus NOTE: Where it is feasible, a syllabus (headnote) will be released, as isbeing done in connection with this case, at the time the opinion is issued.The syllabus constitutes no part of the opinion of the Court but has beenprepared by the Reporter of Decisions for the convenience of the reader. See United States v. Detroit Timber & Lumber Co., 200 U. S. 321, 337. SUPREME COURT OF THE UNITED STATES Syllabus KANSAS v. CHEEVER CERTIORARI TO THE SUPREME COURT OF KANSAS No. 12609. Argued October 16, 2013Decided December 11, 2013 We may be able to use position and size attributes to identify content parts of the document. For example, different parts of the document have varying positions, font sizes and line heights.
  • XML Format: Font ID 1 (Slip Opinion) OCTOBER TERM, 2013 Syllabus NOTE: Where it is feasible, a syllabus (headnote) will be released, as isbeing done in connection with this case, at the time the opinion is issued.The syllabus constitutes no part of the opinion of the Court but has beenprepared by the Reporter of Decisions for the convenience of the reader. See United States v. Detroit Timber & Lumber Co., 200 U. S. 321, 337. SUPREME COURT OF THE UNITED STATES Syllabus KANSAS v. CHEEVER CERTIORARI TO THE SUPREME COURT OF KANSAS No. 12609. Argued October 16, 2013Decided December 11, 2013 In this document, the font attribute is 1 for the Note section, 3 for the names of the parties to the case and 4 for the name of the presiding court.
  • XML Format: attributes Font sizes are defined after the elements in the XML document. In this example, the title of the Supreme Court of the United States has height=20 and font=4, which corresponds to fontspec id=4, which is defined as Times 20pt Black. ... SUPREME COURT OF THE UNITED STATES Font ID 4 is Times 20pt
  • XML Format: attributes Font specifications are defined within the page element. If subsequent pages require new font IDs they are added after the element. ...
  • XML Format: Inconsistencies Font sizes for specific id attributes are not consistent across all documents, BUT... Docket 12-609 In this document The Supreme Court of the United States has font id 4. Docket 12-729 In this document The Supreme Court of the United States has font id 5.
  • XML Format: Commonalities ...we noticed that the position attributes for many content parts such as left margin and line height were consistent across all documents. These content parts are noted above. Content Parts with consistent text attribute values: Page elements have a height=1188 and width=918 on all documents. Decision date has the same attributes for left=395 and height=16 on all documents. Cite as is always top="171" left="366" width="189" height="16" font="0" Type of Opinion, e.g. (Slip Opinion) is always top="174" left="234" font="1" Opinion of the Court is always top="171" left="366" width="189" height="16" font="0" Note and Notices is always left="272" height="13" font="1" Exception: sometimes the first line is indented at left=284 The last line of a Note or Notice is always a reference to another court case, which always has top="280" left=" 272" width="343" height="13" font="1"> The name of the case (usually [partyA] v. [partyB]) is always height="20" font="3".
  • XML Format: left & height attributes 1 (Slip Opinion) OCTOBER TERM, 2013 Syllabus NOTE: Where it is feasible, a syllabus (headnote) will be released, as isbeing done in connection with this case, at the time the opinion is issued.The syllabus constitutes no part of the opinion of the Court but has beenprepared by the Reporter of Decisions for the convenience of the reader. See United States v. Detroit Timber & Lumber Co., 200 U. S. 321, 337. SUPREME COURT OF THE UNITED STATES Syllabus KANSAS v. CHEEVER CERTIORARI TO THE SUPREME COURT OF KANSAS No. 12609. Argued October 16, 2013Decided December 11, 2013 Position of the left margin and line height may indicate a single text block. In this example, most of the lines of the paragraph are indented to 272, while the first line of the paragraph is indented to 284. Still, each line has the same line height.
  • Other Standard Format Elements Page Headings Footnotes Page header content alternates on even and odd pages, but the position and text attributes for the included content is the same. Footers are always noted in superscript within the text and displayed after a rule at the bottom of the page.
  • Tools for Text Mining Text Editing Initiative Court Listener: https://www.courtlistener.com/ Semantic Parsing for Legal Texts (conference proceedings): http://www.lrec-conf. org/proceedings/lrec2012/workshops/27.LREC%202012%20Workshop%20Proceedings% 20SPLeT.pdf Search and Replace We looked at a number of pre-processing tools to add semantic information to the documents, such as identifying docket number, case names, decision dates, presiding court, Justice name, etc., as well as cited legislation and prior court cases.
  • Text Encoding Initiative Text Encoding Initiative allows content producers to tag parts of a document with semantic information. footnotes glossing or commenting on any passage could be added; pointers linking parts of this text to others could be added; proper names of various kinds could be distinguished from the surrounding text; detailed bibliographic information about the text's provenance and context could be prefixed to it; a linguistic analysis of the passage into sentences, clauses, words, etc., could be provided, each unit being associated with appropriate category codes; the text could be segmented into narrative or discourse units; systematic analysis or interpretation of the text could be included in the encoding, with potentially complex alignment or linkage between the text and the analysis, or between the text and one or more translations of it; passages in the text could be linked to images or sound held on other media. Source: TEI Lite: Encoding for Interchange: an introduction to the TEI
  • Parsing Metadata Court Name in XML: SUPREME COURT OF THE UNITED STATES Could be rendered as the following, because the court always has attribut