Spy Pond Partners

Frances D. Harrison

Chair, TRB Data and Information Technology Section Subcommittee on Metadata

January 22, 2006

Current Perspectives on Transportation Metadata Needs

2006 TRB Annual Meeting

2

Agenda

• Metadata – what and why

• Metadata in transportation • Library• Geospatial • ITS – Archived Traffic Data• ITS – Road Weather Data• TransXML – road and bridge design, construction, safety

• TRB Data Section Subcommittee on Metadata• Mission and Role• Activities and Next Steps

3

Don’t you hate it when…

• You ask five people in your agency the number of highway system miles and get five different answers

• You spend an hour a day answering the same questions about your agency’s data

• Your “data guru” quits and leaves gigabytes of partially cleaned data with cryptic documentation

• You spend thousands of dollars building a data set only to discover that it already existed in another agency

• You try to combine data gathered from five different corridor studies sponsored by different agencies but can’t fit the pieces together

• You can’t use half of the data you find because you don’t know what it means or how good it is

4

Wouldn’t it be great if…

• Agencies had policies in place to require good, complete documentation of all data sets – and devoted the resources to make this happen

• There were standard templates for documenting every major kind of transportation data

• There were tools to produce and validate this documentation

• There was an easy way for agencies to share their data and move towards standardized data definitions and coding schemes

• There was agreement about how to measure and describe data quality for different types of data

• There were better and quicker ways of finding data on the web than Google

5

The Solution: Metadata!

TrainingTraining

WorkflowWorkflow

RepositoriesRepositories

ToolsTools

PoliciesPolicies

StandardsStandards

6

Definition of Metadata

“Information about a thing, apart from the

thing itself”

-Ned Batchelder

“Structured data about an object that

supports functions associated with the designated object”

-J. Greenburg

Thing Metadata

Book TitleAuthorPublication DateISBN Number

Photograph Subject

Photographer

DateWeb Page URI

Title

Language

GIS Coverage Location

Accuracy

Format

Data Element Element Name

Data Type

Length

Possible Values

7

Card Catalogue

8

Document Properties

9

Geospatial Metadata

10

Data Dictionary – Attribute Information

Source: Traffic Management Data DictionarySource: Traffic Management Data Dictionary

11

Metadata for Sharing Photographs – Flickr Folksonomies

12

Metadata Maximizes and Preserves Data Value

DiscoveringDiscovering UnderstandingUnderstanding

SharingSharing

13

But Metadata is Hard

1. Too abstract

2. Too complicated

3. Not sexy

4. Not easy

5. Tedious

6. Time consuming

7. Expensive

8. Hard to get agreement

9. Not on the critical path

10. Not enforced

11. Low priority

12. Stepchild of the stepchild

14

Do we need a 12 step program for metadata?

1. Recognize there is a problem

2. Believe there is a way to solve it

3. Commit to doing something about it

4. Examine your current practices

5. Publicly admit that things need to change

6. Prepare to change the status quo

7. Get help from colleagues and other experts

8. Identify specific practices to be improved

9. Correct them

10. Institutionalize the process in your agency

11. Periodically reaffirm why you are doing this

12. Help others by sharing your successes and lessons learned

15

Metadata in Transportation - examples

• Library

• Geo-spatial

• ITS – Archived Traffic Data

• ITS – Road Weather Data

• TransXML - data exchange formats for road and bridge design, construction, safety

16

Existing Standards (to name a few…)

• Dublin Core – cross-domain resource description

• MODS, MARC, METS –metadata standards for digital libraries

• FGDC CSDGM – Metadata for geospatial datasets

• ISO 19115/19139 Geographic Information - Metadata

• ISO 11179 – Metadata Registries

• IEEE 1489 – ITS Data Dictionaries

• ISO 14817 – Meta Attributes for ITS/TICS Data Dictionaries

• IEEE 1487 – Metadata Registries for ITS (based on ISO 11179 and IEEE 1489)

• DDI – Data Documentation Initiative (social science survey data documentation)

17

National Transportation Library – Modified Dublin Core

18

Transportation Research Thesaurus

19

FGDC Metadata

• Executive Order 12906 (1994) • Mandated creation of metadata standards in conjunction with the National Spatial

Data Infrastructure (NSDI)• Federal Geographic Data Committee (FGDC) – interagency group charged with

creation and maintenance of the standards

• Widely adopted and used

• Wealth of resources: tools, tutorials, training

• Enabler of metadata clearinghouses – access to wide variety of geospatial datasets

• Harmonization with ISO 19115

20

FGDC Content Standard for Digital Geospatial Metadata

21

Archived ITS Data

• ASTM E2259-03 Standard Guide for Archiving and Retrieving ITS-Generated Data• Published June 2003• General description of metadata types: archive structure, processing

documentation and data collection system

• ASTM Task Group E17.54.02.1: “Standard Specification for Metadata Content for ITS-Generated Data• Based on FGDC – given wide acceptance and tool availability• Tackling specific issues related to traffic data – e.g. quality, aggregation

methods

22

Clarus

• National repository of road weather data – network of networks

• Separation of Producers and Consumers - Metadata Essential• 80 elements defined• Based on quality checking and user requirements

• Data exchange formats• Need to address data sharing between “Weather World” and “ITS World”

23

TransXML

• Transportation Data Exchange Schema – NCHRP 20-64

• Four areas • Roadway Design (integration of GIS data, construction cost estimates)• Construction (bidding, payments, status tracking)• Safety (crash data and highway safety analysis)• Bridge Design (verification of structural analysis)

• Definition of data structures – entities, attributes and relationships (UML)

• XML Schema posted for comment: www.transxml.org

• Sample applications end of January

• Final Report – June 2006

• Presentation at AASHTO IS (June 4-7 – Ft Worth, TX)

24

Example: Data Exchanges for Construction Progress

Progress Payment System

Installed QuantitiesPay Item Quantities

MaterialsMaterial Test ResultsPlacement Location

Material SamplesTests PerformedTest Outcomes

Project Engineer Progress

Reporting

Laboratory Systems

Field Data Collection Systems

25

26

TransXML – Issues

• Leverages existing schema and standards – different styles, overlapping elements, moving targets• GML• LandXML• Global Justice• NHTSA Crash Records (MMUCC and FARS)• AASHTOWare data schema• aecXML

• Balance between consistent framework and niche schemas

• Stakeholder participation lower than anticipated

• Never enough resources for communication, education and liaison

• Chicken and egg considerations• “Vendors will only act if agencies demand it”• “Agencies will come on board if the vendors support it”

27

Conclusions

• Needs and technology converging to enable advances in practice

• Good models exist – foundation for continued work

• More standards and models needed• Dublin Core or FGDC extensions and refinements for different data objects• Data dictionaries and schemas for common data exchanges• Taxonomies, controlled vocabularies• Code lists and format rules for shared attributes

• Standards are not enough - tools, training, policies, practices

• Great need for coordination and information sharing across diverse efforts

• Crosswalks – to promote understanding and enable harmonization

28

TRB Metadata Metamorphosis

• 1998-2005 Metadata Subcommittee operated under Urban Data – led by Marcus Wigan• Research • Education• Coordination

• Need for broader approach• Metadata is a cross-cutting issue – not appropriate as a subcommittee under Urban

Data• Metadata is key to data sharing – but requires leadership and coordination• Metadata is abstract, not widely understood - hard to get people’s attention• How to get to the next level?

• Working Group – February 2005• Joanna Zmud chair• Mission: Recommend plan of action for the Data and Technology Section by summer

meeting in August• Outcome: Section subcommittee immediately – morph into Task Force

29

TRB Metadata: The Next Generation

• Section Subcommittee – August 2005• Members from most Data Section Committees

+ Library Science

• Mission and Roles• Promote common understanding and appreciation of metadata

value• Share information about metadata related activities across

transportation data communities• Identify priority needs for new metadata standards• Promote research that provides technical groundwork for standards• Promote coordination and consistency across standards efforts• Promote best practices in use of metadata• Promote metadata training and tools development

30

TRB Metadata Subcommittee – Activities To Date

• Monthly Conference Calls

• Compiling Resource Lists: Standards, Tools, Implementation Examples

• Draft Research Agenda – Eight Initiatives• White Paper• State of the Practice Survey – help to set priorities• Best Practice Synthesis• End User Needs Investigation• MPO Use of Metadata for Modeling and GIS-T• Gap Closure• Feasibility Study for Metadata Registry or Clearinghouse• Crosswalk and Code List Catalogue

• Comments and Input Welcome – fdharrison@gmail.com

31

Future Task Force Product: Strategic Plan for Metadata

• Establish need for and benefit of metadata standards

• Recommend and prioritize types of standards for development

• Identify potential barriers to acceptance

• Identify coordination needs across standards efforts

• Identify stakeholders and participants

• Recommend methods to promote use

32

Acknowledgements

• Kuo-Ann Chiao (NYMTC)

• Scott Dahl (US Census Bureau)

• Ralph Gillmann, FHWA

• James Hall (University of Illinois, Springfield)

• Joyce Koeneman (RITA-NTL)

• Guy Rousseau (ARC)

• Jack Stickel (Alaska DOT)

• Xiaduan Sun (University of Louisiana)

• Shawn Turner (Texas A&M)

• Anita Vandervalk (CSI)

• Marcus Wigan, Oxford Systematics

• Eric Ziering (CSI)

• Joanna Zmud, NuStats