Paul Gorman, MD Lois Delcambre, PhDDavid Maier, PhD

Bundles in the wild………..Bundles in the wild………..

Observational team:Observational team:Paul GormanPaul GormanJoan AshJoan AshMary LavelleMary LavelleJason LymanJason Lyman

……………………..Bundles in captivity..Bundles in captivityComputer science team:Computer science team:

Lois DelcambreLois DelcambreDave MaierDave MaierShawn BowersShawn BowersLongxing DengLongxing DengMathew WeaverMathew Weaver

(Wild) Bundles(Wild) Bundles

(Wild) Bundles(Wild) Bundles

There is benefit in creating There is benefit in creating (active processing of information)(active processing of information)

There is benefit in reusing There is benefit in reusing (trigger memory)(trigger memory)

There is benefit in sharing There is benefit in sharing (establish collective, situated awareness)(establish collective, situated awareness)

SLIMPad DemoSLIMPad Demo

SuperimposedApplication

BaseApplicationuser

SimultaneousViewing

Enhanced Base-LayerViewing

BaseApplication

AddedSuperimposedFunctionality

IndependentViewing

Three viewing styles for superimposed applications.

SuperimposedApplication

BaseApplication

borrowsfunctionality

user user

Superimposed and base layers Superimposed and base layers with markswith marks

Superimposed Superimposed LayerLayer

BaseBaseLayerLayer InformatioInformatio

n Sourcen Source11

InformatioInformatio

n Sourcen Source22

InformatioInformatio

n Sourcen Sourcenn

…

marksmarks

SuperimposedApplication

The general architecture for managingsuperimposed information

Superimposed Information Management

ApplicationData

ApplicationSpecific

API

GenericManagement

TRIMStore

creates and manages

Mark Management

Superimposed Layer Superimposed Layer Information Manager (SLIM) Architecture:Information Manager (SLIM) Architecture:

ContributionsContributions

SLIM API - for the application developerSLIM API - for the application developer

TRIM store - for generic storage of TRIM store - for generic storage of superimposed informationsuperimposed information

Mark Management - to create/resolve marksMark Management - to create/resolve marks

SLIM APISLIM API

Bundle

bundleName : StringbundleXPos : NumberbundleYPos : NumberbundleHeight : NumberbundleWidth : Number

Scrap

scrapName : StringscrapXPos : NumberscrapYPos : Number

SLIMPad

padName : String Mark

markId : String

1 *

1

*

*

0..1

Structured Bundle Model for SLIMPad.

AbstractBundle

Mark Management

SLIMPad

MarkManager

MarkDB

user

XML Documents

PDF files

Web Pages

Excel Spreadsheets

PPT Files

SuperimposedInformationManagement

XMLViewer

PDFViewer

IEExplorer

MSExcel

MSPowerPoint

HTMLModule

ExcelModule

PowerPointModule

XMLModule

PDFModule

Model

Schema Data

Instance Data with Marks

InformationSource1

InformationSource2

SuperimposedLayer

BaseLayer

marksmarks

Model-Based Superimposed InformationModel-Based Superimposed Information

But the model and schema are optionalBut the model and schema are optional

Generic Representation of Information Generic Representation of Information

Topic MapTopic Map

Topic Map Topic Map DefintionDefintion

Topic Map Topic Map InstancesInstances

XMLXML

DTDDTD

XML XML DocumentDocument

MetamodelMetamodel

SuperimposedSuperimposedLayerLayer

Basic Set of Basic Set of AbstractionsAbstractions

Model Constructs Model Constructs and Relationshipsand Relationships

Schema-LevelSchema-LevelDataData

Instance-LevelInstance-LevelDataData

Three Basic Types of MappingsThree Basic Types of Mappings

ModelModel11

SchemaSchema11

InstancesInstances11

SourceSource TargetTarget

MappedMapped

ConvertedConverted

ConvertedConverted

MappedMapped

ConvertedConverted

MappedMapped

ConvertedConverted

ConvertedConverted

Inter-ModelInter-Model

Inter-SchemaInter-Schema

Model-to-Model-to-SchemaSchema

ModelModel22

SchemaSchema11

InstancesInstances11

ModelModel11

SchemaSchema11

InstancesInstances11

ModelModel11

SchemaSchema11

InstancesInstances1 1

ModelModel11

SchemaSchema22

InstancesInstances11

ModelModel22

SchemaSchema22

InstancesInstances11

(CS Team) Plans (CS Team) Plans

““Schema” for bundlesSchema” for bundles: through : through templatestemplates that predefine that predefine certain bundles and labels, and possibly predefine certain bundles and labels, and possibly predefine scraps. less restrictive than trad. schema.scraps. less restrictive than trad. schema.

SharingSharing: a bundle that belongs to multiple pads, a pad : a bundle that belongs to multiple pads, a pad with multiple simultaneous users, and interchange of with multiple simultaneous users, and interchange of pads and bundles between users. pads and bundles between users.

““Scoping”Scoping” of base layers and marks over those layers. of base layers and marks over those layers. There are several general issues for modeling and There are several general issues for modeling and management of superimposed information that have management of superimposed information that have emerged from this work and other efforts.emerged from this work and other efforts.

(CS Team) Plans (cont.)(CS Team) Plans (cont.)

DistributionDistribution: more than a single mark manager. Is there a : more than a single mark manager. Is there a single space of marks or must applications keep track of single space of marks or must applications keep track of which manager handles a particular mark? which manager handles a particular mark?

New Kinds of Sources and MarksNew Kinds of Sources and Marks: beyond XML, PDF, : beyond XML, PDF, MS Word, MS Excel, MS PowerPoint and HTML. We are MS Word, MS Excel, MS PowerPoint and HTML. We are also interested in marks into a “base” layer that is actually also interested in marks into a “base” layer that is actually superimposed information for another application.superimposed information for another application.

Application InterfaceApplication Interface: try a more query-like API. Also, : try a more query-like API. Also, consider an API generatorconsider an API generator

Superimposed Information Mapping/ConversionSuperimposed Information Mapping/Conversion: applying : applying and extending our metamodel.and extending our metamodel.

www.cse.ogi.edu/footprintswww.cse.ogi.edu/footprints

demos - including the QTVR of the ICU (with demos - including the QTVR of the ICU (with toys) and SLIMPadtoys) and SLIMPad

personnelpersonnel project descriptionproject description paperspapers

““Bundles in the Wild: Tools for Managing Bundles in the Wild: Tools for Managing Information to Maintain Situation Awareness”Information to Maintain Situation Awareness”

““Bundles in Captivity: An Application of Bundles in Captivity: An Application of Superimposed Information”Superimposed Information”

papers discussing superimposed informationpapers discussing superimposed information

<RDF> <Construct ID=”ElementType”/> <Construct ID=”AttributeType”/> <Construct ID=”Element”/> <Construct ID=”Attribute”/> <Construct ID=”Content”/> <Lexical ID=”ContentType”/>

<Connector ID=”elemTypeName”> <domain resource=”#ElementType”/> <range resource=”#String”/> <domainMult value=”*”/> <rangeMult value=”1”/> </Connector> <Connector ID=”nestedElemType”> <domain resource=”#ElementType”/> <range resource=”#ElementType”/> <domainMult value=”0..1”/> <rangeMult value=”*”/> </Connector> <Conformance ID=”elemInstOf”> <domain resource=”#Element”/> <range resource=”#ElementType”/> <domainMult value=”*”/> <rangeMult value=”0..1”/> </Conformance> …</RDF>

ElementType

AttributeType

Element Attribute

PrimitiveContent

elemTypeName : String

attTypeName : String

tagName : String attName : StringattValue : String

<<conformance>>elemInstOf

holds

attributeOf

attTypeOf

<<conformance>>attInstOf

*

*

*

*

0..1

0..11

1*

0..11

nestedElemType

nestedElem

*

0..1

0..1

Figure 7. The XML model represented using UML with a sample of the RDF representation.

<<Lexical>>PrimitiveContentType

text : String

*

<<conformance>>contentHasType

0..1elemContType

0..1*

Representing Model using RDFRepresenting Model using RDF

(instanceOf, “(instanceOf, “TopicTypeTopicType”, Construct)”, Construct)(instanceOf, “(instanceOf, “TopicInstanceTopicInstance”, ”, Construct)Construct)

(instanceOf, “(instanceOf, “ttypenamettypename”, Connector)”, Connector)(domain, “ttypename”, TopicType)(domain, “ttypename”, TopicType)(range, “ttypename”, String)(range, “ttypename”, String)(domainMult, “ttypename”, “*”)(domainMult, “ttypename”, “*”)(rangeMult, “ttypename”, “1”)(rangeMult, “ttypename”, “1”)

(instanceOf, “(instanceOf, “topic_instOftopic_instOf”, ”, Conformance)Conformance)(domain, “topic_instOf”, (domain, “topic_instOf”, TopicInstance)TopicInstance)(range, “topic_instOf”, TopicType)(range, “topic_instOf”, TopicType)(domainMult, “topic_instOf”, “*”)(domainMult, “topic_instOf”, “*”)(rangeMult, “topic_instOf”, “1”)(rangeMult, “topic_instOf”, “1”)

TopicType

ttypename : String

TopicInstance

*

1

<<conformance>>topic_instOf

(instanceOf, “(instanceOf, “painting_ttpainting_tt”, TopicType)”, TopicType)(ttypename, “painting_tt”, “painting”)(ttypename, “painting_tt”, “painting”)(instanceOf, “(instanceOf, “painter_ttpainter_tt”, TopicType)”, TopicType)(ttypename, “painter_tt”, “painter”)(ttypename, “painter_tt”, “painter”)

(instanceOf, “(instanceOf, “byPainter_rtbyPainter_rt”, ”, TopicRelType)TopicRelType)(relType, “byPainter_rt”, “by painter”)(relType, “byPainter_rt”, “by painter”)(topicType1, “byPainter_rt”, painting_tt)(topicType1, “byPainter_rt”, painting_tt)(topicType2, “byPainter_rt”, painter_tt)(topicType2, “byPainter_rt”, painter_tt)

(instanceOf, “(instanceOf, “biography_atbiography_at”, ”, AnchorType)AnchorType)(anchorRole, “biography_at”, (anchorRole, “biography_at”, “biography”)“biography”)(topicType, “biography_at”, painter_tt)(topicType, “biography_at”, painter_tt)

Representing Schema using RDFRepresenting Schema using RDF

Topic Types (schema):Topic Types (schema):painting, painterpainting, painter

Topic Rel Types (schema):Topic Rel Types (schema):by painterby painter

Anchor Types (schema):Anchor Types (schema):biographybiography

paintingpainting painterpainterby painterby painter

biographybiography

Representing Instances using RDFRepresenting Instances using RDF

(instanceOf, “painter1”, TopicInstance)(instanceOf, “painter1”, TopicInstance)(title, “painter1”, “Paul Klee”)(title, “painter1”, “Paul Klee”)(topicInsID, “painter1”, “5”)(topicInsID, “painter1”, “5”)(topic_instOf, “painter1”, painter_tt)(topic_instOf, “painter1”, painter_tt)(instanceOf, “painting1”, TopicInstance)(instanceOf, “painting1”, TopicInstance)(title, “painting1”, “Captive”)(title, “painting1”, “Captive”)(topicInsID, “painting1”, “19”)(topicInsID, “painting1”, “19”)(topic_instOf, “painting1”, painting_tt)(topic_instOf, “painting1”, painting_tt)

(instanceOf, “byPainter1”, TopicRelInst)(instanceOf, “byPainter1”, TopicRelInst)(rel_instOf, “byPainter1”, byPainter_rt)(rel_instOf, “byPainter1”, byPainter_rt)(topicIns1, “byPainter1”, painting1)(topicIns1, “byPainter1”, painting1)(topicIns2, “byPainter1”, painter1)(topicIns2, “byPainter1”, painter1)

(instanceOf, “biography1”, AnchorInst)(instanceOf, “biography1”, AnchorInst)(anchor_instOf, “biography1”, biography_at)(anchor_instOf, “biography1”, biography_at)(address, “biography1”, a1)(address, “biography1”, a1)

(instanceOf, “a1”, Address)(instanceOf, “a1”, Address)(markID, “a1”, (markID, “a1”, “URLMarkManager@954308545”)“URLMarkManager@954308545”)

Topic Instances (instance):Topic Instances (instance):Paul Klee, CaptivePaul Klee, Captive

Topic Relationship (instance):Topic Relationship (instance):a by painter relationshipa by painter relationship

Anchor (instance):Anchor (instance):a biography anchora biography anchor

Address (instance):Address (instance):mark to urlmark to url

Harvesting Information to Harvesting Information to Sustain our Forests:Sustain our Forests:

Creating anCreating anAdaptive Management PortalAdaptive Management Portal

NSF DIGITAL GOVERNMENT PROGRAMNSF DIGITAL GOVERNMENT PROGRAM

Tim Tolle & Lois DelcambreTim Tolle & Lois Delcambrettolle@fs.fed.us lmd@cse.ogi.eduttolle@fs.fed.us lmd@cse.ogi.edu

Co-Project DirectorsCo-Project Directors

Project focuses on Project focuses on the:the:

Adaptive Adaptive

ManagementManagement AreasAreas

USDA Forest ServiceUSDA Forest ServiceUSDI Bureau of Land USDI Bureau of Land

ManagementManagementUSDI Fish and USDI Fish and Wildlife ServiceWildlife Service

Team MembersTeam Members

Tim Tolle Tim Tolle Regional Coordinator for AMA, US Forest ServiceRegional Coordinator for AMA, US Forest Service

Eric LandisEric Landis Forest Information System Specialist, ConsultantForest Information System Specialist, Consultant

Craig PalmerCraig Palmer Natural Resources Monitoring Expert, UNLVNatural Resources Monitoring Expert, UNLV

Fred PhillipsFred Phillips Professor, Head, Mgt. of Science and Tech., OGIProfessor, Head, Mgt. of Science and Tech., OGI

Patty ToccalinoPatty Toccalino Asst. Prof., Environmental Science and Eng., OGIAsst. Prof., Environmental Science and Eng., OGI

Lois DelcambreLois Delcambre Professor, Computer Science and Eng., OGIProfessor, Computer Science and Eng., OGI

David MaierDavid Maier Professor, Computer Science and Eng., OGIProfessor, Computer Science and Eng., OGI

Shawn BowersShawn Bowers PhD Student, Computer Science and Eng., OGIPhD Student, Computer Science and Eng., OGI

Mat WeaverMat Weaver PhD Student, Computer Science and Eng., OGIPhD Student, Computer Science and Eng., OGI

Forest/environmental expertiseForest/environmental expertise Computer science expertiseComputer science expertise

Staff Scientist, Pacific Northwest National LaboratoryStaff Scientist, Pacific Northwest National LaboratoryMark WhitingMark Whiting

Science Advisor, USDI, National Park ServiceScience Advisor, USDI, National Park ServiceRegina RochefortRegina Rochefort

Communications Director, USDA Forest Service, PNW Communications Director, USDA Forest Service, PNW Research StationResearch StationCynthia L. MinerCynthia L. Miner

Chief, Office of Technical Support, Forest Resources, Chief, Office of Technical Support, Forest Resources, USDI Fish and Wildlife ServiceUSDI Fish and Wildlife ServiceMonty KnudsenMonty Knudsen

Executive Director, IMFN SecretariatExecutive Director, IMFN SecretariatFred JohnsonFred Johnson

MD, Asst. Professor, Division of Medical Informatics and MD, Asst. Professor, Division of Medical Informatics and Outcomes Research, OHSUOutcomes Research, OHSU Paul GormanPaul Gorman

Sustainable NorthwestSustainable NorthwestMartin GoebelMartin Goebel

USDA Forest Service, Pacific NW RegionUSDA Forest Service, Pacific NW RegionRobert DevlinRobert Devlin

President, IUFRO, Oxford Forestry Institute, Dept of Plant President, IUFRO, Oxford Forestry Institute, Dept of Plant SciencesSciencesJeff BurleyJeff Burley

Co-Inventor of the Topic Map ModelCo-Inventor of the Topic Map ModelMichel BiezunskiMichel Biezunski

Advisory BoardAdvisory Board

Forest/environmental expertiseForest/environmental expertise Computer science expertiseComputer science expertise

The Research: Reuse Human AttentionThe Research: Reuse Human Attention

19921992bird sightings bird sightings at National Parkat National Park

19951995migration patternsmigration patternsof two particular speciesof two particular species(painstakingly identified(painstakingly identifiedspecies for bird sightings)species for bird sightings)

study

20002000you need a survey of all species - for ayou need a survey of all species - for abiodiversity studybiodiversity studythe classification of the species of the classification of the species of the bird sightings are gone!the bird sightings are gone!

Goal: Reuse Human Attention (cont.)Goal: Reuse Human Attention (cont.)

The original bird sightingsThe original bird sightings Two species identified Two species identified (pink & white)(pink & white)

Goal: Reuse Human AttentionGoal: Reuse Human Attention

White bird sightingsWhite bird sightings Pink bird sightingsPink bird sightingsOther bird sightingsOther bird sightings

When you can’t modify the base data:When you can’t modify the base data:Use superimposed informationUse superimposed information

Two species identified (pink & white)

pink birdswhite birds

Things we’ve learned from Task 1 Things we’ve learned from Task 1 NSF Digital GovernmentNSF Digital Government

work is project-basedwork is project-based primary product is information: assessments, primary product is information: assessments,

studies, surveys, environmental impact statementsstudies, surveys, environmental impact statements multiple agencies are involvedmultiple agencies are involved each agency serves as information each agency serves as information gatherergatherer; ;

information information brokerbroker; information ; information consumerconsumer even though information is a primary product, even though information is a primary product,

information technology is secondary (stewardship information technology is secondary (stewardship of the land is the primary mission) of the land is the primary mission)

Table 1. The XML model described in terms of the superimposed-information metamodel. The elements of the XML model (bottom) are instances of the corresponding Metamodel element definitions (top).

Metamodel Elements

Constructs Lexicals Connectors Conformance Connectors

XML Model

Element Type  Attribute Type  Element  Attribute Primitive Content Type Primitive Content

PrimitiveContent

Nested Element Type Connects Two Element Types Nested ElementConnects Two Elements Element ContentConnects an Element to Primitive Content Element Content TypeConnects an Element Type to Prim. Content Type Element AttributeConnects an Element to an Attribute Attribute Element TypeConnects an Element Type to an Attribute Type

Element Instance OfConnects an Element to its Element Type Attribute Instance OfConnects an Attribute to its Attribute Type Content Instance OfConnects Prim. Content to its Primitive Content Type

A Notional View of AMA PortalA Notional View of AMA Portal

Terminology- Based Access

Relevant Information Highlighted(Base Info.)

User-Supplied Annotation

Figure 3: SLIMPad Screenshot showing two marks: one to Excel and one to XML.

TopicType

ttypename : String

<<Mark>> Address

markId : String

topic Type1

1

*

Figure 9. CARTE’s Structured Map model.

TopicInstance

title : String topicInsID : Number

TopicRelType

relType : String

TopicRelInst

AnchorInst AnchorType

anchorRole : String

topic Type2

1

1 1 topic Ins1

topic Ins2

<<conformance>> topic_instOf

<<conformance>> rel_instOf

<<conformance>> anchor_instOf

topicType

* * * *

1

*

1

topicIns

* 1

address

1

1

1 *

*

*

Bundle

bundleName : String bundleXPos : Number bundleYPos : Number

Scrap

scrapName : String scrapXPos : Number scrapYPos : Number

AbstractBundle AbstractTemplate

SlimPad

padName : String

PadTemplate

templateName : String

<<conformance>> pad_instOf

BundleTemplate

bundleTypeName : String

parent Bundle

nested Bundle

parent Template

nested Template

<<Mark>> ScrapMark

markId : String

<<conformance>> bundle_instOf

enclosing Bundle

bundle Content scrap

Mark

1

*

1

*

1

*

* 1

*

*

0..1

0..1

Figure 10. SLIMPad’s Structured Bundles Model.

subBundle

subBundle

subTemplate

subTemplate

BundleTemplate TopicType

a. Construct to Construct

source target

mapping Bundle TopicInstance

b. Instance to Schema Construct

mapping

Bundle TopicInstance

c. Connector to Connector

mapping

BundleTemplate TopicType

topic_instOf bundle_instOf

AbstractTemplate

d. Connector to Construct

PadTemplate BundleTemplate nested Template

parent Template

TopicType

TopicRelType

relType : String

topic Type1

topic Type1

mapping

Figure 13. Inter-model mappings from Structured Bundles to Structured Maps represented visually.

source target

source target source target

Topic Map ExampleTopic Map Example

PaintingPainting PainterPainterby painterby painter

Influenced byInfluenced by

““Captive”Captive” ““Paul Klee”Paul Klee”by painterby painter influenced byinfluenced by

““Francisco de Goya”Francisco de Goya”

““1914”1914”by painterby painter

mentionedmentioned biographybiographyreferencedreferenced

referencedreferencedhttp://...http://...

biographybiography biographybiography

http://...http://...http://...http://...

critiquedcritiqued

critiquedcritiqued

mentionedmentioned

http://...http://...

http://...http://...