SSC San Diego … on Point and at the Center of C4ISR

Distributed Multimedia Distributed Multimedia Systems 07Systems 07

The Future of Visual and Iconic The Future of Visual and Iconic Language Applications and Language Applications and Inscription Technology: Inscription Technology:

A New Way to ChatA New Way to Chat

Distributed Multimedia Distributed Multimedia Systems 07Systems 07

The Future of Visual and Iconic The Future of Visual and Iconic Language Applications and Language Applications and Inscription Technology: Inscription Technology:

A New Way to ChatA New Way to Chat

LorRaine Tauchi Duffy, PhDSPAWARSYSCEN San Diego 5361

LorRaine.Duffy@navy.milSept 6 2007

SSC San Diego…on Point and at the Center of C4ISR2 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

SSC San Diego TEAMSSC San Diego TEAM

• LorRaine Duffy, PhD. Code 5361, LorRaine.Duffy@navy.mil• Emily Wilson Medina, Code 2734, ewilson@spawar.navy.mil• Sunny Fugate. Code 2725,, fugate@spawar.navy.mil• Omar Amezcua, Code 536204, omar.amezcua@navy.mil• Gary Rogers, Code 536204, rogers@spawar.navy.mil• Nghia Tran, Code 23711, nghia.tran@navy.mil• Hoa Phan, Code 2856, hoa.phan@navy.mil• Vincent Dinh, PhD. Code 2856, dinhvv@spawar.navy.mil• Marion Ceruti, PhD. Code 536, Marion.ceruti@navy.mil• Erin O’Conner, San Diego State Student Foundation

SSC SD S&T Grand Challenge: A 5 year commitment to technology development

SSC San Diego…on Point and at the Center of C4ISR3 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

TacticalTactical Situation Assessment Situation Assessment (SA)(SA)

Background Background• Based on text chat: Primary and often singular means of communicating situation updates on intermittent/ discontinuous networks:– Strikegroups: Establish hundreds of chatrooms, 2-4000 users in each “channel, ”based on functional roles; establish at beginning of mission, continuing for >18 months

– Joint/Coalition Forces: Establish chatrooms with joint/coalition service participation, based on mission objectives; allows joint access to service-specific chatrooms to maintain non-intrusive situation awareness of service-specific activities

• Functions: – Preserves interaction history– Supports edge users (on low/intermittent b/w)– Immediate COP (common operational picture) change and context updates

– General information sharing updates on regular basis (across months) to establish battle rhythm management;

– Supports Cross-Domain operations; those without the COP

SSC San Diego…on Point and at the Center of C4ISR4 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

What Chat Looks Like TodayWhat Chat Looks Like Today

R xxxxxxZ OCT 05

FM COMCARSTRKGRU TWELVE

TO USJFCOM

USPACOM

ALLIED COMMAND TRANSFORMATION

AC2C4ISR LANGLEY VIRGINIA

DISA WASHINGTON DC

SPAWAR SYSTEMS CENTER SAN DIEGO C

NAVAL POSTGRADUATE SCHOOL MONTERY CA

USS ENTERPRISE

USS ANZIO

INFO COMMANDER NAVAL NETWORK WARFARE COMMAND

BT

MSGID/GENADMIN/COMCARSTKGRU TWELVE/-/OCT//

SUBJ/WELL DONE//

RMKS/1. I WISH TO SEND NOTE OF PERSONAL THANKS FOR YOUR

SUPPORT OF OUR JOINT/COALITION OPEN STANDARDS TACTICAL

CHAT DEMONSTRATION ON 19 OCT 05. YOUR ORGANIZATIONS

PROVIDED EXCEPTIONAL SUPPORT IN TERMS OF ENGINEERING,

EQUIPMENT TO HOST THE TEST, AND PERSONNEL TO PARTICPATE

AND PROVIDE FEEDBACK. THE TEST WENT EXCEEDINGLY WELL

AND WOULD NOT HAVE BEEN POSSIBLE WITHOUT YOUR ASSISTANCE

AND DEDICATION. SPECIAL APPRECIATION GOES TO:

The 50’s Defense Messaging (and Email):

Produced Time-late

Difficult to read

Not used for real-time ops, but as formality

Example Chat Log:

Produced in real time

Still difficult to read

SSC San Diego…on Point and at the Center of C4ISR5 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Challenge: Creating the Challenge: Creating the S&TS&T

• Objective: Enable Rapid Situation Assessment Updates through mixture (not engineering integration) of technologies (multi-disciplinary effort) in an area that is relatively new, but ubiquitous (and not very well respected…)

• What is chat going to look like in 10 years? A New “’leet speak”– Must not be solely text-driven (too slow; too ambiguous); focus on

visual and iconic images and gestures, integrated with text, I-18-N (Internationalization)

– Must be able to accommodate edge (tactical) users• Chemical-Biological Warfighters in MOPP (Mission Oriented Protective

Posture) gear• Special Operations in Inhospitable Settings: oil spills, fire, tsunamis,

oxygen-deprived environments– Must be able to integrate with geographic land-based terrain maps

AND non-geographic computer-based “terrain” (global network ops-network topology)

– Must be able to incorporate object relationships and time in providing more precise context, mor complex information in smaller “space”

– Must be small (in bandwidth) and robust (survive network drops)– Not necessarily keyboard-driven, point and click, nor restricted to

PDAs• Embody the challenge of an NBIC (Nanotechnology/Biotechnology/

Information Systems/Cognitive Sciences) approach to development• Need to stay independent of commercial developments

SSC San Diego…on Point and at the Center of C4ISR6 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Technical Approach: 3 Development Technical Approach: 3 Development Areas Areas

• Linguistic Research & Analysis– Improve the ability to mathematically derive the content of current chat messages, by defining candidate linguistic “themes/words” of military language, for transition to an augmented icon-based language, to improve efficiency of knowledge sharing

• Visual and Iconic Language– Develop an iconic/visual (gesture) language to augment text, that can be used to more quickly communicate complex relationships and the evolution of relationships of objects of interest in geographic and non geographic environments; a true situation assessment “leet speak”

• Technology of Inscription – Prototype a keyboard-independent technology of inscription for this new language, for use in a Net-Centric environments

SSC San Diego…on Point and at the Center of C4ISR7 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Linguistic Research & Linguistic Research & AnalysisAnalysis

• Analyzing chatroom text represents a subset of a larger body of research : unstructured text analysis– Internet documents, blogs, emails, chatrooms

• Military usage of chat is widespread, but inadequate for typical information retrieval and management

• Statistical analysis of chat--its structure, topical organization, and user trends--must first be applied, few chatroom metrics exist

• Acquire sizeable corpus of chat content for chat user database (domain specific/general; millions of lines of chat are required)

• Statistically reduce content to significant words, phrases, concepts (within domain and in general)

• Produce candidate list for iconic/visual representation

SSC San Diego…on Point and at the Center of C4ISR8 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Visual And Iconic LanguageVisual And Iconic Language

• Goal is to build a mixed-media situation assessment-specific chat-based lexicon (integrating words, pictures, gestures, with hardware), based on linguistic analysis

• In order to:– Increase information carrying capacity – Increase expressiveness of the chat medium– Decrease ambiguity by using a formal iconography and shared context– Increase the ability to reliably share the same context

• How can we best augment and disambiguate textual content using visual representations of language, objects, and information?

SSC San Diego…on Point and at the Center of C4ISR9 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Visual and Iconic Language Visual and Iconic Language (cont)(cont)

Method:• Gather knowledge of existing iconic language research, development, & design, both military and non-military

• Develop software to support visual language communication

• Develop icon-set based on linguistics results

• Expand visual language dictionary and insert standard icon sets and acronyms

• Choose gesture set and develop a visual mapping to items in the dictionary

• Test icon suitability• Visual And Icon Language (VAIL) Conference, Aug 2-3, 2007: :  http://cs.unm.edu/~vail

SSC San Diego…on Point and at the Center of C4ISR10 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Visual And Iconic LanguageVisual And Iconic Language

SSC San Diego…on Point and at the Center of C4ISR11 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Technology of InscriptionTechnology of Inscription

Prototype a keyboard-independent language-entry technology for net-centric warfare– Phase 1: Wireless data glove – Phase 2 : Visual & Icon-based language integrated/gesture-smart glove with 3-D modeling (requiring knowledge of biomechanics)

– Phase 3: Miniaturization via nanotechnology-based circuitry, novel data entry technology

• Keeping in mind that Nintendo Wii and Microsoft’s “Table Interface” will dominate commercial market

SSC San Diego…on Point and at the Center of C4ISR12 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

Many Parallel Developments to Many Parallel Developments to Inscription TechnologiesInscription Technologies

RONI GALGANO / Union-Tribune Evan Morikawa, a senior at High Tech High School, was one of four winners in the Greater San Diego Science & Engineering

Fair for creating a chording keyboard glove that sends wireless signals to his personal digital assistant. Minority Report © 20TH Century Fox

© SSC SD 2006

Nintendo Wii

Microsoft Surface Table

SSC San Diego…on Point and at the Center of C4ISR13 DMS 07Sept 06, 2007 © SSC SD Duffy, LorRaine 09/01/2007

SummarySummary

• 3 Areas of Research Encompassing NBIC principles

• Multidisciplinary Team:– computational linguist, software architect, computer programming/science, electrical engineers, physicist, psychologist, 3-D modeler, chemist, iconic artist…

• Long Range Applied Goals Drive Integrated Knowledge for unique direction and outcome– Not commercially driven, like business-based research

– Not long-range, single-focus science, as in academics

– Not a trivial problem, but a unique one…