2010 10 Sottilare AIIDE Keynote Presentation

U.S. Army Research, Development and Engineering Command

A perspective on the role

of AI and interactive

digital media

in military training

Presentation to the AI and Interactive Digital Entertainment Conference (AIIDE)October 2010Stanford University

Robert A. Sottilare, Ph.D.Chief Technology OfficerSFC Paul Ray Smith Simulation & Training Technology [email protected]

SFC Paul Ray Smith Simulation & Training Technology Center

Disclaimer

• I will be talking about commercial products in addition

to government-developed technology

• What I say should not be construed as an endorsement

of any commercial products

• Much of what will be covered in this talk has application

outside of a military context

• Challenge is the word of the day…


Agenda

• A little about my organization and my background…

• Motivation to exploit AI and IDM for military training

• AI and IDM in serious games and military training

• Opportunities to exploit AI and IDM

• Challenges and Next Steps


My Organization

SFC Paul Ray Smith

Simulation & Training Technology Center

part of the

Army Research Laboratory


A little about me…

• you can find me on

• 25+ years in military training & simulation

research, development and acquisition

• My research interests are in adaptive

tutoring

• my research is conducted in the Learning in

Intelligent Tutoring Environments (LITE) Lab

• and yes… I like the Simpsons

LITE Lab


Motivation to exploit AI and Interactive

Media for Military Training


Motivation to exploit AI and

Interactive Media for Military Training

• U.S. Army is deployed worldwide on a consistent basis

so training technology needs to be:

– accessible wherever the Soldier is located

– interactive regardless of infrastructure available

– intuitive so it can be used in the absence of instructors

• The time allotted for training is limited so training

needs to be:

– efficient as well as effective

– engaging, challenging and relevant to the Soldier’s

mission

– adaptive to the trainee’s needs and capabilities

Today, Soldiers do more than their traditional missions…


Learning & Entertainment

[1] Gagne, R. M. (1965). The conditions of learning. New York: Holt, Rinehart and Winston.

Learning Objectives1

• Gain attention

• Inform

• Stimulate recall

• Present stimulus

material

• Provide learner

guidance

• Elicit performance

• Provide feedback

• Assess performance

• Enhance retention

transfer

LEARNING FUN

Gain/maintain attention

Inform

Stimulate interaction

Stimulate immersion

Motivate

Guide


Training

the process of bringing a person,

team or organization to an

agreed standard of proficiency

by practice and instruction

• relevant task and conditions

• standards

• measures of performance

• constructive and timely feedback


Exploit AI and Interactive Digital

Media to improve Soldier…

• learning: acquisition of knowledge and/or skill

• performance: accomplishment of a given task

measured against some standard

• retention: ability to perform a learned behavior (task)

over time without the reinforcement of practice

• competence: demonstrated level of ability through

performance over time; potential for future performance


Available and emerging

technologies

• intelligent technologies for training

– adaptive tutoring systems

– virtual humans

• serious games

• virtual worlds

• mobile learning


PURPOSE: Design, develop, apply and

evaluate intelligent agent technologies

(tools and methods) to enhance training,

leader development and education; and

to reduce associated support costs

TECHNICAL CHALLENGES: Application

of intelligent agents in ill-defined domains;

understanding natural language in multi-

sided conversations; interrupted

conversations team tutoring; rapid

authoring; and virtual human cognition

Intelligent Technologies for Training

OBJECTIVE: Use intelligent agents to

supplement one-to-one and one-to-many

TLE experiences where human support is

limited, impractical or completely

unavailable

Virtual Humans

Adaptive Tutors


Intelligent Technologies for Training

Virtual HumansAdaptive Tutors

use real behaviors and

physiological

measures to model and

predict trainee

cognitive states

use simulated

cognitive states to

drive virtual human

behaviors


TECHNICAL CHALLENGES: Determining

optimal sensor suite to accurately/passively

identify cognitive state; development of machine

learning models to adapt instruction

Adaptive Tutoring Research

PURPOSE: Enable computer-based intelligent

tutors to adapt to trainee “readiness-to-learn”

by evaluating the influence of performance

variables in the trainee model

OBJECTIVE: Experimentally evaluate trainee

state variables (self-reported, observed,

physiological and behavioral) to accurately/

passively assess the trainee’s cognitive state

& adapt content/interaction to meet trainee

needs

sensors predicted trainee state instructional strategy selection

http://www.emotiv.com/apps/epoc/299/


Individual Tutoring Systems

Sottilare, R. (2010). Toward the Development of an Intelligent Tutoring System for Distributed Team Training through Passive Sensing. In

Proceedings of the 10th Intelligent Tutoring Systems Conference, Pittsburgh, June 2010.


Sottilare, R. (2010). Toward the Development of an Intelligent Tutoring System for Distributed Team Training through Passive Sensing. In

Proceedings of the 10th Intelligent Tutoring Systems Conference, Pittsburgh, June 2010.

Team Tutoring Systems


Adaptive Tutoring Systems

• Few military training simulations today implement intelligent tutoring…

• In most of those cases, the trainee model is mainly focused on performance goals and know little or nothing about:

– trainee cognition (including affect)

– trainee past performance (competence level)

• Many of today’s tutors focus in well-defined domains (e.g. physics, math) and the world is messy…

• The difficult work is ahead of us…


Virtual Humans


Virtual Humans

• What parts of real people are represented in virtual

humans?

– physical appearance, behaviors and interaction

– limited cognition (including affect)

– culture, values?

– goals, beliefs

– social interaction

– trust

– persistence…

Arnold, Lazarus, Frijda, Scherer, Ortony et al.


Appraisal Theory

Arnold, Lazarus, Frijda, Scherer, Ortony et al.

courtesy of Stacy Marsella, Institute for Creative Technologies


Virtual Humans

• Many current and potential applications:

– training partners, adversaries or neutral characters

– tutoring, mentoring, coaching

– marketing

– social support

– entertainment


Virtual Humans

courtesy of Jon Gratch, Institute for Creative Technologies


• Create medically accurate virtual patients

with advanced interaction capabilities

• Train Combat Medics on the proper

procedures for sick call

TECHNICAL CHALLENGES:

• Identifying/creating appropriate and meaningful medical dialog/content

• Developing adaptive, realistic human behaviors

Virtual Patients/Virtual Sick Call


Serious Games


Tactical Combat Casualty Care


PURPOSE:

Develop a capability for sustainment

training that included interrogations of

detainees, civilians and persons of

interest

TECHNICAL CHALLENGES:

• Speech and natural language processing

• Question and Answer type of conversations

• Authoring capability to create new characters

Tactical Questioning System

OBJECTIVE:

• Develop 3D Interactive environment

• Soldier interacts with virtual humans

to sustain their interrogation skills in

realistic manner

• Utilize translator and foreign

languages


Multi-national Training Experiment

using Virtual Battlespace 2

• uses commercial game

• primarily kinetic operations

• distributed shared environment


UrbanSim

PC-based game that allows Battalion and

Brigade Commanders to practice situational

understanding and operations in complex,

contemporary environments.

Key Learning Objectives:

– Develop and maintain situational

understanding

– Develop Commander’s intent

– Develop Commander’s critical information

requirements

– Direct operations

– Anticipate 2nd and 3rd order effects

– Command outside the chain of command


Bi-lateral Negotiation

Simulation (BiLAT)

Challenge – combine pedagogy, game play design, interactiveresearch technologies, and software development to address key objectives

PC, game engine-based, cognitive training tool used

for developing skills in how to plan for and conduct

bi-lateral meetings and/or negotiations

in different cultural settings

Key Learning Objectives:

Negotiation Strategy: Planning and executing

tactics that help achieve mission objectives

Cultural Awareness: Applying culturally-

appropriate interpersonal skills

Trust-Building Strategy: Developing/applying

tactics to gain negotiating partners’ trust


Social and Cultural Modeling

Bi-lateral Negotiation

Simulation (BiLAT)


Virtual Worlds

• Next Generation VW Platforms must be:

– scalable – work across operating systems and devices

– easily deployed

– secure

– open

– interoperable

– future proof - scale dynamically

– economically sound – good business model

David Smith, Chief Innovation Officer, Lockheed Martin Global Training and Logistics

A Virtual Worlds Architecture Framework: A Shorter Path to Mastery for All DOD Personnel


OnLine Interactive

Virtual Environment (OLIVE)


Multinational Training Experiment

using OLIVE


Enhanced Dynamic Geo-social Environment (EDGE)

3

5

Blends traits from MMO, serious games and virtual world technology to replicate the operational environment as accurately as possible.


Federal Virtual Worlds Challenge

• What is the Federal Virtual Worlds Challenge?

• It is an open, global Challenge to the world for the best of

the best implementations within virtual environments.

• The event is conducted to explore innovative and interactive

solutions in virtual environments. The criteria are

intentionally unbounded to encourage creative results.

• This year the focus is on artificial intelligence!

• http://www.fvwc.army.mil/index.php

http://www.fvwc.army.mil/index.php





Mobile Learning


Combat Medic Card Games


Combat Hunter Mobile


Challenges and Next Steps

• Economic challenges

• AI for training challenges

• IDM for training challenges

• Interaction design challenges

• Grand challenges for educational technology

• Next steps for educational technology


Economic Challenges

• The U.S. Army is a large population distributed

worldwide and covering a variety of occupational

specialties

– Maintaining the proficiency of this population is challenging

and expensive

– New missions = new training content

– New personnel and equipment = continuous training

• AI technology (tools and methods) are needed to:

– reduce training costs

– while maintaining training effectiveness

– and improving availability and adaptability


AI for Training Challenges

• Adaptable tutoring systems

– improve perception/prediction of trainee’s state

– improve assessment of learning, performance, competence and retention

– improve selection of appropriate instructional strategies based on the trainee’s state (e.g. competency level)

• content, tempo, challenge, flow and feedback

– automate authoring of content and expert models


IDM for Training Challenges

• Interactive mixed reality

training environments

– everyone wants the holodeck,

but its not economically or

technically feasible now

– explore alternative evolutionary

steps to get holodeck-like

capabilities now

– natural locomotion

– no adverse effects (e.g.

simulator sickness)

– real-time, distributed effects


Interaction Design

Challenges for AI and IDM

• Trainee situational awareness

comparable to “live”

environments

• Passive machine perception

of trainee and “live”

environment

• Natural Language

Understanding & Generation

• Natural locomotion in virtual

environments


Grand Challenges for

Educational Technology

• Personalize Education

• Assess Student Learning

• Support Social Learning

• Diminish Boundaries

• Develop Alternative Teaching Methods

• Enhance the Role of Stakeholders

• Address Policy Changes

Woolf, B. P. (2010). A Roadmap for Education Technology. National Science Foundation # 0637190


Next Steps for

Educational Technology

• User Modeling

• Mobile Training

– Sensing trainee state during individual training

– Interactive team training and performance assessment

• Networking Tools

• Serious Games

• Intelligent Environments

• Educational Data Mining (authoring tools)

– Content

– Models (student models, expert models…)

• Rich Interfaces

Adapted from: Woolf, B. P. (2010). A Roadmap for Education Technology. National Science Foundation # 0637190


Questions

2010 10 Sottilare AIIDE Keynote Presentation

Documents

Transcript of 2010 10 Sottilare AIIDE Keynote Presentation