Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

106
1 Computing for Human Experience Keynote at On-the-Move Federated Conference, October 2011: http://www.onthemove-conferences.org/ Amit Sheth Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled Computing Wright State University, Dayton, OH, USA Special thanks & contributions: Cory Henson

description

"Computing for Human Experience: Semantics empowered Cyber-Physical, Social and Ubiquitous Computing beyond the Web" Keynote at On the Move Federated Conferences, Crete, Greece, October 18, 2011. http://www.onthemove-conferences.org/ Abstract: http://www.onthemove-conferences.org/index.php/keynotes/amitsheth

Transcript of Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

Page 1: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

1

Computing for Human Experience

Keynote at On-the-Move Federated Conference, October 2011: http://www.onthemove-conferences.org/

Amit Sheth

Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, OH, USA

Special thanks & contributions: Cory Henson

Page 2: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

2

with 7 billion people

Our 1 world

living 1 experience at a time

Page 3: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

3

with a stream of experiences to be …

shared

rememberedcultiv

ated

realized enjo

yed

Page 4: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

4

Today, technology increasingly engages individuals, society and humanity with …

5 billionmobile phones

40+ billion mobile sensors

1-2 billioncomputer

s

http://www.gartner.com/it/page.jsp?id=703807

Page 5: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

5

With constant connectivity enabled through global networks

5 billionmobile phones

40+ billion mobile sensors

1-2 billioncomputer

s

Page 6: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

6

So, how can we leverage this tech to improve our experiences

without losing ourselves in the process?

Page 7: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

7

Or drown in data?

Page 8: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

8

From machine-centric to human-centric design

Machines to accommodate our experiences,as opposed to the other way around

Computing to liberate

Page 9: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

9

To accomplish this requires a fundamental shift in how we interact and communicate with computational machines

We must take a more holistic view of computation,as a shared universe, populated by people and machines working in harmony to achieve our highest aspirations

Page 10: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

10

We have caught glimpses of this vision …

Man-Machine Symbiosis – T. O’Reilly

Humanist Computing – J. Rossiter

Memex – V. Bush

Ambient Intelligence – E. Zelkha, B. Epstein

Experiential Computing – R. Jain

Ubiquitous Computing – M. Weiser

Page 11: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

11

Let this affect be positive

The ways in which technology and humansinteract are fundamentally changing

In turn, our (human) experiences are changing

Our activities, decisions, thoughts, and feelingsare affected by the ubiquitous integration of technology into the fabric of our lives

Page 12: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

12

Computing for Human Experience

Amit Sheth

Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, OH, USA

Special thanks & contributions: Cory Henson

Page 13: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

Amit Sheth

Ashutosh Jadhav

Hemant Purohit

Vinh Nguyen

Michael Cooney

Lu ChenPavan

KapanipathiPramod

Anantharam

Sujan Perera

Alan Smith

Pramod Koneru

Maryam Panahiazar

Sarasi Lalithsena Prateek Jain

Matthan Sink

Cory Henson

Ajith Ranabahu

Kalpa Gunaratna

Delroy Cameron

Sanjaya Wijeratne

Wenbo Wang

Page 14: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

14

Human capabilities such as sensing, perception, attention, memory, decision making, control, etc.

CHE is an approach to improving the human condition through computational means, and with minimal burden

This may be achieved through the contextual assistance, augmentation, and absolution of human capabilities

Page 15: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

15

Consider the following example …

Page 16: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

A cross-country flight from New York to Los Angeles on a Boeing 737 plane generates a massive 240 terabytes of data

- GigaOmni Media

16

Page 17: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

17

But, how much data is generated regarding the health and well-being of the pilot or passengers?

zero, none, zilch!

Page 18: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

18

Image the ability to monitor and control our health with the same care and precision that goes into the 737.

And not just providing doctors with such control, but you and me.

Page 19: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

19

Health information is now available from multiple sources

• medical records• background knowledge • social networks• personal observations • sensors• etc.

Page 20: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

20

Sensors, actuators, and mobile computing are playing an increasingly important role in providing data for early phases of the health-care life-cycle

This represents a fundamental shift: • people are now empowered to monitor and manage their own health; • and doctors are given access to more data about their patients

Page 21: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

21

Unfortunately, when personal health data is collected and presented, it often looks like this … gibberish.

Page 22: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

22

Health Metrics with Meaning

Personal Health Dashboard

What is needed is a more intuitive and intelligent representation of our health.

Image: http://bit.ly/lV2V73

Page 23: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

23

How is this accomplished?

Page 24: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

24

• Integration of heterogeneous, multimodal data• Bridging the physical-cyber-social divide• Elevating abstractions that machines and people understand• Semantics at an extraordinary Scale

These enablers are brought together through Semantic Web technologies

Key Enablers of CHE

Page 25: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

25

Foundation on which these enablers stand

Page 26: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

26

Foundation on which these enablers stand

Page 27: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

27

Integration of heterogeneous, multimodal data

Page 28: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

28

Background Knowledge: ontologies, knowledge bases, LOD, databases, etc.

Social/Community Data: social network data, wisdom of the crowds, etc.

Sensor Data: observations from machine sensors, citizen sensors (i.e., patients, doctors), laboratory experiments, etc.

Personal Context: location, schedule, items (e.g., accessible sensors), etc.

Personal Medical History: Electronic Medical Records, Personal Health Records, Patient Visit Records, etc.

Integration of heterogeneous, multimodal data

Page 29: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

29

Communications using online technologies to share opinions, insights, experiences and perspectives with each other.

What is Social Media?

Page 30: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

30

Blogs – DiabetesMine, HealthMatters, WebMD, NYT HealthBlog, etc.

Microblogs – Livestrong, Stupid Cancer, etc.

Social Networks – OrganizedWisdom, PatientsLikeMe, DailyStrength, NursesRecommendDoctors, CureTogether, etc.

Podcasts – John Hopkins Medical Podcasts, Mayo Clinic, etc.

Forums – Revolution Health Groups, Google Health Groups, etc.

Popular types of Healthcare Social Media

Page 31: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

31http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462

60% of physicians either use or are interested in using social networks

Page 32: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

32http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462

People are turning to each other online to understand their health

Page 33: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

33

83% of online adults search for health information

http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462

Page 34: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

34

83% of online adults search for health information

60% of them look for the experience of “someone like me”

http://www.slideshare.net/IQLab/social-media-101-for-pharma-3494462

Page 35: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

35

"I don't know, but I can try to find out" is the default setting for people with health questions.

Savannah Fox, The Social Life of Health Information, Pew Internet Report, May 12, 2011. Available at http://www.pewinternet.org/Reports/2011/Social-Life-of-Health-Info.aspx

"I know, and I want to share my knowledge" is the leading edge of health care.

Page 36: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

36

Intra Community Activity and connectivity– How well connected are individual nodes (People)– What keeps them strongly connected over time

(Relationship types - Knowledge of Content)

Inter-Community Connectivity• Any bridges to connect to the other community? (People)• Any Similarity in actions with the other community (Can

Content help?)Image: http://themelis-cuiper.com

Will the two communities coordinate well during an event- crisis or disaster?• Interplay between all three dimensions

– P, C, N

People-Content-Network Analysis

For more info: http://www.slideshare.net/knoesis/understanding-usercommunity-engagement-by-multifaceted-features-a-case-study-on-twitter

Page 37: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

37

People-Content-Network Analysis

External Knowledge bases

Dynamic Domain Model for the event

Event oriented Community

Social Network

Mined User Interests and User Types

User Profiles

SEMANTIC ASSOCIATION TO UNDERSTAND ENHANCED ENGAGEMENT LEVEL

Page 38: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

38

Bridging the physical-cyber-social divide

Computation is no longer confined to pure symbol manipulation. The previously strict relations between the digital and physical world are blurring.

Page 39: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

39

Psyleron’s Mind-Lamp (Princeton U), connections between the mind and the physical world.

Neuro Sky's mind-controlled headset to play a video game.

MIT’s Fluid Interface Group: wearable device with a projector for deep interactions with the environment

Bridging the physical-cyber divide

Page 40: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

40

Foursquare is an online application which integrates a persons physical location and social network.

Bridging the physical-cyber-social divide

Community of enthusiasts that share experiences of self-tracking and measurement.

FitBit Community allows the automated collection and sharing of health-related data, goals, and achievements

Page 41: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

41

Tweeting Sensorssensors are becoming social

Bridging the physical-cyber-social divide

Page 42: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

42

Bridging the physical-cyber-social divide

Select topicSelect date

Topic tree

Spatial Marker

N-gram summaries

Wikipedia articles

Reference news

Related tweets

Images & Videos

Tweet trafficSentiment AnalysisNetwork

Analysis

Community Interaction of Various user

types

For more info: http://twitris.knoesis.org/

Page 43: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

43

Heliopolis is a suburb of

Cairo.

Dynamic Model Creation

Continuous Semantics

Page 44: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

44

Even

ts

“Both Ahmadinejad & Mousavi declare victory in Iranian Elections.”

“situation in tehran University is so worrisome. police have attacked to girls dormitory #tehran #iranelection”

“Reports from Azadi Square - 4 people killed by police, people killed police who shot. More shots being fired #iranelections”June 12 2009 June 13 2009 June 15 2009

Key p

hra

ses

Mod

els

Ahmadinejad & Mousavi are politicians in

Iran

Tehran University is a University in

Iran

Azadi Square is a city square in

Tehran

Dynamic Model Creation

Page 45: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

45

The design and building of physical-cyber-social systems requires effective conceptualization and communication between people and machines.

To reach this vision requires advancement in the area of machine perception, enabling machines the ability to abstract over low-level observations.

Page 46: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

46

Abstraction

Abstraction provides the ability to interpret and synthesize information in a way that affords effective understanding and communication of ideas, feelings, perceptions, etc. between machines and people.

Page 47: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

47

The process of interpreting stimuli is called perception; and studying this extraordinary human capability can lead to insights for developing effective machine perception.

People are excellent at abstraction; of sensing and interpreting stimuli to understand and interact with the world.

Page 48: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

“real-world”

conceptualization of “real-world”

Sensor

Sensor Data / Social Data

observation

perception

Physical

Cyber

Social

48

Page 50: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

50

Sensor and Sensor Network (SSN) Ontology

http://www.w3.org/2005/Incubator/ssn/XGR-ssn/

Page 52: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

52

http://linkedsensordata.com

Page 53: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

53

With the help of sophisticated inference, both people and machines are also capable of perceiving entities, such as apples.

• the ability to degrade gracefully with incomplete information

• the ability to minimize explanations based on new information

• the ability to reason over data on the Web

• fast (tractable)

perceivesEntityPerceiver

Page 54: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

54

minimizeexplanations

degrade gracefully

tractable

Web reasoning

Abductive Logichigh complexity

Deductive Logic (e.g., OWL)(relatively) low complexity

Perceptual Inference(i.e., abstraction)

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth, Pascal Hitzler. Representation of Parsimonious Covering Theory in OWL-DL. Proceedings of the 8th International Workshop on OWL: Experiences and Directions (OWLED 2011), San Francisco, CA, USA, June 5-6, 2011.

Page 55: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

55

• Goal is to account for observed symptoms with plausible explanatory hypotheses (abductive logic)

• Driven by background knowledge modeled as a bipartite graph causally linking disorders to manifestations

Yun Peng, James A. Reggia, "Abductive Inference Models for Diagnostic Problem-Solving"

m1

m2

m3

d1

d2

d3m4

disorder manifestationcauses

explanationobservations

Parsimonious Covering Theory

Page 56: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

56

PCT Parsimonious Cover

• coverage: an explanation is a cover if, for each observation, there is a causal relation from a disorder contained in the explanation to the observation

• parsimony: an explanation is parsimonious, or best, if it matches some criteria of suitability (i.e., single disorder assumption)

Page 57: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

57

Given

PCT problem P is a 4-tuple D, M, C, Γ⟨ ⟩

• D is a finite set of disorders• M is a finite set of manifestations• C is the causation function [C : D Powerset(M)]⟶• Γ is the set of observations [Γ M ]⊆

Δ is a valid explanation (i.e., is a parsimonious cover)

Goal

Translate P into OWL, o(P), such that o(P) ⊧ Δ

Convert PCT to OWL

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth, Pascal Hitzler. Representation of Parsimonious Covering Theory in OWL-DL. Proceedings of the 8th International Workshop on OWL: Experiences and Directions (OWLED 2011), San Francisco, CA, USA, June 5-6, 2011.

Page 58: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

58

headache

extreme exhaustion

severe ache and pain

stuffy nose

sneezing

sore throat

severe cough

mild ache and pain

mild cough

flu

cold

fever

disorder manifestationcauses

PCT Background

Knowledge in OWL

disorders (D)

for all d ∈ D, write d rdf:type Disorderex: flu rdf:type Disorder

cold rdf:type Disorder

manifestations (M)

for all m ∈ M, write m rdf:type Manifestation

ex: fever rdf:type Manifestationheadache rdf:type Manifestation …

causes relations (C)

for all (d, m) ∈ C, write d causes mex: flu causes fever

flu causes headache …

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth, Pascal Hitzler. Representation of Parsimonious Covering Theory in OWL-DL. Proceedings of the 8th International Workshop on OWL: Experiences and Directions (OWLED 2011), San Francisco, CA, USA, June 5-6, 2011.

Page 59: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

59

observations (Γ)

for mi ∈ Γ, i =1 … n, write

Explanation owl:equivalentClasscauses value m1 and … causes value mn

ex: Explanation owl:equivalentClasscauses value sneezing andcauses value sore-throat causes value mild-cough

explanation (Δ)

Δ rdf:type Explanation, is deduced

ex: cold rdf:type Explanationflu rdf:type Explanation

and

PCT Observations and

Explanations in OWL

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth, Pascal Hitzler. Representation of Parsimonious Covering Theory in OWL-DL. Proceedings of the 8th International Workshop on OWL: Experiences and Directions (OWLED 2011), San Francisco, CA, USA, June 5-6, 2011.

Page 60: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

60

The ability to perceive efficiently is afforded through the cyclical exchange of information between observers and perceivers.

Traditionally called the Perception Cycle

(or Active Perception)

sendsfocus

sends observation

Observer

Perceiver

Page 61: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

61

Nessier’s Perception Cycle

Page 62: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

62

Cognitive Theories of Perception (timeline)

1970’s – Perception is an active, cyclical process of exploration and interpretation. - Nessier’s Perception Cycle

1980’s – The perception cycle is driven by background knowledge in order to generate and test hypotheses. - Richard Gregory (optical illusions)

1990’s – In order to effectively test hypotheses, some observations are more informative than others. - Norwich’s Entropy Theory of Perception

Page 63: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

63

Key Insights

• Background knowledge plays a crucial role in perception; what we know (or think we know/believe) influences our perception of the world.

• Semantics will allow us to realize computational models of perception based on background knowledge.

• Internet/Web expands our background knowledge to a global scope; thus our perception is global in scope

• Social networks influence our knowledge and beliefs, thus influencing our perception

Contemporary Issues

Page 64: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

64

observes

inheres in

Integrated together, we have an general model – capable of abstraction – relating observers, perceivers, and background knowledge.

perceives

sendsfocus

sends observation

Observer Quality

EntityPerceiver

Page 65: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

65

Modeled in set-theoretic notation with components mapped to Parsimonious Covering Theory and OWL

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, 2011 (accepted).

Page 66: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

66

Applications of

HealthcareWeather Rescue

Page 67: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

67

Weather Application

Detection of events, such as blizzards, from weather station observations on LinkedSensorData

Weather Application

Demos: Real-Time Feature Streams

Page 68: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

68

Weather ApplicationSECURE: Semantics Empowered Rescue Environment

Rescue robots detect different types of fires, which may require different methods/tools to extinguish, and relays this knowledge to first responders.

Demo: SECURE: Semantics Empowered Rescue Environment

Page 69: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

69

Weather ApplicationHealthcare Application

Detection of errors in Electronic Medical Records and missing knowledge in a cardiology domain model

EMR

Page 70: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

70

Weather ApplicationHealthcare Application

EMR: "Her prognosis is poor both short term and long term, however, we will do everything possible to keep her alive and battle this infection."

SNM:40733004_infection SNM:68566005_infection_urinary_tract

A syntax based NLP extractor (such as Medlee) can extract this term and annotate as SNM:40733004_infection

By utilizing IntellegO and cardiology background knowledge, we can more accurately annotate the term as SNM:68566005_infection_urinary_tract

without IntellegO

with IntellegO

Problem Problem

Page 71: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

71

Weather ApplicationHealthcare Application

EMR: ”The patient is to receive 2 fluid buloses."

SNM:32457005_body_fluid

A syntax based NLP extractor (such as Medlee) can extract this term and annotate as SNM:32457005_body_fluid

without IntellegO

Problem

Fluid is part of buloses treatment, not a problem

with IntellegO

By utilizing IntellegO and cardiology background knowledge, we can determine that this is an incorrect annotation.

Treatment

Page 72: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

72

In 2008, the rate of data generation surpassed storage capacity. With 7 billion people, and a growing number of sensors, how can such a such a system scale? By shining a light on relevant human experience, supported by knowledge, while dimming the minutia of data.

Semantic Scalability

http://gigaom.com/cloud/sensor-networks-top-social-networks-for-big-data-2

Page 73: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

73

Semantic Scalability

ex. – keyword based search/index, non-textual data, multimodal data for an event

It is clear that purely syntax-based solutions will not scale

Page 74: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

74

Semantic Scalability

1. Focusing attention on important information and ignoring irrelevant data

2. Converting low-level data (observations) to high-level knowledge (abstractions)

3. Utilizing CHE technology to more evenly distribute responsibility and activities among people and machines

Path to Web scale semantics

Page 75: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

75

We were able to demonstrate 50% savings in sensing resource requirements during the detection of a blizzard.

1. Focusing attention on important information and ignoring irrelevant data

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, 2012. (accepted)

Page 76: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

76

2. Converting low-level data to high-level knowledge (observations to abstractions)

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, 2012. (accepted)

Experiment – during a blizzard, we utilized Intelleg0 to collect and analyze over 110,000 sensor observations, from:

• 800 weather stations (~5 sensors per station)• across 5 states (Utah, Nevada, Colorado, Wyoming, and

Idaho) • for 6 days (April 1 – 6, 2003)

Page 77: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

77

2. Converting low-level data to high-level knowledge (observations to abstractions)

We were able to demonstrate an order of magnitude resource savings between storing observations vs. relevant abstractions

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, 2012. (accepted)

Page 78: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

78

2. Converting low-level data to high-level knowledge (observations to abstractions)

Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web. Applied Ontology, 2012. (accepted)

While this is a good result, the benefit provided for a single person – a single experience – is far more dramatic.

Page 79: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

There are almost 7 billion people on earth, and only ~10-15 million doctors (~700:1 - 467:1)

3. Utilizing CHE technology to more evenly distribute responsibility and activities among people and machines

79

Page 80: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

80

Providing people with the tools to monitor and manage their own health will dramatically reduce the burden on doctors, and improve the health of the people

These doctors are severely overburdened, answering less than 60% of questions posed by patients regarding their health and well being

Page 81: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

81

The health-care ecosystem of the future includes

machines, people, and social networks continuously, ubiquitously, and unobtrusively

monitoring and managing our health

Page 82: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

82

CHE approach to Health Care

1 2 3

Continuous Monitoring Personal Assessment Medical Service

Auxiliary Information – background knowledge, social/community support, personal context, personal medical history

Page 83: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

83

Continuous Monitoring Phase

Monitoring health metrics and vital signs utilizing unobtrusive body sensors

Continuously collecting information, watching for worrisome symptoms

Page 84: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

84

Personal Assessment Phase

Assessment of symptoms from personal observation and/or health sensorsavailable at home.

Utilizing background knowledge, personal medical history, and current sensor data to formulate and ask specific questions of patient that will aid in explaining symptoms.

Page 85: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

85

Medical Services Phase

Assessment of symptoms gathered from continuous and personal phases, with additional sophisticated equipment, advanced treatment, and specialized medical knowledge not previously available.

Utilizing background knowledge, personal medical history, and current sensor data to formulate a diagnosis.

Page 86: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

86

Continuous Personal Medical

Personal Medical History(e.g., Electronic Medical Record,genomic sequence)

Background Knowledge(e.g., Ontologies, Knowledgebases)

Auxiliary Information

Symptoms/Explanations

Symptoms/Explanations

access & update (PHR)

access &update (PHR)

access & update (EMR)

CHE approach to Health Care

Personal Context(e.g., available sensors,location, schedule)

Social/Community Support(e.g., Patient Network, crowd sourcing)

Page 87: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

87

Continuous Monitoring Phase: Example

• Abnormal heart rate• Clammy skin

• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic Shock

• Check phone for instructions• Patient has history of Heart Disease

Observed Symptoms Possible Explanations

Electronic Medical Record

Health Alert

Page 88: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

88

Basis is a wrist-watch that also monitors pulse rate, movement, temperature, and galvanic skin response.

Continuous Phase Technology

Page 89: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

89

Fitbit Tracker uses a MEMS 3-axis accelerometer that measures your motion patterns to tell you your calories burned, steps taken, distance traveled, and sleep quality.

Continuous Phase Technology

Page 90: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

90

Personal Assessment Phase: Example

Are you feeling lightheaded?

Are you have trouble taking deep breaths?

yes

yes

1. Take medication: Methimazole2. See doctor: how about Tues. @ 11am?

• Patient has history of Hyperthyroidism• Patient has prescription for Methimazole

Have you taken your Methimazole medication?

Do you have low blood pressure when standing?

yes

• Abnormal heart rate• Clammy skin• Lightheaded• Trouble breathing• Low blood pressure

• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic Shock

Observed Symptoms Possible Explanations

Electronic Medical Record

Health Alert

no

Page 91: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

91

Lark is a sleep sensor that monitors circadian rhythms and functions as an "un-alarm," vibrating to wake you at a point in your sleep cycle when you feel alert, not groggy.

Personal Phase Technology

Page 92: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

92

Instant Heart Rate takes your pulse when you place your finger over your phone’s camera lens. The app uses light from the camera flash to detect color changes caused by blood moving through your finger.

Personal Phase Technology

Page 93: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

93

Telcare makes a blood glucose meter (right) for diabetics that broadcasts readings to a mobile-phone app (center) where patients can see results and set goals.

Personal Phase Technology

Page 94: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

94

iBGStar is a plug-in glucose meter for the iPhone, developed by Sanofi-Aventis, providing the ability for patients to monitor and manage Diabetes.

Personal Phase Technology

Page 95: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

95

Withings Blood Pressure Monitor provides easy and convenient blood pressure readings in the convenience of home.

Personal Phase Technology

Page 96: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

96

WebMD provides a wealth of health information and an application to diagnose symptoms.

Personal Phase Technology

Page 97: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

97

Medical Services Phase: Example

• Patient has history of Hyperthyroidism• Patient has prescription for Methimazole

Are your blood sugar levels low?

• Abnormal heart rate• Clammy skin• Lightheaded• Trouble breathing• Low blood pressure

• Hypoglycemia• Hyperthyroidism

Observed Symptoms Possible Explanations

Electronic Medical Record

Page 98: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

98

Doctor.

Medical Phase Technology

Page 99: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

99

Health Guard provides a secure way to store and analyze health records for casual browsing or emergency use (i.e., MS Health Vault records).

Medical Phase Technology

Page 100: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

100

Mobile MIM gives physicians a sophisticated, hands-on mobile system for viewing and annotating radiology images, such as CT scans.

Medical Phase Technology

Page 101: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

101

Dr. Watson is a health and medical question and answering system developed by IBM, utilizing supercomputer intelligence for medical diagnostics.

Medical Phase Technology

Page 102: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

102

Continuous Personal Medical

Personal Medical History(e.g., Electronic Medical Record,genomic sequence)

Background Knowledge(e.g., Ontologies, Knowledgebases)

Auxiliary Information

Symptoms/Explanations

Symptoms/Explanations

access & update (PHR)

access &update (PHR)

access & update (EMR)

CHE approach to Health Care

Personal Context(e.g., available sensors,location, schedule)

Social/Community Support(e.g., Patient Network, crowd sourcing)

Page 103: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

103

Improving the experience of health-care improves all other experiences

CHE holds the potential to revolutionize the practice of health-care by embracing the relationship between ourselves, our machines, and our health

Page 104: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

104

“The most profound technologies are those that

disappear. They weave themselves into the fabric of

everyday life until they are indistinguishable from

it.” – M. Weiser

Physical-Cyber-Social

Abstraction

Integration Scalability

Page 105: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

105

thank you, and please visit us at

http://knoesis.org

Kno.e.sis – Ohio Center of Excellence in Knowledge-enabled ComputingWright State University, Dayton, Ohio, USA

More: Vision Paper: Computing for Human Experience:http://wiki.knoesis.org/index.php/Computing_For_Human_Experience

Computing for Human Experience

Page 106: Computing for Human Experience [v4]: Keynote @ OnTheMove Federated Conferences

Image credits:

Page 2:http://blogs.ebrandz.com/tag/social-networks/http://news.cnet.com/8301-13579_3-10141672-37.htmlhttp://www.thehindu.com/opinion/columns/sainath/article123884.ecehttp://www.samplestuff.com/2011/06/11/frugal-family-fun-nightsPage 3:http://www.naturewalls.org/category/stream/Page 4:http://www.cbi.umn.edu/about/babbage.htmlhttp://news.cnet.com/8301-13579_3-10141672-37.htmlhttps://fpm-www3.fpm.wisc.edu/safety/occupationalHealth/Ergonomics/LaptopErgonomics/LaptopErgonomicsQuickReference/tabid/106/Default.aspxPage 5:http://news.cnet.com/8301-13579_3-10141672-37.htmlhttps://fpm-www3.fpm.wisc.edu/safety/occupationalHealth/Ergonomics/LaptopErgonomics/LaptopErgonomicsQuickReference/tabid/106/Default.aspxhttp://jimilocker.info/?p=165Page 22:http://www.healthsciencestrategy.com/2011/04/will-mhealth-apps-and-devices-empower-epatients-for-wellness-and-disease-management-a-case-study-2/Page 27:http://www.fitbit.com/http://www.relativitycorp.com/socialnetworkmarketing/http://heinrich.house.gov/index.cfm?sectionid=11&parentid=2&sectiontree=2,11&itemid=131Page 46:http://depositphotos.com/4946293/stock-photo-Abstract-earth-puzzle.htmlPage 47:http://massthink.wordpress.com/2007/06/10/husserl-in-indubitable-response-to-descartes-and-kant/Page 61:http://www.ida.liu.se/~eriho/COCOM_M.htmhttp://www.idemployee.id.tue.nl/g.w.m.rauterberg/lecturenotes/ucd%20lecture-3/sld019.htmPage 79:http://thewvsr.com/doctors.htm