DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things

When Artificial Intelligence meets the Internet of Things. DEBS’15 tutorial Speaker: Opher Etzion

2

The autonomous car

Needs sensors for observing what happens now, needs intelligence to understand what it observes, needs intelligence to drive, needs

actuators to carry out the driving.…

3

Like the human body, we need to sense, to make sense of what we sense, to make constant decisions and to carry them out .

Sensing

Making sense from the sensing

Real-time decision making

Acting

OUTLINE

A quick intro to IoT and its relationship with AI

Some applications of Intelligent IoT

The AI perspective

The future perspective

4

Topic I

TOPIC II

Topic III

Topic IV

OUTLINE



The AI perspective


5

Topic I

TOPIC II

Topic III

Topic IV

None of the authorized drivers location is near the car’s location

theft is concluded

Use a built-in car stopper to slow the intruder and dispatch the security company

A person enters a car and the car starts moving;

the person does not look like one of the authorized drivers

Such applicationsbecome possible

since everything isconnected

6

7

The term “Internet of Things” was coined by Kevin Ashton in 1999 .

His observation was that all the data on the Internet has been created by a human .

His vision was: “we need to empower computers with their own means of gathering information, so they can see, hear, and smell the world by

themselves .”

8

The world of sensors

1 Acoustic, sound, vibration2 Automotive, transportation3 Chemical4 Electric current, electric potential, magnetic, radio5 Environment, weather, moisture, humidity6 Flow, fluid velocity7 Ionizing radiation, subatomic particles8 Navigation instruments9 Position, angle, displacement, distance, speed, acceleration10 Optical, light, imaging, photon11 Pressure12 Force, density, level13 Thermal, heat, temperature14 Proximity, presence

9

The value of sensors

Kevin Ashton: “track and count everything, and greatly reduce waste, loss, and cost. We could know when things needs replacing, repairing or recalling, and whether they were fresh or past their best”

The value is in the ability to know and react in a timely manner to situations that are detected by sensors

10

Differences between the traditional Internet to the Internet of Things

Topic Traditional Internet Internet of Things

Who creates content? Human Machine

How is the content consumed?

By request By pushing information and triggering actions

How content is combined?

Using explicitly defined links

Through explicitly defined operators

What is the value? Answer questions Action and timely knowledge

What was done so far? Both content creation (HTML…) and content consumption (search engines)

Mainly content creation

Two separate but connected goals: Awareness and Reaction

Awareness Reaction

Event

Detect Derive Decide Do

Did SomethingHappen?

What should we do about it?

It Happened

Detect

Some Noun of

Importance but different

The act of bringing into a system’s sphere of understanding knowledge about an event.

The detection is done by sensors, instrumentation and human reports.

Swim

Lane

Trigger Event

Activity

StateChange

13

Intelligent Detection

Determine what is actually been sensed: vision understanding, voice understanding, text understanding.

DeriveThe act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event.

Raw eventsRaw

eventsRaw events

A Person or a computer recognizes the pattern and enters the derived event or just reacts to it directly.

15

Event processing: making sense from what we sense…

Combining data from multi-sensors to get observations, alerts, and actions in real-time gets us to the issue of detecting patterns in event streams

16

Intelligent derivation

Find the causality between events and situations. We discuss the notion of causality later.

Decide

Automated decision by decision management system

The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision.

No Decision

Pass through: Sometimes there is no decision. There is only one course of action.

Automated Goal Oriented: Algorithmic decision via a decision management system that seeks a optimizing quantitative goals.

18

Intelligent Decision

Finding the best decision some times under real-time constraints may require an intelligent process.

DoThe act of performing the course of action that was decided upon.

Notification: Sending a signal of sort to either a person or system. This would include calling a web-service or subscription to alerts.

20

Intelligent Actions

Intelligent actuators

21

Knowledge acquisition for IoT based systems

How do we know how to make sense of all these data?

OUTLINE



The AI perspective


22

Topic I

TOPIC II

Topic III

Topic IV

23

IoT and robotics

Robots serve as intelligent actuators

24

Healthcare robotics

Rehabilitation robots: enhancing patients with motoric and cognitive skills

Assistive robots: Robots for independent living of disabled persons

25

Some future healthcare robotics applications

Automated assistance of monitored patients

26


Help in sit-to-stand and sit-down actions for people with motor disabilities

27


Autonomous moving of drugs and medical equipment within the hospital

28


Support of medical staff in various activities

29


People movement and movement monitoring

30


People assistance in panic and danger situations

31

I(

The classical use of robots are for industrial purposes: production, machinery control, product design…

Industrial Robots

32

I(

Autonomic management and coordination of production activities among multiple robots

Industrial Robots and IoT

33

I(

Autonomous management of equipment and instruments


34

I(

Immediate reaction to critical situations such as: high temperature, harmful chemicals in the air


35

I(

Autonomic control of electrical and energy plants


36

Robotics for defense

Robots are used for unmanned tools (ground and air) for transport and intelligence , threat detection and combat

37

Robotics for defense and IoT

Autonomous and smart detection of harmful chemicals and biological weapons

38


Autonomic control of land vehicles and aircrafts

39


Identification and access prevention of suspicious people intruding to sensitive places

40


Rescue trapped people

41

The Internet of things for the elderly

and healthcare in general

42

Safety sensors

Motion sensor

Door sensor

ChairSensor

Voice Sensor

Alert family member

Alerts example:Door was not locked within 2 minutes after entranceFalling event detectedVocal distress detectedNo motion for certain time period detected

43

Medical sensors for the elderly

44

E-Health sensors

Personalized alerts based on collection of monitors

45

Pre-mature babies monitoring

Personalized alerts based on collection of monitors: when nurse should be alerted, when physician should be alerted.

There are many false alerts that are ignored, Missing or ignored alert is sometimes fatal

46

Track everything in a hospital

47

Track the progress of a surgery relative to the plan

Detect significant deviation from plan that requires rescheduling and trigger real-time rescheduling of surgeries, assignments, and equipments.

48

Dynamic planning

Example: traffic control; patient treatment; serviceman scheduling

49

AI meets IoT – Apple’s Perspective

Siri was released as Apple’s “intelligent personal assistant”.

A sensor enabled Siri is targeted as a “smart home solution”

50

AI meets IoT – Google’s Perspective

Google acquired a collection of IoT related companies and then acquired AI company DEEPMIND that uses Neural Nets and Reinforcement learning. The aim is to develop a machine with intelligence of a toddler with IoT providing sensing capabilities

51

AI meets IoT – Facebook’s Perspective

Facebook acquires wit.ai – a speech recognition company. Making the Internet of Things voice controlled.

OUTLINE



The AI perspective


52

Topic I

TOPIC II

Topic III

Topic IV

53

Vision understanding

Robocup tournament: Robots playing football. Strong vision capabilities are required.

54


Tracking objects over time from a collection of cameras

55


Grace, the robot, can communicate with her surrounding, understand gestures, attended conferences, understands that she had to stand in a line, go in an elevator and ask people to press the floor number…

56

Speech recognition

Acoustic analysis, linguistic Interpretation

57

Causality

In order to derive situations from events there is a need to identify causalities.

Statistical methods can infer correlations.

Causality inference is more tricky….

58

Causalities in events

Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to detect event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events.

Necessity and relevance

59


Type II: The event E1 is an input to a processing element PE and event E2 is an output of PE. In this case E2 is a derived (virtual) event. The specification of PE is part of the system, thus the context and conditions are known.

Necessity and relevance

60


Type III: The event E1 is an event that is sent from a computerized system to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1

Necessity? and relevance?

61

Causal inference

How the knowledge about causality is being acquired?

Expert knowledge

Statistical inference

Inference using semantic or association net

Necessity? and relevance?

62

Dangers of using correlation as causality indicator

Correlation between A and B:

1. A causes B

2. B causes A

3. There is C which causes both A and B

4. A combination of all three interpretations

The faster windmills are observed to rotate, the more wind is observed to be.

Therefore wind is caused by the rotation of windmills.

63



1. A causes B

2. B causes A



Sleeping with one's shoes on is strongly correlated with waking up with a headache.

Therefore, sleeping with one's shoes on causes headache.

(correct answer: going to bad drunk causes both)

64



1. A causes B

2. B causes A



As ice cream sales increase, the rate of drowning deaths increases sharply.

Therefore, ice cream consumption causes drowning. (real answer: they are both in the same context – summer).

Temporal indeterminacy

Inexact indicator Probability

Event did not occur 0.4

Event occurred before T1 0.1

Event occurred in [T1, T2] 0.45

Event occurred after T2 0.05

T1 T2

False positives and negatives

False positive:The pattern is matched;The real-world situation does not occur

False negative:The pattern is not matched;The real-world situation occurs

Learning from experience

67

Data is not good enough…

68

Real-time decision under uncertainty

Robust RTOptimization

Stochastic RTOptimization

Simulation-based RT optimization

Handling event uncertainties

Uncertain whether an reported event has occurred (e.g. accident)

Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties)

Uncertain when an event occurred (will occur): timing of forecasted congestion

Uncertain where an event occurred (will occur): location of forecasted congestion

Uncertain about the level of causality between a car heading towards highway and a car getting into the highway

Uncertain about the accuracy of a sensor input: count of cars, velocity of cars…

The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting

Uncertain about the validity of a forecasting pattern

Uncertain about the quality of the decision about traffic lights setting

Predictive Event Processing (1)

VS.

Photo by Michael Gray, Flickr

Predictive Event Processing (2)

VS.

+

Predictive Event Patterns

Pattern Future event, probability, time interval “4 high value deposits from different geographic locations within 3 days”

“0.6 chance for a large transfer abroad, in 1 day”

“Output event will occur with distribution D over interval (t1,t2)”

Stock decrease of > 5% in 3 hours Good chance for 2% increase within 2 hours

Limitations of the use of rules in specifying predictive event patterns

Limitations:1. Partial patterns

2. Uncertain input events

3. Complex relationship between random variables

Rule = hard-coded probabilistic Relationship

Dynamic event predictionTime Series Prediction

Graphical models

Temporal Graphical models

Learning patterns and causalities

EventPatterns

Pattern and causality acquisition

This is a direction to reduce the complexity of application development

There are challenges in doing it – since “detected situations” are “inferred events” and may not be reflected in past events

76

Security challenges of IoT

Getting security feeling is a necessary condition for the success of IoT to become pervasive.

77

Dangers and challenges

Confusing a sensor

Changing the rules of the game

Abusing an actuator

78

Security considerations of IoT

Murder by the Internet

“With so many devices being Internet connected, it makes murdering people remotely relatively simple, at least from a technical perspective. That’s horrifying,” said IID president and CTO Rod Rasmussen. “Killings can be carried out with a significantly lower chance of getting caught, much less convicted, and if human history shows us anything, if you can find a new way to kill, it will be eventually be used.”

EXAMPLES: Turn off pacemakers, Shutdown car systems while driving, stop IV drip from functioning

79

Confusing a sensor

The same as confusing the human eyes. See things that don’t exist, don’t see things that exist, distort picture…

80

Confusing a sensor

Can be used to sabotage anti-crime systems, to commit fraud, or just damage something or someone…

81

Confusing a sensor

Example from another domain: the Twitter hoax

82


The logic is rule-based. The ease of modification can be abused to add/delete/modify rules, change thresholds…

83


Changing data relevant for the system: maps, pictures, person’s data…

84

Abusing actuators

Deviating from course, shutting down, activating in wrong mode…

OUTLINE



The AI perspective


85

Topic I

TOPIC II

Topic III

Topic IV

TOPIC 4

A futuristic view of the Internet of Things following Ray Kurzweil’s predictions:

86

87Driverless car

Sensors that replace the human driver’s sensing, and actuators that drive the car.

2017

88

Automated personal assistant

Sensors that determine the context serves as active advisors. They understand your context and even listen to your conversations and give you suggestions of what to say (e.g. through google glass).

2018

89

Computing implants inside the human body

Sensors and actuators that fight any disease, operate in the level of cell, and reprogram the body to stop the aging process.

2020

2040

Short term: switch off our fat cells

Longer term: stay young forever

90

May 14, 2014

91

Summary: The Internet of Everything participates in many of the predictions about the future, including Kurzweil’s singularity.

The responsibility is upon us to create this future…

92

My main motivation is to use the experience and knowledge I have accumulated over the years to make a better world

DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things

Technology

Transcript of DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things