Motivated Machine Learning for Water Resource Management

UNESCO Crossing the Chasm

Motivated Machine Learning forMotivated Machine Learning for

Water Resource ManagementWater Resource Management

Janusz StarzykSchool of Electrical Engineering and Computer Science, Ohio University, USA

www.ent.ohiou.edu/~starzyk

UNESCO Workshop on Integrated Modeling Approaches to Support Water Resource Decision Making:

Crossing the Chasm


Challenges in Water Management Embodied Intelligence (EI) Embodiment of Mind EI Interaction with Environment How to Motivate a Machine Goal Creation Hierarchy GCS Experiment Promises of EI

To economy To society

Outline


Water management is challenging for various reasons:

Strategies in water management are developed mostly on national level

There is a growing competition between countries for water

Water policy making effects environment and society, health and development, and economy

Growing demands of countries’ populations for water Leads to hydrological nationalism Creates a need to integrate water sciences and policy

making There is an acute need for legitimate scientific data Decision making in water-related health, food and

energy systems are critical to economical development and security

Challenges in Water Management


Decision makers must answer important questions:

How do we make water use sustainable? Who owns the water? What policies, institutional and legal framework

can promote sustainable use of water? How to protect water resources from overuse

and contamination? Water problems became too complex,

interconnected and large to be handled by any one institution or by one group of professionals

It is a challenge to evolve strategies and institutional frameworks for quick policy changes towards an acceptable water use

It is necessary to create assessment and modeling tools to improve policy making and facilitate interaction.



Why development of integrated modeling to support decision making is important ?

Computerized models were used for many years to support water related decisions.

Models often simplify dynamics of economic, social and environmental interactions and lead to inappropriate policy making and management decisions.

This note proposes models to emerge from interaction with real dynamically changing environments with all of their complexities and societal dependencies.

The main objective is to suggest an integrated modeling framework that may assist with the tasks of water related decision making.



What are deficiencies of machine created models?

Artificial neural networks, fuzzy logic, and genetic algorithms have all been used to model the hydrological cycle

However, it is still difficult to apply these tools in making real-life water decisions as the related parameters are not explicitly known

What may be needed is a motivated machine learning for characterizing the data and making predictions about their dynamic changes It could support intelligent decision making in dynamically

changing environment It could be used to observe impacts of alternative water

management policies It may consider social, cultural, political, economic and

institutional elements that influence decision making This strategic note presents a goal creation approach in

embodied intelligence (EI) that motivates machine to develop into a useful research toll.



Embodied intelligence may support decision making:

EI mimics biological intelligent systems, extracting general principles of intelligent behavior and applying them to design intelligent agents

It uses emerging, self-organizing, goal creation (GC) system that motivates embodied intelligence to learn how to efficiently interact with the environment Knowledge is not entered into such systems, but rather

is a result of their successful use in a given environment. This knowledge is validated through active interaction

with the environment. Motivated intelligent systems adapt to unpredictable

and dynamic situations in the environment by learning, which gives them a high degree of autonomy

Learning in such systems is incremental, with continuous prediction of the input associations based on the emerging models - only new information is registered in the memory



Use the motivated learning scheme to integrate modelling and decision making:

It is suggested to apply ML approach to water management in changing environments where the existing methods fail or work with difficulty. For instance, using classical machine learning to

predict the future for physical processes works only under the assumption that same processes will repeat.

However, if a process changes beyond certain limits, the prediction will not be correct.

GC systems may overcome this difficulty and such systems can be trained to advice humans.

Design concepts, computational mechanisms, and architectural organization of embodied intelligence are presented in this talk

The talk will explain internal motivation mechanism that leads to effective goal oriented learning

In addition, a goal creation mechanism and goal driven learning will be described.



“…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al. E. R. Kandel won Nobel Price in 2000 for his work on physiological

basis of memory storage in neurons.

“…The question of intelligence is the last great terrestrial frontier of science...” from Jeff Hawkins On Intelligence. Jeff Hawkins founded the Redwood Neuroscience Institute devoted

to brain research

Intelligence

AI’s holy grailFrom Pattie Maes MIT Media Lab


Traditional AITraditional AI Embodied Intelligence Embodied Intelligence Abstract intelligence

attempt to simulate “highest” human faculties:

– language, discursive reason, mathematics, abstract problem solving

Environment model Condition for problem

solving in abstract way “brain in a vat”

Embodiment knowledge is implicit in the

fact that we have a body– embodiment supports brain

development

Intelligence develops through interaction with environment Situated in environment Environment is its best model


Design principles of intelligent systemsDesign principles of intelligent systemsfrom Rolf Pfeifer “Understanding of Intelligence”, 1999

Interaction with complex environment

cheap design ecological balance redundancy principle parallel, loosely

coupled processes asynchronous sensory-motor

coordination value principle Agent

Drawing by Ciarán O’Leary- Dublin Institute of Technology


Embodied Intelligence Embodied Intelligence

Definition Embodied Intelligence (EI) is a mechanism that learns

how to survive in a hostile environment

– Mechanism: biological, mechanical or virtual agentwith embodied sensors and actuators

– EI acts on environment and perceives its actions– Environment hostility is persistent and stimulates EI to act– Hostility: direct aggression, pain, scarce resources, etc– EI learns so it must have associative self-organizing memory– Knowledge is acquired by EI


Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment of a MindEmbodiment of a Mind

Embodiment contains intelligence core and sensory motor interfaces under its control to interact with environment

Necessary for development of intelligence

Not necessarily constant or in the form of a physical body

Boundary transforms modifying brain’s self-determination


Brain learns own body’s dynamic Self-awareness is a result of

identification with own embodiment Embodiment can be extended by

using tools and machines Successful operation is a function

of correct perception of environment and own embodiment

Embodiment of a Mind


INPUT OUTPUT

Simulation or

Real-World System

TaskEnvironment

Agent Architecture

Long-term Memory

Short-term Memory

Reason

ActPerceive

RETRIEVAL LEARNING

EI Interaction with Environment

From Randolph M. Jones, P : www.soartech.com


How to Motivate a MachineHow to Motivate a Machine ? ?

The fundamental question is how to motivate a machine to do anything, in particular to increase its “brain” complexity?

How to motivate it to explore the environment and learn how to effectively work in this environment?

Can a machine that only implements externally given goals be intelligent?If not how these goals can be created?


I suggest that hostility of environment motivates us. It is the pain that moves us. Our intelligence that tries to minimize this pain motivates our actions,

learning and development

We need both the environment hostility and the mechanism that learns how to reduce inflicted by the environment pain

How to Motivate a MachineHow to Motivate a Machine ? ?

In this work I propose, based on the pain, mechanism that motivates the machine to act, learn and develop.

So the pain is good.Without the pain there will be no intelligence. Without the pain there will be no motivation to develop.


Pain-center and Goal CreationPain-center and Goal Creation

Simple Mechanism Creates hierarchy of

values Leads to formulation of

complex goals Reinforcement :

• Pain increase• Pain decrease

Forces exploration

+

-

Environment

Sensor

MotorPain level

Dual pain levelPain increase

Pain decrease

(-)

(+)

Excitation

(-)

(-)

(+)

(+)

Wall-E’s goal is to keep his plants from dying


Primitive Goal CreationPrimitive Goal Creation

- +

Pain

Dry soilPrimitive

level

opentank

sit on garbage

refillfaucet

w. can water

Dual pain


Abstract Goal CreationAbstract Goal Creation The goal is to reduce the primitive pain level Abstract goals are created to reduce abstract pains in order to satisfy the primitive goals Abstract pain center

- +

PainDual pain

+

Dry soil

Abstract pain

“water can” – sensory input

to abstract pain center

Sensory pathway(perception, sense)

Motor pathway(action, reaction)

Primitive Level

Level I

Level IIfaucet

-

w. can

open

water

ActivationStimulationInhibitionReinforcementEchoNeedExpectation


Abstract Goal HierarchyAbstract Goal Hierarchy

A hierarchy of abstract goals is created - they satisfy the lower level goals

ActivationStimulationInhibitionReinforcementEchoNeedExpectation

- +

+

Dry soilPrimitive Level

Level I

Level IIfaucet

-

w. can

open

water

+

Sensory pathway(perception, sense)

Motor pathway(action, reaction)

Level IIItank

-

refill


GCS vs. Reinforcement LearningGCS vs. Reinforcement Learning

Environment

CriticStates

Value Function

Policy

reward

action

Environment

CriticStates

Value Function

Policy

reward

action

RL Actor-critic design Goal creation system

Case study: “How can Wall-E water his plants if the water resources are limited and hard to find?”

Sensorypathway

Motorpathway

GCS

Environment

Pain

States

Gate control

Desired action &state

Action decision

Action


Goal Creation Experiment

Sensory-motor pairs and their effect on the environment

-lake waterfallrain29

lake waterreservoir wateropenpipe22

reservoir waterwater in tankrefilltank15

water in tankwater in canopenfaucet8

water in canmoisturewater the plantwater can1

DECREASESINCREASESMOTORSENSORYPAIR #


Results from GCS schemeResults from GCS scheme

0 100 200 300 400 500 6000

2

4pa

in

Dry soil

0 100 200 300 400 500 6000

1

2

pain

No water in can

0 100 200 300 400 500 6000

1

2

pain

No water in tank

0 100 200 300 400 500 6000

0.5

1

pain

No water in reservoir

0 100 200 300 400 500 6000

2

4

pain

No water in lake


Averaged performance over 10 trials:

GCS:

RL:0 100 200 300 400 500 600

0

0 .5

1

pain

P rim itive p a in

0 100 200 300 400 500 6000

0.5

1

pain

L ac k o f fo o d

0 100 200 300 400 500 6000

0.2

0 .4

pain

L ac k o f m o ne y

0 100 200 300 400 500 6000

0.2

0 .4

pain

L ac k o f b ank s aving s

0 100 200 300 400 500 6000

0.2

0 .4

pain

L ac k o f jo b o p p o rtunity

0 100 200 300 400 500 600-1

0

1

pain

L ac k o f s c ho o l o p p o rtun ity

Machine using GCS learns to control all abstract pains and maintains the primitive pain signal on a low level in

demanding environment conditions.

0 100 200 300 400 500 6000

10

20

30

GCS vs. Reinforcement LearningGCS vs. Reinforcement Learning



Action scatters in 5 CGS simulations

0 100 200 300 400 500 6000

5

10

15

20

25

30

35

40Goal Scatter Plot

Go

al ID

Discrete time



The average pain signals in 100 CGS simulations

0 100 200 300 400 500 6000

0.5

Primitive pain – dry soil

Pai

n

0 100 200 300 400 500 6000

0.10.2

Lack of water in can

Pai

n

0 100 200 300 400 500 6000

0.10.2

Lack of water in tank

Pai

n

0 100 200 300 400 500 6000

0.10.2

Lack of water in reservoir

Pai

n

0 100 200 300 400 500 6000

0.050.1

Lack of water in lake

Pai

n

Discrete time


Compare RL (TDF) and GCSCompare RL (TDF) and GCS

Mean primitive pain Pp value as a function of the number of iterations.

Dashed lines indicate moment when Pp is getting stable - green line for TDF - blue line for GCS.


Comparison of execution time on log-log scale TD-Falcon green GCS blue

Combined efficiency of GCS 1000 better than TDF

Compare RL (TDF) and GCSCompare RL (TDF) and GCS

Problem solved

Conclusion: embodied intelligence, with motivated learning based on goal creation system, effectively integrates environment modeling and decision making – thus it is poised to cross the chasm


Promises of embodied intelligencePromises of embodied intelligence To society

Advanced use of technology– Robots– Tutors– Intelligent gadgets

Intelligence age follows– Industrial age– Technological age– Information age

Society of minds– Superhuman intelligence– Progress in science– Solution to societies’ ills

To industry Technological development New markets Economical growth

ISAC, a Two-Armed Humanoid RobotVanderbilt University


2002 2010 2020 2030

Biomimetics and Bio-inspired SystemsImpact on Space Transportation, Space Science and Earth Science

Mis

sio

n C

om

ple

xity

Biological Mimicking

Embryonics

Extremophiles

DNA Computing

Brain-like computing

Self Assembled Array

Artificial nanoporehigh resolution

Mars in situlife detector

Sensor Web

Skin and Bone

Self healing structureand thermal protection

systems

Biologically inspired aero-space systems

Space Transportation

Memristors

Biological nanoporelow resolution


Sounds like science fictionSounds like science fiction

If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.

But if it doesn’t seem like science fiction, it’s definitely wrong.

From presentation by Foresight Institute


Questions?Questions?

Motivated Machine Learning for Water Resource Management

Documents

Transcript of Motivated Machine Learning for Water Resource Management