LQ Dynamic Optimization and Differential Games - … · LQ Dynamic Optimization and Differential...

LQ Dynamic Optimization and

Differential Games

Jacob Engwerda

Tilburg University, The Netherlands

JOHN WILEY & SONS, LTD

Innodata0470015519.jpg


Differential Games


Differential Games

Jacob Engwerda

Tilburg University, The Netherlands

JOHN WILEY & SONS, LTD

Copyright # 2005 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester,West Sussex PO19 8SQ, England

Telephone (+44) 1243 779777

Email (for orders and customer service enquiries): [email protected] our Home Page on www.wiley.com

All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted inany form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, exceptunder the terms of the Copyright, Designs and Patents Act 1988 or under the terms of a licence issuedby the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP, UK, without thepermission in writing of the Publisher. Requests to the Publisher should be addressed to the PermissionsDepartment, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex PO19 8SQ,England, or emailed to [email protected], or faxed to (+44) 1243 770620.

Designations used by companies to distinguish their products are often claimed as trademarks. All brand namesand product names used in this book are trade names, service marks, trademarks or registered trademarks of theirrespective owners. The Publisher is not associated with any product or vendor mentioned in this book.

This publication is designed to provide accurate and authoritative information in regard to thesubject matter covered. It is sold on the understanding that the Publisher is not engaged in rendering professionalservices. If professional advice or other expert assistance is required, the servicesof a competent professional should be sought.

Other Wiley Editorial Offices

John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA

Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA

WileyVCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany

John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia

John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop # 02-01, Jin Xing Distripark, Singapore 129809

John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not beavailable in electronic books.

Library of Congress Cataloging-in-Publication Data

p. cm.ISBN 0-470-85756-01. CancerRisk factorsMathematical models. I. Edler, Lutz, 1945- II. Kitsos, Christos Par., 1951-RC268.48.Q34 2004616.99040710 015118dc22 2004028290

British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library

ISBN-13 978-0-470-01524-7 (HB)ISBN-10 0-470-01524-1 (HB)

Typeset in 10/12pt Times by Thomson Press (India) Limited, New DelhiPrinted and bound in Great Britain by Antony Rowe Ltd, Chippenham, WiltshireThis book is printed on acid-free paper responsibly manufactured from sustainable forestryin which at least two trees are planted for each one used for paper production.

http://www.wiley.com

Contents

Preface ix

Notation and symbols xi

1 Introduction 11.1 Historical perspective 1

1.2 How to use this book 10

1.3 Outline of this book 10

1.4 Notes and references 14

2 Linear algebra 152.1 Basic concepts in linear algebra 15

2.2 Eigenvalues and eigenvectors 21

2.3 Complex eigenvalues 23

2.4 CayleyHamilton theorem 31

2.5 Invariant subspaces and Jordan canonical form 34

2.6 Semi-definite matrices 42

2.7 Algebraic Riccati equations 43


2.9 Exercises 55

2.10 Appendix 59

3 Dynamical systems 633.1 Description of linear dynamical systems 64

3.2 Existenceuniqueness results for differential equations 70

3.2.1 General case 70

3.2.2 Control theoretic extensions 74

3.3 Stability theory: general case 78

3.4 Stability theory of planar systems 83

3.5 Geometric concepts 91

3.6 Performance specifications 96

3.7 Examples of differential games 105

3.8 Information, commitment and strategies 114


3.10 Exercises 115

3.11 Appendix 118

4 Optimization techniques 1214.1 Optimization of functions 121

4.2 The EulerLagrange equation 125

4.3 Pontryagins maximum principle 133

4.4 Dynamic programming principle 154

4.5 Solving optimal control problems 162


4.7 Exercises 163

4.8 Appendix 170

5 Regular linear quadratic optimal control 1755.1 Problem statement 175

5.2 Finite-planning horizon 177

5.3 Riccati differential equations 192

5.4 Infinite-planning horizon 196

5.5 Convergence results 209


5.7 Exercises 219

5.8 Appendix 224

6 Cooperative games 2296.1 Pareto solutions 230

6.2 Bargaining concepts 240

6.3 Nash bargaining solution 246

6.4 Numerical solution 251


6.6 Exercises 254

6.7 Appendix 259

7 Non-cooperative open-loop information games 2617.1 Introduction 264


7.3 Open-loop Nash algebraic Riccati equations 278


7.5 Computational aspects and illustrative examples 299

7.6 Convergence results 305

7.7 Scalar case 312

7.8 Economics examples 319

7.8.1 A simple government debt stabilization game 320

7.8.2 A game on dynamic duopolistic competition 322


7.10 Exercises 327

7.11 Appendix 340

8 Non-cooperative feedback information games 3598.1 Introduction 359


vi CONTENTS


8.4 Two-player scalar case 383

8.5 Computational aspects 389

8.5.1 Preliminaries 390

8.5.2 A scalar numerical algorithm: the two-player case 393

8.5.3 The N-player scalar case 399

8.6 Convergence results for the two-player scalar case 403


8.8 Exercises 413

8.9 Appendix 421

9 Uncertain non-cooperative feedback information games 4279.1 Stochastic approach 428

9.2 Deterministic approach: introduction 433

9.3 The one-player case 435

9.4 The one-player scalar case 444

9.5 The two-player case 450

9.6 A fishery management game 455

9.7 A scalar numerical algorithm 461

9.8 Stochastic interpretation 472


9.10 Exercises 475

9.11 Appendix 481

References 485

Index 495

CONTENTS vii

Preface

Interaction between processes at various levels takes place everywhere around us. Either

in industry, economics, ecology or on a social level, in many places processes influence

each other. Particularly in those cases where people can affect the outcome of a process,

the question arises how do people come to take a particular action.

To obtain a clearer view of this question within mathematics the paradigms of optimal

control theory and game theory evolved, and dynamic game theory resulted as a merging

of both these topics. This theory brings together the issues of optimizing behavior, the

presence of multiple agents, enduring consequences of decisions and robustness with

respect to variability in the environment. Within this field, linear quadratic differential

games developed and these play an important role for three reasons: first, many

applications of differential game theory fall into this category, secondly, there are

many analytical results available and, thirdly, efficient numerical solution techniques

can be used to solve these games.

Going through the literature, one can find many instances of results dealing with linear

quadratic differential games. Unfortunately, there is no textbook focusing on this subject

and giving a rigorous self-contained treatment of this theory. This textbook intends on the

one hand to fill this gap and on the other hand to show the relevance of this theory by

illustrating the theoretical concepts and results by means of simple economics examples.

Given this background, the organization of the book was chosen as follows. The last

four chapters of the book deal with differential games. Since the theoretical development

of these chapters sets out some knowledge on the one-player case and dynamic

optimization techniques, these chapters are preceded by a rigorous treatment of this

one-player case, the so-called regular linear quadratic control problem, and a chapter on

dynamic optimization theory. To tackle these issues, however, one needs to be familiar

with some preliminary work on linear algebra and dynamical systems. For that reason

those subjects are dealt with first. The first chapter gives some historical developments

and an outline of the book.

Having worked for more than two years on this book I ask myself whether it was worth

all the trouble. Was it worth spending so much spare time and research time to produce

something from which all one can hope is that it will be digestible and does not contain

too many flaws and, on the other hand, does not contain a number of closely related

interesting issues leaving the reader with information without the theory to back it up?

The main motivation for writing this book was in the hope that it somehow might

contribute a small amount to peoples understanding and that some people will appreciate

this book and use it for this purpose. . .

Last, but not least, I would like to thank the people who have contributed, most of them

unknowingly, to this book. First of all my wife, Carine, and children, Elsemiek, Ton and

Heiko, for offering me the chance to use my spare time for doing this. Next, I am

indebted to Hans Schumacher for many discussions over the past decade concerning

various issues on dynamic games, and to both him and Malo Hautus for the use of their

excellent course notes on dynamic optimization. Furthermore Arie Weeren, Rudy Douven

and Bram van den Broek contributed indirectly to the sections on differential games

through their Ph.D. theses. Finally, Hendri Adriaens is acknowledged for his technical

support on LaTEX and Tomasz Michalak for his proof-reading of an early version of this

book.

Answers to the exercises included in this book can be found at http://www.wiley.

com/go/engwerda

x PREFACE

Notation and symbols

Matrix (Operations)

xT transpose of the vector x

k x k2 length xTx of vector x0 zero; zero vector; zero matrix

I identity matrix

A1 inverse of matrix AAT transpose of matrix A

adj(A) adjoint of matrix A

det(A) determinant of matrix A

rank(A) rank of matrix A

trace(A) trace of matrix A

A spectrum of matrix AAjS spectrum of matrix A restricted to the invariant subspace SAib matrix A with ith column replaced by bAji matrix A with row j and column i deleted

Ker A null space of A

NA null space of AIm A column space of A

RA column space of AdiagfA,Bg diagonal matrix with diagonal entries A and Bdim(S) dimension of subspace S

E eigenspace corresponding with eigenvalue Eg generalized eigenspace corresponding with eigenvalue

Ec center subspace

Es stable subspace

Eu unstable subspace

vec(A) vector of stacked columns of matrix A

A B Kronecker product of matrices A and BS V direct sum subspaces S and VS? orthogonal complement of subspace SC0 set of complex numbers with non-positive real partC0 set of complex numbers with non-negative real partCn set of complex vectors with n entries

Cnm set of n m matrices with complex entries

C1 set of continuous differentiable functions

CD set of continuous functions satisfying some differentiability properties

(definition 3.4)

i set of control functions of player iaff set of affine functions of the state variableF set of all stabilizing feedback matricesL2 set of all measurable Lebesgue square integrable functions on 0;1L2;loc set of all measurable functions that are square integrable over all

finite intervals 0; T Minv set of M-invariant subspacesN set of natural numbers

Ppos set of n-dimensional M-invariant graph subspacesQ set of rational numbers

R set of real numbers

Rn set of vectors with n real entries

Rnm set of n m matrices with real entriesNd set of bargaining problems (see Chapter 6)U set of control functions for which the differential equation has a solution

in the extended sense

Ure fu 2 UjJu exists as a finite number and limt!1 xt exists gUs set of locally square integrable control functions yielding a stable

closed-loop system

Miscellaneous

ARE algebraic Riccati equation

ij Kronecker , ij 1 if i j, ij 0 otherwisef x0; h Frechet/Gateaux differential of f at x0 with increment h@f Frechet derivative of f@if ith Frechet partial derivative of fES; d egalitarian bargaining solutioninf infimum

IS; d ideal pointKS; d KalaiSmorodinsky bargaining solutionlim limit

NS; d Nash bargaining solution0h higher-order terms in hni1pi product of the n pi variablesRDE Riccati differential equation

sup supremum

jzj modulus,ffiffiffiffiffiffiffiffiffiffiffiffiffiffix2 y2

p, of complex number z x iy

Re(z) real part, x, of complex number z x iyx y vector inequality xi > yi; i 1; . . . ;N

xii NOTATION AND SYMBOLS

1

Introduction

1.1 Historical perspective

Dynamic game theory brings together four features that are key to many situations in

economics, ecology and elsewhere: optimizing behavior, the presence of multiple agents/

players, enduring consequences of decisions and robustness with respect to variability in

the environment.

To deal with problems which have these four features the dynamic game theory

methodology splits the modeling of the problem into three parts. One part is the modeling

of the environment in which the agents act. To obtain a mathematical model of the agents

environment a set of differential or difference equations is usually specified. These

equations are assumed to capture the main dynamical features of the environment. A

characteristic property of this specification is that these dynamic equations mostly contain

a set of so-called input functions. These input functions model the effect of the actions

taken by the agents on the environment during the course of the game. In particular, by

viewing nature as a separate player in the game who can choose an input function that

works against the other player(s), one can model worst-case scenarios and, consequently,

analyze the robustness of the undisturbed game solution.

A second part is the modeling of the agents objectives. Usually the agents objectives

are formalized as cost/utility functions which have to be minimized. Since this

minimization has to be performed subject to the specified dynamic model of the

environment, techniques developed in optimal control theory play an important role in

solving dynamic games. In fact, from a historical perspective, the theory of dynamic

games arose from the merging of static game theory and optimal control theory. However,

this merging cannot be done without further reflection. This is exactly what the third

modeling part is about. To understand this point it is good to summarize the rudiments of

static games.

Most research in the field of static game theory has been and is being concentrated

on the normal form of a game. In this form all possible sequences of decisions of each

player are set out against each other. So, for example, for a two-player game this results in

a matrix structure. Characteristic for such a game is that it takes place in one moment of

LQ Dynamic Optimization and Differential Games J. Engwerda# 2005 John Wiley & Sons, Ltd

LQ Dynamic Optimization and Differential Games - … · LQ Dynamic Optimization and Differential...

Documents

Transcript of LQ Dynamic Optimization and Differential Games - … · LQ Dynamic Optimization and Differential...