Minima, Maxima, Saddle points

c©Levent Kandiller, Principles of Mathematics in Operations Research, The International Series in

Operations Research and Management Science, Vol. 97, Springer, 2007. ISBN: 0-387-37734-4

Minima, Maxima, Saddle pointsLevent Kandiller

Industrial Engineering Department

Cankaya University, Turkey

Minima, Maxima, Saddle points – p.1/9



Scalar Functions

Let us remember the properties for maxima, minima andsaddle points when we have scalar functions with twovariables with the help of the following examples.




Scalar Functions

Example . Let f(x, y) = x2 + y2. Find the extreme points:




Scalar Functions


−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

10

0.5

1

1.5

2




Scalar Functions


−1

−0.5

0

0.5

1

−1

−0.5

0

0.5

10

0.5

1

1.5

2

∂f(x,y)∂x

= 2x.= 0 ⇒ x = 0,

∂f(x,y)∂y

= 2y.= 0 ⇒ y = 0.

Since we have only one critical point, it is either the maximum or the

minimum. We observe that f(x, y) takes only nonnegative values.

Thus, we see that the origin is the minimum point.




Scalar Functions

Example . Find the extreme points of

f(x, y) = xy − x2 − y2 − 2x − 2y + 4.




Scalar Functions


f(x, y) = xy − x2 − y2 − 2x − 2y + 4.

The function is differentiable and has no boundary points.




Scalar Functions


f(x, y) = xy − x2 − y2 − 2x − 2y + 4.

fx = ∂f(x,y)∂x

= y − 2x − 2, fy = ∂f(x,y)∂y

= x − 2y − 2.

Thus, x = y = −2 is the critical point.




Scalar Functions


f(x, y) = xy − x2 − y2 − 2x − 2y + 4.

fx = ∂f(x,y)∂x

= y − 2x − 2, fy = ∂f(x,y)∂y

= x − 2y − 2.


fxx = ∂2f(x,y)∂x2 = −2 = ∂2f(x,y)

∂y2 = fyy, fxy = ∂2f(x,y)∂x∂y

= 1.




Scalar Functions


f(x, y) = xy − x2 − y2 − 2x − 2y + 4.

fx = ∂f(x,y)∂x

= y − 2x − 2, fy = ∂f(x,y)∂y

= x − 2y − 2.


fxx = ∂2f(x,y)∂x2 = −2 = ∂2f(x,y)

∂y2 = fyy, fxy = ∂2f(x,y)∂x∂y

= 1.

The discriminant (Jacobian) of f at (a, b) = (−2,−2) is

fxx fxy

fxy fyy

= fxxfyy − f2xy = 4 − 1 = 3.

Since fxx < 0, fxxfyy − f2xy > 0 ⇒ f has a local maximum at (−2,−2).




Scalar Functions

Theorem . The extreme values for f(x, y) can occur only at

i. Boundary points of the domain of f .

ii. Critical points (interior points where fx = fy = 0, or points where

fx or fy fails to exist).




Scalar Functions

Theorem . If the first and second order partial derivatives of f are

continuous throughout an open region containing a point (a, b) and

fx(a, b) = fy(a, b) = 0, you may be able to classify (a, b) with the

second derivative test:

i. fxx < 0, fxxfyy − f 2xy > 0 at (a, b) ⇒ local maximum;

ii. fxx > 0, fxxfyy − f 2xy > 0 at (a, b) ⇒ local minimum;

iii. fxxfyy − f 2xy < 0 at (a, b) ⇒ saddle point;

iv. fxxfyy − f 2xy = 0 at (a, b) ⇒ test is inconclusive (f is singular).




Quadratic forms

Definition . The quadratic termf(x, y) = ax2 + 2bxy + cy2

is positive definite (negative definite) if and only if a > 0(a < 0) and ac − b2 > 0.




Quadratic forms


is positive definite (negative definite) if and only if a > 0(a < 0) and ac − b2 > 0.f has a minimum (maximum) atx = y = 0 if and only if fxx(0, 0) > 0 (fxx(0, 0) < 0)

and fxx(0, 0)fyy(0, 0) > f 2xy(0, 0).




Quadratic forms


is positive definite (negative definite) if and only if a > 0(a < 0) and ac − b2 > 0.f has a minimum (maximum) atx = y = 0 if and only if fxx(0, 0) > 0 (fxx(0, 0) < 0)

and fxx(0, 0)fyy(0, 0) > f 2xy(0, 0).

If f(0, 0) = 0, we term f as positive (negative)semi-definite provided the above conditions hold.




Quadratic forms

Now, we are able to introduce matrices to the quadraticforms:

ax2 + 2bxy + cy2 = [x, y]

a b

b c

x

y

.




Quadratic forms

Thus, for any symmetric A, the product f = xT Ax is a purequadratic form: it has a stationary point at the origin and nohigher terms.




Quadratic forms


xAT x = [x1, x2, · · · , xn]

a11 a12 · · · a1n

a21 a22 · · · a2n

......

. . ....

an1 an2 · · · ann

x1

x2

...

xn




Quadratic forms


xAT x = [x1, x2, · · · , xn]

a11 a12 · · · a1n

a21 a22 · · · a2n

......

. . ....

an1 an2 · · · ann

x1

x2

...

xn

= a11x21 + a12x1x2 + · · · + annx

2n =

∑n

i=1

∑n

j=1 aijxixj.




Quadratic forms

Definition . If A is such that aij = ∂2f∂xi∂xj

(hence

symmetric), it is called the Hessian matrix.




Quadratic forms

Definition . If A is such that aij = ∂2f∂xi∂xj

(hence

symmetric), it is called the Hessian matrix.If A is positive definite (xTAx > 0, ∀x 6= θ) and if f hasa stationary point at the origin (all first derivatives at theorigin are zero), then f has a minimum.




Quadratic forms

Remark . Let f : Rn 7→ R and x∗ ∈ R

n be the local minimum,

∇f(x∗) = θ and ∇2f(x∗) is positive definite.




Quadratic forms



∇f(x∗) = θ and ∇2f(x∗) is positive definite. We are able to explore

the neighborhood of x∗ by means of x∗ + ∆x, where ‖∆x‖ is

sufficiently small (such that the second order Taylor’s approximation is

pretty good) and positive.




Quadratic forms






pretty good) and positive. Then,

f(x∗ + ∆x) ∼= f(x∗) + ∆xT∇f(x∗) + 12∆xT∇2f(x∗)∆x.




Quadratic forms






pretty good) and positive. Then,

f(x∗ + ∆x) ∼= f(x∗) + ∆xT∇f(x∗) + 12∆xT∇2f(x∗)∆x.

The second term is zero since x∗ is a critical point and the third term is

positive since the Hessian evaluated at x∗ is positive definite. Thus, the

left hand side is always strictly greater than the right hand side,

indicating the local minimality of x∗.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let f(x1, x2) = 13x3

1 + 12x2

1 + 2x1x2 + 12x2

2 − x2 + 19. Find the stationaryand boundary points, then find the minimizer and the maximizer over−4 ≤ x2 ≤ 0 ≤ x1 ≤ 3

24 35 24 3524 35 24 35 24 35 24 3524 35 24 35 24 35 24 35




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let f(x1, x2) = 13x3

1 + 12x2

1 + 2x1x2 + 12x2


0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

24 35 24 3524 35 24 35 24 35 24 3524 35 24 35 24 35 24 35




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let f(x1, x2) = 13x3

1 + 12x2

1 + 2x1x2 + 12x2


∇f(x) =

∂f∂x1

∂f∂x2

=

x21 + x1 + 2x2

2x1 + x2 − 1

.=

0

0

⇒

(x1 − 1)(x1 − 2) = 0

x2 = 1 − 2x1

24 35 24 3524 35 24 35 24 35 24 3524 35 24 35 24 35 24 35




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

∇f(x) =

∂f∂x1

∂f∂x2

=

x21 + x1 + 2x2

2x1 + x2 − 1

.=

0

0

⇒

(x1 − 1)(x1 − 2) = 0

x2 = 1 − 2x1

Therefore, xA =

24 1

−1

35 , xB =

24 2

−3

35 are stationary points inside the region

defined by −4 ≤ x2 ≤ 0 ≤ x1 ≤ 3.

24 35 24 35 24 35 24 3524 35 24 35 24 35 24 35




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

∇f(x) =

∂f∂x1

∂f∂x2

=

x21 + x1 + 2x2

2x1 + x2 − 1

.=

0

0

⇒

(x1 − 1)(x1 − 2) = 0

x2 = 1 − 2x1

Therefore, xA =

24 1

−1

35 , xB =

24 2

−3

35 are stationary points inside the region

defined by −4 ≤ x2 ≤ 0 ≤ x1 ≤ 3. Moreover, we have the following boundaries

xI =

24 0

x2

35 , xII =

24 3

x235 and xIII =

24 x1

−4

35 , xIV =

24 x1

0

35

defined by

xC =

24 0

035 , xD =

24 0

−4

35 , xE =

24 3

0

35 , xF =

24 3

−4

35.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let the Hessian matrix be

∇2f(x) =

∂2f∂x1∂x1

∂2f∂x1∂x2

∂2f∂x2∂x1

∂2f∂x2∂x2

=

2x1 + 1 2

2 1

.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E


∇2f(x) =

∂2f∂x1∂x1

∂2f∂x1∂x2

∂2f∂x2∂x1

∂2f∂x2∂x2

=

2x1 + 1 2

2 1

. Then, we

have ∇2f(xA) =

3 2

2 1

and ∇2f(xB) =

5 2

2 1

.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E


∇2f(x) =

∂2f∂x1∂x1

∂2f∂x1∂x2

∂2f∂x2∂x1

∂2f∂x2∂x2

=

2x1 + 1 2

2 1

. Then, we

have ∇2f(xA) =

3 2

2 1

and ∇2f(xB) =

5 2

2 1

.

Let us check the positive definiteness of ∇2f(xA):

vT∇2f(xA)v = [v1, v2]24 3 2

2 1

3524 v1

v2

35 = 3v2

1+ 4v1v2 + v2

2.

If v1 = −0.5 and v2 = 1.0, we will have vT∇2f(xA)v < 0. On the other hand, if

v1 = 1.5 and v2 = 1.0, we will have vT∇2f(xA)v > 0. Thus, ∇2f(xA) is indefinite.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E


∇2f(x) =

∂2f∂x1∂x1

∂2f∂x1∂x2

∂2f∂x2∂x1

∂2f∂x2∂x2

=

2x1 + 1 2

2 1

. Then, we

have ∇2f(xA) =

3 2

2 1

and ∇2f(xB) =

5 2

2 1

.

Let us check ∇2f(xB):

vT∇2f(xB)v = [v1, v2]

24 5 2

2 13524 v1

v2

35 = 5v2

1+ 4v1v2 + v2

2= v2

1+ (2v1 + v2)2 > 0.

Thus, ∇2f(xB) is positive definite and xB =

24 2

−3

35 is a local minimizer with

f(xB) = 19.166667Minima, Maxima, Saddle points – p.9/9



Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let us check the boundary defined by xI :

f(0, x2) =1

2x2

2 − x2 + 19 ⇒df(0, x2)

dx2= x2 − 1

.= 0 ⇒ x2 = 1.

Since d2f(0,x2)dx2

2

= 1 > 0, x2 = 1 > 0 is the local minimizer outside

the feasible region. As the first derivative is negative for

−4 ≤ x2 ≤ 0, we will check x2 = 0 for minimizer and x2 = −4 for

maximizer.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let us check the boundary defined by xII :

f(3, x2) =1

2x2

2 + 5x2 +65

2⇒

df(3, x2)

dx2= x2 + 5

.= 0 ⇒ x2 = −5.

Since d2f(0,x2)dx2

2

= 1 > 0, x2 = −5 < −4 is the local minimizer

outside the feasible region. As the first derivative is positive for

−4 ≤ x2 ≤ 0, we will check x2 = −4 for minimizer and x2 = 0 for

maximizer.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let us check the boundary defined by xIII :

f(x1, 0) =1

3x3

1+1

2x2

1+19 ⇒df(x1, 0)

dx1= x2

1+x1.= 0 ⇒ x1 = 0,−1.

Since d2f(x1,0)dx2

1

= 2x1 + 1, x1 = 0 is the local minimizer

(d2f(0,0)dx2

1

= 1 > 0) on the boundary, and x1 = −1 is the local

maximizer (d2f(−1,0)dx2

1

= −1 < 0) outside the feasible region. As

the first derivative is positive for 0 ≤ x2 ≤ 3, we will check x2 = 3

for maximizer.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

Let us check the boundary defined by xIV :

f(x1,−4) = 13x3

1 + 12x2

1 − 8x1 + 31 ⇒ df(x1,−4)dx1

= x21 + x1 − 8

.= 0

⇒ x1 = −1±√

1+322 . Since d2f(x1,−4)

dx2

1

= 2x1 + 1 again, the positive

root x1 = −1+√

332 = 2.3723 is the local minimizer

(d2f(2.3723,0)dx2

1

> 0), and the negative root is the local maximizer but

it is outside the feasible region. As the first derivative is positive

for 0 ≤ x2 ≤ 3, we will check x2 = 3 for maximizer again.




Collaborative Work:

0

0,2

0,4

0,6

0,8 1

1,2

1,4

1,6

1,8 2

2,2

2,4

2,6

2,8 3

-4

-3,5

-2,9

-2,4

-1,9

-1,4

-0.9

-0.4

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

A

B

C

E

To sum up, we have to consider (2,−3), (0, 0) and (2.3723,−4)

for the minimizer; (3, 0) and (0,−4) for the maximizer:

f(2,−3) = 19.16667, f(0, 0) = 19, f(2.3723,−4) = 19.28529

⇒ (0, 0) is the minimizer!

f(3, 0) = 32.5, f(0,−4) = 31 ⇒ (3, 0) is the maximizer!


Minima, Maxima, Saddle points

Documents

Transcript of Minima, Maxima, Saddle points