Unconstrained Minimization (I)
Transcript of Unconstrained Minimization (I)
Unconstrained Minimization (I)
Lijun [email protected]://cs.nju.edu.cn/zlj
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Basic Terminology
Unconstrained Optimization Problem
is convex always have a domain dom π π , dom π β π
is twice continuously differentiable dom π is open, such as 0, β
The problem is solvable There exists an optimal point π₯β
inf π π₯ π π₯β πβ
Basic Terminology
Unconstrained Optimization Problem
β is optimal if and only if
Special cases: a closed-form solution General cases: an iterative algorithm A sequence of points π₯ , π₯ , β¦ β ππ¨π¦ π with
A minimizing sequence for the problem The algorithm is terminated when
β
Equivalent
π π₯ β πβ as π β β
π π₯ πβ π
Requirements of Iterative Algorithm
Initial Point A suitable starting point
Sublevel Set is Closed
Satisfied for all if the function is closed
Continuous functions with dom π π Continuous functions with open domains
and π π₯ β β as π₯ β bd dom π
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Examples
Convex Quadratic Minimization Problem
Optimality Condition
β (unique solution)2. If is singular and , any solution
of β is optimal3. If , no solution, unbound below
β
Examples
Convex Quadratic Minimization Problem
3. If , no solution, unbound below π π π, π β β π , π β₯ β π Let π₯ π‘π
Examples
Least-Squares Problem
are problem data
Optimality Condition
Normal Equations
β β
β
Examples
Unconstrained Geometric Programming
Optimality Condition
No analytical solution An Iterative Algorithm dom π π , any point can be chosen as π₯
ββ
β
Examples
Analytic Center of Linear Inequalities
is called as the logarithmic barrier for the inequalities
The solution of this problem is called the analytic center of the inequalities
An Iterative Algorithm π₯ must satisfy π π₯ π
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Strong Convexity
is strongly convex on , if
1. A Quadratic Lower Bound
π π¦ π π₯ π»π π₯ π¦ π₯12 π¦ π₯ π» π π§ π¦ π₯
π π₯ π»π π₯ π¦ π₯π2 π¦ π₯
Strong Convexity
is strongly convex on , if
1. A Quadratic Lower Bound
When , reduce to the first-order condition of convex functions
π π¦ π π₯ π»π π₯ π¦ π₯π2 π¦ π₯ , βπ₯, π¦ β π
Strong Convexity
is strongly convex on , if
1. A Quadratic Lower Bound
2. A Condition for Suboptimality
π π¦ π π₯ π»π π₯ π¦ π₯π2 π¦ π₯ , βπ₯, π¦ β π
π π¦ min π π₯ π»π π₯ π¦ π₯π2 π¦ π₯
π π₯ π»π π₯ π¦ π₯π2 π¦ π₯ , π¦ π₯
1π π»π π₯
π π₯1
2π π»π π₯
Strong Convexity
is strongly convex on , if
1. A Quadratic Lower Bound
2. A Condition for Suboptimality
If the gradient is small at π₯, then it is nearly optimal
π π¦ π π₯ π»π π₯ π¦ π₯π2 π¦ π₯ , βπ₯, π¦ β π
πβ π π₯1
2π π»π π₯ π π₯ πβ1
2π π»π π₯
π»π π₯ 2ππ β π π₯ πβ π
Strong Convexity
is strongly convex on , if
3. An Upper Bound of β
πβ π π₯β
π π₯ π»π π₯ π₯β π₯π2 π₯β π₯
πβ π»π π₯ π₯β π₯π2 π₯β π₯
π π₯ π»π π₯ π₯β π₯π2 π₯β π₯
Strong Convexity
is strongly convex on , if
3. An Upper Bound of β
β, as 0 The optimal point β is unique
π2 π₯β π₯ π»π π₯ π₯β π₯
π₯β π₯2π π»π π₯
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Smoothness
is smooth on , if
1. A Quadratic Upper Bound
π π¦ π π₯ π»π π₯ π¦ π₯12 π¦ π₯ π» π π§ π¦ π₯
π π₯ π»π π₯ π¦ π₯π2 π¦ π₯
Smoothness
is smooth on , if
1. A Quadratic Upper Bound
2. An Upper Bound of Gradientsπ π¦ π π₯ π»π π₯ π¦ π₯
π2 π¦ π₯ , βπ₯, π¦ β π
π π₯ π»π π₯ π¦ π₯π2 π¦ π₯ , π¦ π₯
1π π»π π₯
π π₯1
2π π»π π₯
min π π¦ min π π₯ π»π π₯ π¦ π₯π2 π¦ π₯
Smoothness
is smooth on , if
1. A Quadratic Upper Bound
2. An Upper Bound of Gradientsπ π¦ π π₯ π»π π₯ π¦ π₯
π2 π¦ π₯ , βπ₯, π¦ β π
πβ π π₯1
2π π»π π₯
12π π»π π₯ π π₯ πβ
Condition Number
Condition Number of a Matrix
is both strongly convex and smooth
Condition number of
Has a strong effect on the efficiency of optimization methods
π ππ
π π» π π₯
Condition Number
Geometric Interpretations Width of a convex set , in the
direction where
Minimum width and maximum width of
Condition number of cond πΆ is small implies
πΆ it is nearly spherical
π πΆ, π supβ
π π§ infβ
π π§
π inf π πΆ, π , π sup
π πΆ, π
cond πΆππ
Condition Number
Geometric Interpretations -sublevel set of
is both strongly convex and smooth
Condition number of
πΆ π₯ π π₯ πΌ , πβ πΌ π π₯
cond πΆ π ππ
π΅ β πΆ β π΅
π΅ π¦ π¦ π₯β 2 πΌ πβ
π
/
π΅ π¦ π¦ π₯β 2 πΌ πβ
π
/
Discussions
Parameters and Known only in rare cases Unknown in general
They are conceptually useful The convergence behavior of
optimization algorithms depend on them Characterize the convergence rate
In Practice Estimate their values Design parameter-free algorithms
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Iterative Methods
A Minimizing Sequence
is the the iteration number is the output of iterative methods is the step or search direction is the step size or step length
Shorthand
Descent Methods
Descent Methods
Except when is optimal The search direction makes an acute
angle with the negative gradientπ»π π₯ Ξπ₯ 0
π π₯ π π₯ π»π π₯ π₯ π₯
π»π π₯ Ξπ₯ 0 β π»π π₯ π₯ π₯ 0 β π π₯ π π₯
Descent Methods
Descent Methods
Except when is optimal The search direction makes an acute
angle with the negative gradient
is called as descent direction
π»π π₯ Ξπ₯ 0
General Descent Method
The AlgorithmGiven a starting point π₯ β dom πRepeat
1. Determine a descent direction Ξπ₯.
2. Line search: Choose a step size π‘ 0.3. Update: π₯ β π₯ π‘βπ₯.
until stopping criterion is satisfied.
Line Search Determine the next iterate along the line
General Descent Method
The AlgorithmGiven a starting point π₯ β dom πRepeat
1. Determine a descent direction Ξπ₯.
2. Line search: Choose a step size π‘ 0.3. Update: π₯ β π₯ π‘βπ₯.
until stopping criterion is satisfied.
Stopping Criterion
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Exact Line Search
Minimize along the Ray
The cost of the minimization problem with one variable is low
The minimizer along the ray can be found analytically
Outline Unconstrained Minimization Problems Basic Terminology Examples Strong Convexity Smoothness
Descent Methods General Descent Method Exact Line Search Backtracking Line Search
Backtracking Line Search
Most line searches used in practice are inexact Approximately minimize along the ray Just reduce βenoughβ
Backtracking Line Searchgiven a descent direction βπ₯ for π at π₯ βππ¨π¦ π, πΌ β 0, 0.5 , π½ β 0, 1π‘ β 1
while π π₯ π‘Ξπ₯ π π₯ πΌπ‘π»π π₯ βπ₯, π‘ β π½π‘
Backtracking Line Search
The line search is called backtracking It starts with unit step size and then
reduces it by the factor
It eventually terminates is a descent direction, i.e., For small enough
πΌ is the fraction of the decrease in π predicted by linear extrapolation that we will accept
π‘ β 1 , π‘ β π½π‘
π π₯ π‘Ξπ₯ π π₯ π‘π»π π₯ βπ₯ π π₯ πΌπ‘π»π π₯ βπ₯
Backtracking Line Search
Graph InterpretationThe backtracking exit inequalityπ π₯ π‘Ξπ₯ π π₯ πΌπ‘π»π π₯ Ξπ₯holds for π‘ β 0, π‘
Backtracking Line Search
Graph InterpretationThe backtracking line search stopswith a step length π‘ that satisfies
π‘ 1, or π‘ β π½π‘ , π‘
Backtracking Line Search
Graph InterpretationThe backtracking line search stopswith a step length π‘ that satisfies
π‘ min 1, π½π‘
Backtracking Line Search
Require
A Practical Implementation1. Multiply by until 2. Check whether the above inequality
holds is typically chosen between and is often chosen between and
Summary
Unconstrained Minimization Problems First-order Optimality Condition Strong Convexity and Implications Smoothness and Implications
Descent Methods General Descent Method Exact Line Search Backtracking Line Search