Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

8
Question about Gradient descent Hung-yi Lee

Transcript of Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Page 1: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Question aboutGradient descent

Hung-yi Lee

Page 2: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Larger gradient, larger steps?

𝑦=𝑎𝑥2+𝑏𝑥+𝑐

|𝜕 𝑦𝜕 𝑥 |=¿2𝑎𝑥+𝑏∨¿

𝑥0

¿ 𝑥0−𝑏2𝑎

∨¿

𝑥0

¿2𝑎𝑥0+𝑏∨¿

Best step:

−𝑏2𝑎

¿2𝑎𝑥0+𝑏∨ 2𝑎

Page 3: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Contradiction

𝑤𝑡+1←𝑤 𝑡−𝜂𝜎𝑡 𝑔

𝑡

𝜎 𝑡=√𝛼 (𝜎𝑡− 1 )2+(1−𝛼 ) (𝑔𝑡 )2

𝑤𝑡+1←𝑤 𝑡−𝜂

√∑𝑖=0𝑡

(𝑔𝑖 )2𝑔𝑡

Original Gradient descent

Adagrad

RMSprop

𝑤𝑡+1←𝑤 𝑡−𝜂𝑔𝑡

𝑔𝑡=𝜕𝐶 (𝑤 𝑡 )𝜕𝑤

Larger gradient, larger step

Divided by first derivative

Divided by first derivative

Page 4: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Second Derivative

𝑦=𝑎𝑥2+𝑏𝑥+𝑐

|𝜕 𝑦𝜕 𝑥 |=¿2𝑎𝑥+𝑏∨¿

−𝑏2𝑎

𝑥0

¿ 𝑥0−𝑏2𝑎

∨¿

𝑥0¿2𝑎𝑥0+𝑏∨¿

Best step:

𝜕2 𝑦𝜕 𝑥2

=2𝑎 The best step is|First derivative|

Second derivative

¿2𝑎𝑥0+𝑏∨ 2𝑎

Page 5: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

More than one parameters

𝑤1

𝑤2

𝑤1

𝑤2

|First derivative|

Second derivativeThe best step is

a

b

c

d

c < a

c > d

Larger second derivative

smaller second derivative

a > b

Page 6: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

What to do with Adagrad and RMSprop?

|First derivative|

Second derivative

The best step is

Use first derivative to estimate second derivative

√ ( first derivative )2

𝑤1 𝑤2

larger second derivative

smaller second derivative

Page 7: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Acknowledgement

• This question is raised by 李廣和

Page 8: Question about Gradient descent Hung-yi Lee. Larger gradient, larger steps? Best step:

Thanks for your attention!


Proximal gradient methods - Princeton UniversityA proximal view of gradient descent To motivate proximal gradient methods, we first revisit gradient descent xt+1 = xt−η t∇f(xt)

Proximal gradient methods - Princeton UniversityA proximal view of gradient descent To motivate proximal gradient methods, we first revisit gradient descent xt+1 = xt−η t∇f(xt)

Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Mini-batch deeplearning.ai gradient descent · Batch vs. mini-batch gradient descent Vectorization allows you to efficiently compute on mexamples. Andrew Ng Mini-batch gradient descent.

Gradient Descent - University of Washington

Gradient Descent - University of Washington

Exponentiated Gradient versus Gradient Descent for Linear ...manfred/pubs/J36.pdf · Exponentiated Gradient versus Gradient Descent for Linear Predictors* Jyrki Kivinen-Department

Exponentiated Gradient versus Gradient Descent for Linear ...manfred/pubs/J36.pdf · Exponentiated Gradient versus Gradient Descent for Linear Predictors* Jyrki Kivinen-Department

Learning to learn by gradient descent by gradient descent · Learning to learn by gradient descent by gradient descent Marcin Andrychowicz 1, Misha Denil , Sergio Gómez Colmenarejo

Learning to learn by gradient descent by gradient descent · Learning to learn by gradient descent by gradient descent Marcin Andrychowicz 1, Misha Denil , Sergio Gómez Colmenarejo

Gradient descent

Gradient descent

Stochastic Gradient Descent Tricks - bottou.org2.1 Gradient descent It has often been proposed (e.g., [18]) to minimize the empirical risk E n(f w) using gradient descent (GD). Each

Stochastic Gradient Descent Tricks - bottou.org2.1 Gradient descent It has often been proposed (e.g., [18]) to minimize the empirical risk E n(f w) using gradient descent (GD). Each

Gradient Descent - cs.cmu.edu · Gradient Descent • Now that we have seen how horrible gradient descent is, and how there are so many methods with better guarantees, let’s now

Gradient Descent - cs.cmu.edu · Gradient Descent • Now that we have seen how horrible gradient descent is, and how there are so many methods with better guarantees, let’s now

Optimization based on Gradient Descent

Optimization based on Gradient Descent

The Gradient Descent Algorithm

The Gradient Descent Algorithm

Learning to Learn by Gradient Descent with Rebalancing€¦ · that, for instance, are capable of learning to learn without gradient descent by gradient descent. It should be expected

Learning to Learn by Gradient Descent with Rebalancing€¦ · that, for instance, are capable of learning to learn without gradient descent by gradient descent. It should be expected

Exponentiated Gradient versus Gradient Descent for Linear Predictors

Exponentiated Gradient versus Gradient Descent for Linear Predictors

Optimization, Gradient Descent, and Backpropagation

Optimization, Gradient Descent, and Backpropagation

Learning Unitaries with gradient descent optimizationqist2019/slides/3rd/...Gradient Descent Numerics A `transition’ occurs when gradient descent is performed in the overparameterized

Learning Unitaries with gradient descent optimizationqist2019/slides/3rd/...Gradient Descent Numerics A `transition’ occurs when gradient descent is performed in the overparameterized

Boosting Algorithms as Gradient Descent

Boosting Algorithms as Gradient Descent

Lecture 6: stochastic gradient descent Sanjeev Arora Elad …...• Gradient descent algorithm, linear classification • Stochastic gradient descent Title 402-lec6 Created Date 10/6/2016

Lecture 6: stochastic gradient descent Sanjeev Arora Elad …...• Gradient descent algorithm, linear classification • Stochastic gradient descent Title 402-lec6 Created Date 10/6/2016

Gradient Descent: Second Order Momentum and Saturating Error · 1.1 SIMPLE GRADIENT DESCENT First, let us review the bounds on the convergence rate of simple gradient descent without

Gradient Descent: Second Order Momentum and Saturating Error · 1.1 SIMPLE GRADIENT DESCENT First, let us review the bounds on the convergence rate of simple gradient descent without

10-315 Recitation Review of Gradient Descent & Kernelsninamf/courses/315sp19/recitations/2_21-… · Review of Gradient Descent & Kernels Misha 21 February 2019. Gradient Descent:

10-315 Recitation Review of Gradient Descent & Kernelsninamf/courses/315sp19/recitations/2_21-… · Review of Gradient Descent & Kernels Misha 21 February 2019. Gradient Descent:

Gradient Descent Optimization

Gradient Descent Optimization

Gradient descent GAN optimization is locally stablepapers.nips.cc/paper/7142-gradient-descent-gan-optimization-is... · Gradient descent GAN optimization is locally stable ... similarities

Gradient descent GAN optimization is locally stablepapers.nips.cc/paper/7142-gradient-descent-gan-optimization-is... · Gradient descent GAN optimization is locally stable ... similarities