PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs...
Transcript of PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs...
![Page 1: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/1.jpg)
PYTHON FOROPTIMIZATION
BEN MORAN@BENM
HTTP://BENMORAN.WORDPRESS.COM/2014-02-22
![Page 2: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/2.jpg)
INTRODUCTIONPython for optimization
Not optimizing Python programsNot website optimization/SEOMathematical optimization!scipy.optimize and friends
![Page 3: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/3.jpg)
MATHEMATICAL OPTIMIZATION
![Page 4: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/4.jpg)
OBJECTIVE FUNCTION
![Page 5: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/5.jpg)
WITH CONSTRAINTS
![Page 6: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/6.jpg)
APPLICATIONSEngineeringFinanceOperations ResearchMachine learningStatisticsPhysics
![Page 7: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/7.jpg)
PYTHON FOR OPTIMIZATIONExploration/visualization: IPython notebook/Matplotlib/…As a high level modelling languageBatteries included: scipy.optimize, statsmodels, 3rdparty solver supportCython, etc. for C bindings and high performance code
![Page 8: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/8.jpg)
REST OF THE TALKNumerical optimization vs symbolic manipulationGeneral purpose solvers: scipy.optimizeLinear/convex solvers: Python as a modelling languageSmooth optimization: Sympy for derivatives
![Page 9: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/9.jpg)
OPTIMIZATION TYPES
![Page 10: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/10.jpg)
NO STRUCTUREimport numpy as npxx = np.arange(1000)f = np.random.randn(len(xx))x = np.argmin(f)
![Page 11: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/11.jpg)
LOTS OF STRUCTURExx = np.arange(1000)f = 2*xxx = np.argmin(f)
![Page 12: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/12.jpg)
QUADRATICxx = np.arange(-1000,1000)f = (xx-400)**2x = np.argmin(f)
![Page 13: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/13.jpg)
SCIPY.OPTIMIZEimport numpy as npimport scipy.optimizef = lambda x: np.exp((x-4)**2)return scipy.optimize.minimize(f, 5)
status: 0 success: True njev: 8 nfev: 24 hess_inv: array([[ 0.49999396]]) fun: 1.0 x: array([ 3.99999999]) message: 'Optimization terminated successfully.' jac: array([ 0.])
![Page 14: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/14.jpg)
CHOOSING A SOLVER METHODMethods in scipy.optimize.minimize
BFGS (default) 1st Nelder-Mead Powell CG 1st Newton-CG 2nd Anneal Global dogleg 2nd L-BFGS-B 1st boundsTNC 1st boundsCobyla inequalitySLSQP equality/inequality
![Page 15: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/15.jpg)
SIZE: FROM LOW TO HIGH DIMENSIONAL
![Page 16: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/16.jpg)
SMOOTHNESS: FROM SMOOTH TO DISCONTINUOUS
![Page 17: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/17.jpg)
GLOBAL SHAPE: FROM LINEAR TO STRICTLY CONVEX TO MANYLOCAL MINIMA
![Page 18: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/18.jpg)
NELDER-MEADJust uses continuity, expects few minima
import numpy as npimport scipy.optimizef = lambda x: np.exp((x-4)**2)return scipy.optimize.minimize(f, 5, method='Nelder-Mead')
status: 0 nfev: 32 success: True fun: 1.0 x: array([ 4.]) message: 'Optimization terminated successfully.' nit: 16
![Page 19: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/19.jpg)
![Page 20: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/20.jpg)
SIMULATED ANNEALINGExpects multiple minima
import numpy as npimport scipy.optimizef = lambda x: np.exp((x-4)**2)return scipy.optimize.minimize(f, 5, method='anneal')
status: 0 success: True accept: 0 nfev: 251 T: inf fun: 6.760670043672425 x: array(2.617566636676699) message: 'Points no longer changing' nit: 4
![Page 21: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/21.jpg)
NEWTON'S METHODimport numpy as npimport scipy.optimizef = lambda x: np.exp((x-4)**2)fprime = lambda x: 2*(x-4)*np.exp((x-4)**2)return scipy.optimize.minimize(f, 5, jac=fprime, method='Newton-CG')
status: 0 success: True njev: 24 nfev: 7 fun: array([ 1.]) x: array([ 4.00000001]) message: 'Optimization terminated successfully.' nhev: 0 jac: array([ 2.46202099e-08])
![Page 22: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/22.jpg)
![Page 23: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/23.jpg)
USAGE TIPScheck warningsconsider tolerancescheck_grad if supplying derivatives
![Page 24: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/24.jpg)
PYTHON FOR MODELLINGFor large scale problemsWhen we need to go fastSpecialized solvers
![Page 25: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/25.jpg)
LINEAR PROGRAMS AND PULPminimize f(x) = Ax + b
subject to Cx ≤ b
Modelling language for LP'sBackends including Coin-OR, GLPK, GurobiSupports integer constraints
http://pythonhosted.org/PuLP/
PuLP simple LPPuLP Sudoku
![Page 26: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/26.jpg)
CVXPYNonlinear convex optimization, backed by CVXOPTThe LASSO -penalized least squares problem:L1
from cvxpy import *import numpy as npimport cvxopt
# Problem data.n = 10m = 5A = cvxopt.normal(n,m)b = cvxopt.normal(n)gamma = Parameter(sign="positive")
# Construct the problem.x = Variable(m)objective = Minimize(sum(square(A*x - b)) + gamma*norm1(x))p = Problem(objective)gamma.value = 0.2result = p.solve()return p.is_dcp()
![Page 27: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/27.jpg)
WIDE RANGE OF EXPRESSIONSkl_div(x, y)
KL divergencelambda_max(x), lambda_min(x)
the max/min eigenvalue of .log_det
for a positive semidefinite matrix .norm(x, [p = 2]),
L1/L2/infinity/Frobenius/Spectral norm of x
x
log(det(x)) x
![Page 28: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/28.jpg)
DERIVATIVES AND SYMPY
![Page 29: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/29.jpg)
LEAST SQUARES VIA THE FEYNMAN ALGORITHMWrite down the problem…
minimize f(x) = ( −∑i
xi ai)2
… think really hard …
![Page 30: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/30.jpg)
… AND THEN WRITE DOWN THE ANSWER!minimize f(x) = ( −∑
i
xi ai)2
Take derivativesThe gradient is flat at the bottom of the hill. We can find thegradient in each direction explicitly, and set it to zero:xi
= 2( − ) = 0∂f
∂xixi ai
![Page 31: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/31.jpg)
WHERE TO GET DERIVATIVES FROM?Numerical approximation - numdifftools(Automatic differentiation - Julia, Stan, ADMB…)Pencil and paper!Sympy
![Page 32: PYTHON FOR OPTIMIZATION - · PDF filePython for optimization Not optimizing Python programs ... GLPK, Gurobi Supports ... PuLP simple LP PuLP Sudoku](https://reader035.fdocuments.us/reader035/viewer/2022081507/5aa224f27f8b9a46238cae60/html5/thumbnails/32.jpg)
CONCLUSIONTo choose a solver:
Small problem/no hurryUse whatever you like!
Big problem/top speedUnderstand and specialize
Really big problemTry online methods like SGD
Don't understand the problem at all?Try a genetic algorithm