Global Optimization: For Some Problems, There's HOPE

Daniel M. DunlavySandia National Laboratories, Albuquerque, NM, USA

Dianne P. O’LearyDept. of Computer Science and UMIACS

University of Maryland, College Park, MD, USA

Copper Mountain Conference on Iterative MethodsApril 3, 2006

SAND2006-2001CSandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,

for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

Outline

•Unconstrained Minimization Problem•Homotopy Optimization Methods•Numerical Experiments•Protein Structure Prediction Problem•Numerical Experiments•Concluding Remarks

Problem

•Solve the unconstrained minimization problem

•Function Characteristics– Solution exists– Smooth ( )– Multimodal, deep local minima– Good starting points unavailable

Some Related Methods

•Stochastic search methods – Random perturbations

•Simulated annealing– Perturbations, acceptance criterion, schedule

•Evolutionary algorithms– Ensembles/populations

•Smoothing methods– Deformation of function being minimized

•Homotopy/continuation methods– Nonlinear equations,

Outline

•Unconstrained Minimization Problem•Homotopy Optimization Methods•Numerical Experiments•Protein Structure Prediction Problem•Numerical Experiments•Concluding Remarks

Homotopy Optimization Method (HOM)

•Goal– Minimize complicated nonlinear target function

•Steps to solution– Easy template function: , is a known local min.– Define a continuous homotopy function:

– Produce sequence of minimizers of w.r.t.starting at and ending at

Illustration of HOM

Homotopy Optimization using Perturbations & Ensembles (HOPE)

• Improvements over HOM– Produce ensembles of local minimizers of

by perturbing intermediate results– Increase likelihood of predicting a global minimizer

•Algorithmic considerations– Maximum ensemble size– Determining ensemble members

Illustration of HOPEMaximum ensemble size = 2

Considerations using HOPE

•Template function– Known solution, relation to target function

•Homotopy function– Continuous, but smoothness may help

•Parameters– Maximum ensemble size– Perturbation function/amount of perturbation– Number of perturbed versions produced– Number of steps taken in

•Metric for choosing ensemble members

Convergence of HOPE

•HOPE parameterized as existing methods– Probability-one homotopy methods for NLPs– Stochastic search methods

• Pure Adaptive Search/Improving Hit-and-Run

– Simulated Annealing

•HOPE converges in probability– Smooth homotopy: – Closed form for probability– Conditions for convergence with probability one

•Extension to general homotopy maps– One spatial dimension, – Basins of attraction must be known

Outline

•Unconstrained Minimization Problem•Homotopy Optimization Methods•Numerical Experiments•Protein Structure Prediction Problem•Numerical Experiments•Concluding Remarks

Numerical Experiments

•Test Problems– N-modal sine function:– Moré, Garbow, Hillstrom test functions

• 5 functions where local methods failed

– Pintér test function• Random multimodal functions, unique minimizer known

•Standard homotopy functions– Convex, probability-one

•Comparisons with local and stochastic search– More computation using HOM/HOPE– Better results and ensembles of local minimizers

Outline

•Unconstrained Minimization Problem•Homotopy Optimization Methods•Numerical Experiments•Protein Structure Prediction Problem•Numerical Experiments•Concluding Remarks

Protein Structure Prediction

Ala Arg Asp Asn Arg

Protein StructureAmino Acid Sequence

C CN

R

H OH

Given the amino acid sequence of a protein (1D), is it possible to predict its native structure (3D)?

Protein Structure Prediction:Computational Methods

•Molecular dynamics– Langevin dynamics approximated using stochastic

differential equation

•Bioinformatics– Sequence/structure matches to experimentally

determined native structures• Comparative modeling (threading, homology modeling)

•Energy Minimization– Find lowest energy conformation

• Native structure [Anfinsen, 1973]

Some Existing Energy Minimization Methods

•Local methods– Truncated Newton and quasi-Newton methods

• Memory efficient (second derivatives not stored)

•Global methods– Stochastic search, simulated annealing,

evolutionary algorithms, and smoothing methods– Other methods

• Convex global underestimation• Stochastic tunneling• Packet annealing• Derivative-free pattern search

Energy Minimization using HOM

•Goal– Minimize energy function of target protein

•Steps to solution– Energy of template protein:– Define a homotopy function:

• • Deforms template protein into target protein

– Produce sequence of minimizers of starting at and ending at

Energy Minimization using HOPE

•Extensions of HOM– Perturbations

• Specific to protein structure• Bond length, bond angle, and particle perturbations

– Ensembles• Ensembles chosen using homotopy function value

•Benefits over existing minimization methods– Take advantage of sequence-related structural

properties of template and target proteins

Outline

•Problem and Existing Methods•Homotopy Optimization Methods•Numerical Experiments•Protein Structure Prediction Problem•Numerical Experiments•Concluding Remarks

Backbone Model: Particle Properties

•Backbone model– Single chain of particles with residue attributes

– Particles model C atoms in proteins

•Properties of particles– Hydrophobic, Hydrophilic, Neutral

– Diverse hydrophobic-hydrophobic interactions

[Veitshans, et al., 1996.]

Backbone Model: Energy Function

Backbone Model: Energy Function

Backbone Model: Experiments

9 chains (22 particles) with known structureLoop Region

Hydrophobic HydrophilicNeutral

ABCDE F GH I

Backbone Model: Experiments

• 62 template-target pairs– 10 pairs had identical native structures

• Methods– HOM vs. Newton’s method w/trust region (N-TR)

– HOPE vs. ensemble-based simulated annealing (SA)• Different ensemble sizes (2,4,8,16)

• Averaged over 10 runs

• Perturbations where sequences differ

• Measuring success– Structural overlap function:

• Percentage of interparticle distances off by more than 20% of the average bond length ( )

– Root mean-squared deviation (RMSD)

Backbone Model: Results

MethodEnsemble

Size c = 0Success

(%) Mean cMean RMSD

Time (sec)

HOPE 2 33.40 54 0.14 0.17 354 43.10 70 0.08 0.11 658 54.60 88 0.03 0.04 115

16 59.00 95 0.01 0.02 200

SA 2 13.10 21 0.27 0.36 524 20.80 34 0.19 0.26 1078 28.50 46 0.13 0.19 229

16 40.20 65 0.08 0.12 434

Method c = 0Success

(%) Mean cMean RMSD

Time (sec)

HOM 15 24 0.36 0.38 10N-TR 4 6 0.45 0.55 1

Backbone Model: ResultsSuccess of HOPE and SA with ensembles of size 16 for each template-target pair. The size of each circle represents the percentage of successful predictions over the 10 runs.

SAHOPE

Outline

•Problem and Existing Methods•Homotopy Optimization Methods•Numerical Experiments•Protein Structure Prediction Problem•Numerical Experiments•Concluding Remarks

Conclusion

•New homotopy optimization methods– HOM: sequence of minimizers vs. path tracing– HOPE: perturbations and ensembles– Convergence (cast as existing methods)

•Numerical Experiments– HOM/HOPE outperform several standard methods– Standard test functions

• Standard homotopy functions used

– Protein structure prediction• Problem-specific homotopy functions• Take advantage of sequence-related protein properties

Future Directions

•Protein structure prediction– More realistic energy functions (AMBER, CHARMM)

– Protein Data Bank (templates)

– Different size chains for template/target

•HOPE for large-scale problems– Inherently parallelizable

– Communication: initializing ensembles at each step

•HOPE for other optimization problems– Constrained problems

•HOPE for other applications

Thank You

Daniel M. Dunlavy – HOPEdmdunla@sandia.gov

Publications• D.M. Dunlavy, D.P. O'Leary, D. Klimov, and D. Thirumalai

HOPE: A Homotopy Optimization Method for Protein Structure Prediction J. Comput. Biol., 12(10):1275-1288. Dec. 2005

• D. M. Dunlavy and D.P. O'LearyHomotopy Optimization Methods for Global Optimization Sandia National Laboratories, SAND2005-7495. Dec. 2005

• D.M. DunlavyHomotopy Optimization Methods and Protein Structure PredictionPh.D. Thesis, University of Maryland, College Park, Aug. 2005

• D.M. DunlavyGlobal Optimization of a Simplified Protein Energy ModelIn preparation