Summary & Outlook

References[1] A. Warshel and M. Levitt, J. Mol. Biol. 103, 227 (1976)[2] M. Eichinger, P. Tavan, J. Hutter, and M. Parrinello, J. Chem. Phys. 110, 10452 (1999)[3] K. Lorenzen, M. Schwörer, P. Tröster, S. Mates and P. Tavan, J. Chem. Theory Comput. 8, 3628 (2012).[4] M. Schwörer, B. Breitenfeld, P. Tröster, S. Bauer, K. Lorenzen, P. Tavan, and G. Mathias, J. Chem. Phys. 138, 244103 (2013)

[5] J. Hutter, A. Alavi, T. Deutsch, M. Bernasconi, S. Goedecker, D. Marx, M. Tuckerman, and M. Parrinello, CPMD: Car-Parinello Molecular Dynamics, v3.15.1, (c)IBM Corp 1990-2008 and MPI für Festkörperforschung Stuttgart 1997-2001, www.cpmd.org.

[6] P. Tröster, K. Lorenzen, M. Schwörer, and P. Tavan, J. Phys. Chem. B 117, 9486 (2013).

[7] IPHIGENIE is developed under GPL; for more information contact the corresponding author (gerald.mathias@physik.uni-muenchen.de). For a CPMD licence, visit www.cpmd.org .

Our new DFT/PMM approach combines PMM force fields with grid-based DFT in an efficient and accu-rate way. It is implemented in the IPHIGENIE/CPMD program package, which is now available in a MPI/OpenMP-parallel version and enables large-scale DFT/PMM-MD simulations of molecules in con-densed phase.

■ Our implementation scales even for a relatively small simulation system almost perfectly up to 512 Intel Sandy Bridge cores, with still reasonable performance gain up to 2048 cores. ■ We are now ready to study solvation effects of polarizable environments on solute molecules such as nucleobases in DNA or chromophores.

Weak scaling and replica exchange: up to 10.000‘s of cores■ The well-known excellent weak-scaling performance of CPMD is obviously not hampered by the inter-face to IPHIGENIE and we are already setting up large-scale DFT/PMM simulations of relevant biomo-lecules on SuperMUC.■ Furthermore we are currently developing a replica exchange formalism for the DFT/PMM setup, which will add a weak- scaling level of parallelism and thereby increase the number of usable cores per pro-blem by another order of magnitude.

Financial Support: SFB 749, KONWIHR III

4

8

16

32

64

128

256

512

1024

2048

OpenMP threads / MPI process

124816

4 8 16 32 64 128 256 5121024 2048number of cores

spee

dup

calculate fieldfrom charges andextrapolated dipoles

calculate fieldfrom dipoles

DFTcalculate new dipoles

convergence?

integration

convergence?

YESNO calculate new dipoles

DFT

DFT DFT

YESwas

set

?

DFT DFTset

NO

NO

YESDFT DFT

Fig. 8: The joint DFT/PMM-SCF loop with two DFT convergence criteria [4].

We have implemented the above ap-proach in the OpenMP/MPI-parallelized PMM-MD C-code IPHIGE-NIE [3,4,7], which also features an interface to the DFT program CPMD [5]. For this purpose, the FORTRAN77 code CPMD is compiled as a library and statically linked to IPHIGENIE.

■ To use the optimal number of MPI processes for both program parts, a subset of NDFT „DFT only“ pro-cesses can be split off the total number of MPI processes NMPI = NDFT + NPMM+DFT.This subset skips the PMM part of the code and is synchronized with the re-maining NPMM+DFT processes before entering the DFT part (Fig. 7).

■ A joint DFT/PMM SCF iteration se-quentially employs two convergence criteria χDFT in the DFT calculation (Fig. 8) [4]. Thereby, the computatio-nal effort of a DFT/PMM hybrid simulation is less than a factor of two compared to a DFT/MM (unpolarizable) setting.

Strong scaling up to 2048 cores for a small moleculeThe strong scaling on LRZ‘s petaflop system SuperMUC was tested selecting an alanin-dipeptide molecule (22 atoms) as DFT fragment and describing the aqueous environment by 2114 polarizable TL4P water molecules [6]. The image in the top box shows the alanin-dipeptide together with a few selected solvent water molecules and the electrostatic potential they create.

■ Figures 9a) and b) demonstrate that the hybrid approach scales almost linearly up to 512 cores with still reasonable performance gain up to 2048 cores even in this rather small setting. Note here that for this small system IPHIGENIE scales up to NPMM+DFT=128 processes, which corresponds to only ≈ 66 atoms per process.

■ For the largest PMM-only system tested so far (1.5M atoms) IPHIGENIE (without CPMD) scaled up to 2048 cores, although it has not been optimized yet for such large atom counts.

IPHIGENIE CPMD

program start

DFT only?

CPMD startupIPHI startup

PMM calc. DFT calc.

PMM calc DFT calc.

IPHI �nalize

CPMD �nalize

star

tup

phas

eca

lcul

atio

n ph

ase

�nal

ize

phas

e

(DFT only processes)

... ...

Fig. 7: The program flowchart. „DFT only“ MPI processes are depicted as red arrows. The blue arrows symbolize pro-cesses which are used in the PMM and the DFT calculation.

YESNO

Fig. 9a) and b): Scaling of our IPHIGENIE/CPMD imple-mentation for 22 DFT- and 8456 PMM-atoms. DFT/PMM-MD-setup: MT/BLYP, Ecut=80 Ry χDFT= 10-6, ∆t=0.5 fs.

a)

b)

Fig. 4: The dimer setup for exami-ning conservation of energy.

DFT PMM

Implementation

a) DFT

b) PMM, l = 0

c) PMM, l = 2

0 0.5 1 1.5 2t / ps

E /

10-

3 k

cal/m

ol

0

5

5

0

5

5

0

5

5

Fig. 5: Energy conservation in reference simulations of the water dimer. a) DFT Born-Oppenheimer dynamics. b) PMM dynamics at close contact (exact electro-statics) and c) softly restrained to a distance dOO=10 Å (pure SAMM4 electro-statics) [4].

a) l = 0

b) l = 1

c) l = 2

0 0.5 1 1.5 2t / ps

E /

10-

3 k

cal/m

ol

0 5

5

0 5

5

0 5

5

Fig. 6: Energy conservation in DFT/PMM hybrid simulations of the water dimer with the electrostatics treated at different distance class levels l. a) Close contact b) softy restrained to dOO=7 Å, and c) to dOO=10 Å (SAMM4 electrostatics) [4].

To verify the Hamiltonian character of our approach, we check the energy conservation for a hybrid water dimer (Fig. 4), whose hydrogen bond donor is described by DFT, while the accep-tor is modeled by the polarizable TL4P potential [6]. ■ Fig. 6 demonstrates that the energy of the hybrid system is very well conserved in DFT/PMM-MD as is apparent from the PMM-MD and DFT-MD reference simulations in Fig. 5.

Conservation of energy

i

μ

Cμ0 Cμ

1

γ∈Gμ

x

j

u

xu

Cμ2

k

► Electrostatic forces on the DFT atoms

i

μ

Cμ0 Cμ

1

γ∈Gμ

x

j

u

xu

Cμ2

k

Fig. 2: Evaluation of the external potential Фext at the grid points γ. Contributions from Gaussian charges and induced dipoles of a close PMM atom i are evaluated directly (orange), whereas the contributions from a more distant atom j are calculated (at level l = 1) by a Taylor expansion (red) around the position of the DFT atom μ. These Taylor expansi-ons get additional contributions from the multipole moments of distant clusters at higher levels l > 1 (purple) [4].

► Electrostatic forces on the PMM atoms

elecDFT/(P)MM ext( ) ( )H d ρ= Φ∫ r r r■ Task: calculate all DFT/PMM interaction forces as efficiently as possible from

x

x

i

j

Cμ2Cμ

0 Cμ1

uuγ∈Gμ

μ

► Hierarchical decomposition of the system

⇒ an efficient and accurate Hamiltonian DFT/PMM multiscale hybrid method!

MM PMM DFT DFT/(P)MMH H H H H= + + +

vdW bonded elecDFT/(P)MM DFT/MM DFT/(P)MM DFT/(P)MMH H H H= + +and

■ We combine standard MM force fields extended by Gaussian inducible dipoles with grid-based DFT:

, 2PMM

,

1 1 1( | , , ) · ( ) /2 2 2i

qi i j j j i i i i

i j i i iH q σσ α

≠

= Φ − ⟨ ⟩ +∑ ∑ ∑pr p r p E r p

with[1]

Fig. 3: Evaluation of the action of the discretized DFT charge density on the PMM atoms. Grid charges act directly on the close PMM atom i (orange). Actions on more distant atoms j are calculated through the multipole moments of the charge distribution of DFT atom μ (red). The atomic moments within a DFT unit u are then combined to compute the actions on the PMM units u‘ (l > 1) where atomic forces are obtained through Taylor expansions (purple) [4].

Fig. 1: The simulation system is decomposed into a hierarchy of nested clusters, whose sizes increase with the level l of the hie-rarchy. Electrostatic interactions of a DFT atom μ are calculated differently for associated distance classes Cμ

l. A Voronoi tessella-tion of the DFT box assigns the grid points γ to the DFT atoms μ enabling at l = 1 the formation of atomic charge distributions. At the next higher level l = 2 they are combined into so-called structural units u. Analogously, in the PMM fragment the atoms are combined into such units u‘. Then units are combined to larger clusters, larger clusters to super clusters, etc. [4].

■ SAMM4: „structure adapted fast multipole method“, enables the cal-culation of electrostatic forces exactly obeying Newton's reaction prin-ciple with minimal com-putational effort [3].

■ The external potential Фext is computed on the DFT grid from the PMM parti-al charges and induced dipoles differently for each interaction shell using direct interaction, Taylor expansions and multipole expansions.

Hamiltonian DFT/PMM coupling

■ After convergence of the DFT-SCF iteration, the action of the discretized DFT charge density on the PMM atoms is compu-ted symmetrically to the import of the external po-tential.

■ Strategy: extend the fast multipole scheme SAMM4 [2, 3] to symmetrically compute on the DFT grid the external electrostatic potential Фext generated by the PMM atoms and in the remaining simulation system the reverse action of the electronic density on the PMM atoms

4 8 16 32 64 128 256 512 10242048number of cores

0

5

10

15

20

25

30

35

ps o

f tra

ject

ory

/ day

best setupideal

Motivation

A strategy for developing such accurate PMM models has been developed in our group yielding promising models for simulating a liquid water environment [6].

Hybrid molecular dynamics (MD) simulations, in which the forces acting on the atoms are calculated by grid-based density functional theory (DFT) for a solute molecule and by a polarizable molecular mechanics (PMM) force field for a large solvent environment, pose a challenge regarding both the theoretical modeling and the computational realisation. The Hamiltonian DFT/(P)MM approach presented here guarantees energy conservation, ex-cludes artificial distortions of the electron density at the interface between the DFT and PMM fragments, and treats the long-range electrostatic interactions within the hybrid simulation system in a linearly scaling fashion using hierar-chically nested fast multipole expansions [Schwörer et al., J. Chem. Phys. 138, 244103 (2013)].

The figure on the right illustrates the test case of a small alanin-dipeptide sol-vated in bulk water. The scaling behaviour of this system is shown below.

Our approach links the MPI/OpenMP-parallel PMM-MD code IPHIGENIE [3,4,7] to the well-established MPI/OpenMP-parallel DFT code CPMD [5] and offers an unique tool which

■ combines a highly accurate DFT description of a localized subsystem with a PMM description of the extended condensed phase environment that greatly enhances the accuracy compared to conventional unpolarizable MM force fields

■ scales almost perfectly up to 512 Intel Sandy Bridge cores even for a small simu-lation system made up of only 22 DFT atoms and 8456 PMM solvent atoms (which corresponds to as few as 66 PMM atoms per PMM-MPI process)

■ makes studies of large DFT molecules solvated in accurately modeled condensed phase feasible

■ is developed under GPL (only IPHIGENIE part, see [7] for contact information).

Achievements

A Scalable Hamiltonian hybrid DFT/PMM-MD approach for accurately simulating biomolecules on SuperMUC

M. Schwörer*, P. Tavan*, F. Jamitzky§, H. Satzger§, and G. Mathias**Theoretische Biophysik, Lehrstuhl für Biomolekulare Optik, LMU München; §Leibniz Supercomputing Centre (LRZ), Munich