Frank SchmidtICE 5th of December 20121 1) Development of MAD-X with Space Charge and preliminary...

Frank Schmidt ICE 5th of December 2012 1

1) Development of MAD-X with Space Charge and preliminary Application to the Fermilab Booster

2) Benchmarking of PTC_ORBIT & SYNERGIA with Emphasis on Micro-Scale Stability

Visit at Fermilab (Aug – Nov 2012)Visit at Fermilab (Aug – Nov 2012)

F.S.F.S.


1) MAD-X development and applications Valery Kapin.

2) Orbit issues and possible remedies are being pursued together with Leonid Vorobiev. Continuous discussions with Valery Kapin and Giuliano Franchetti from GSI.

3) SYNERGIA installation on my Laptop and getting applications to work Eric Stern.

4) Collaboration with Sasha Valishev to introduce a the exact Hamiltonian into LIFETRAC (not discussed here).

AcknowledgementsAcknowledgements


Development of MAD-X with Space Charge and preliminary Application to the Fermilab Booster

References:

1. V. Kapin & Yu.Alexahin, “Space Charge Simulation Using MADX with Account of Synchrotron Oscillations”, Proc. RuPAC-2010, Protvino, Oct 27, 2010, pp. 204-206.

2. V. Kapin, “Space Charge Simulation Using MADX with Account of Longitudinal Motion”, FNAL Beamsdoc- 3582‑v2, Apr. 2011.

3. Y.Alexahin et al., “EFFECTS OF SPACE CHARGE AND MAGNET NONLINEARITIES ON BEAM DYNAMICS IN THE FERMILAB BOOSTER”, Proceedings of PAC07, Albuquerque, New Mexico, USA.

Task ITask I


MAD-X V3 with Space Charge for MAD-X V3 with Space Charge for debuncher (Valery Kapin 2010)debuncher (Valery Kapin 2010)

• Utilizing the BB elements to create a frozen space charge model but adapting emittances and Twiss parameters for the sigma determination.

• Using the MAD-X Macro technique for all operations

• Phase-I: Lattice Preparation

• Splitting the elements (at least once) and introduce SC kicks in between

• Making sure that the lattice is stable including SC and proper sigma values ( next slide)

• Transferring thick lattices into thin ones (symplectic and only way to track in MAD-X)

• Phase-II: Running (turn-by-turn due to Macro procedures)

• Time varying fields

• Emittance and TWISS calculation during run

• Output (losses, emittances, etc)


Procedure to determine Beam SizesProcedure to determine Beam Sizeswith large SC Tune Swings with large SC Tune Swings

• TWISS fails for integer and/or half integer tunes • Trick to avoid it: turn on SC tune-shift in small steps and let tunes converge at each step

• Lines represent Laslett formula

2

2.1

2.2

2.3

2.4

2.5

1

1.1

1.2

1.3

1.4

1.5

0 5 10 15 20

Qx_formulaQx_num_I/2Qx_num_I

Qy_formulaQy_num_I/2Qy_num_I

N_iter

Reference:

Valery Kapin talk at GSI 2009


Lattice with variable ParametersLattice with variable ParametersM.J.Syphers, “Status of Mu2e Operating Scenario”, Feb, 2010 Beams-doc-3556

Mu2e experiment Accelerator issue: 8 GeV Accumulator and Debuncher storage rings are used Compact bunches (<200ns, <0.025) are prepared in Accumulator ring Bunches are transferred to Debuncher ring and extracted out to the experiment using 3-rd

order resonant extraction with spill times ~150ms.

A stable, slow spill with a very intense proton beam is a big challenge.


• The MACRO technique of MAD-X is working fine but typically one ends up with very complex logic so that even Valery had to work hard to re-understand them after a delay of 2 years! Therefore quite unpractical for new applications.

• Macros are inherently slow!

• Therefore the idea was to reduce the use of Macros as much as possible and to modify the code to do most of the work directly within MAD-X.

• In detail:

• Time varying multipoles

• Time varying phase trombone (Forest/Schmidt: Don’t use them!)

• Time varying cavity voltage (application for Booster lattice)

• Include the sigma determination at each turn

• Allow for stop&go for the MAD-X tracking routine to allow intermediate TWISS calculation at start and the location of the SC elements

• Input via several TFS tables.

Task for MAD-X V5sp for the Task for MAD-X V5sp for the Booster as a new ApplicationBooster as a new Application


• I have left the MAD-X project 18 months ago.

• Laurent Deniau at CERN has taken over till then.

• I have been a manager of the code with limited involvement in the actual code writing other than the PTC part. In particular, the core of the code in C has been written exclusively by Hans Grote.

• In a state of emergency I had to be ready to fix the code quickly.

• MAD-X is not well suited for this task I since deferred evaluation

does not easily allow to put element parameters back into the database.

• Various C routines were needed but Hans did not have to help!

MAD-X Status MAD-X Status


N_macro_surv vs Turn Number N_macro_surv vs Turn Number for the Debuncher for the Debuncher

Timing on CERN computer

• Macro version ~20-24h

• MADX-5sp

~2-4h


Ex,y_rms vs Turn Number for the Ex,y_rms vs Turn Number for the DebuncherDebuncher


Longitudinal Emittance vs Turn Longitudinal Emittance vs Turn Number for the DebuncherNumber for the Debuncher


• MAD-X clean-up changes of the lattice. A small horizontal non-closure is tolerated for the time being.

• The cavities had to be replaced by zero-length cavities.

• The first order edge focusing of Bends has to be simulated with the DIPEDGE elements.

• The booster has a vertical bump and I had to fix a bug to allow for the vertical DIPEDGE element in MADX-5sp.

• Phase-I has been passed including the very large SC tuneshift of ~0.4 and a successful crossing of the half integer (0.48).

• Phase-II has been started and some turns have been tracked. However, it has become clear that for such a large SC tuneshift one has to carefully set the initial parameters like tunes etc.

Progress for the BoosterProgress for the Booster


Benchmarking of PTC_ORBIT & SYNERGIA with Emphasis on Micro-Scale Stability

References:

1. J.A. Holmes et al., “ORBIT: BEAM DYNAMICS CALCULATIONS FOR HIGH-INTENSITY RINGS”, Proceedings of EPAC 2002, Paris, France.

2. E.Forest, A.Molodozhenstev, A.Shishlo, J.Holmes, “Synopsis of the PTC and ORBIT Integration”, KEK Internal, 2007-4, November 2007.

3. L.G. Vorobiev and K.Hirata, “A symplectic Poisson Solver Based on Fast Fourier Transformation”, KEK Report 95-12.

4. L.G. Vorobiev, “Space Charge Templates for High-Current Beam Modeling”, FERMILAB-PUB-08-236-APC.

Task IITask II


H-DETUNING GSI SIS18H-DETUNING GSI SIS18

Giuliano Franchetti

Valery Kapin (2008)Shinji Machida

Giuliano Benchmark WEB page:

http://web-docs.gsi.de/~giuliano/


H-DETUNING PTC-ORBITH-DETUNING PTC-ORBIT


Fake Tune Evolution PTC-ORBITFake Tune Evolution PTC-ORBIT

Turns

Qx


Amplitude blow-up PTC-ORBITAmplitude blow-up PTC-ORBIT

Turns

X [

0.1

3]

RED: zero Amplitude

GREEN: 0.1 sigma Amplitude


Small Amplitude blow-up SYNERGIASmall Amplitude blow-up SYNERGIA

Turns

X [

0.1

3]


SYNERGIA CodeSYNERGIA Code

Synergia is an accelerator simulation package for combining advanced single-particle optics with computationally-intensive

collective effects.

https://compacc.fnal.gov/projects/wiki/synergia2

Synergia is designed to take advantage of computing resources ranging from desktops to supercomputers.

https://compacc.fnal.gov/projects/wiki/synergia2


Synergia PersonnelSynergia Personnel

Synergia is developed and maintained by the Computational Physics for Accelerators group in

Fermilab's Scientific Computing Division

James Amundson, Paul Lebrun, Qiming Lu, Alex Macridin, Leo Michelotti (CHEF), Chong Shik Park,

Panagiotis Spentzouris and Eric Stern

With development contributions from Tech-X:Steve Goldhaber


Physics Of SynergiaPhysics Of Synergia

Single-particle optics are provided by CHEF Direct symplectic tracking

Magnets, cavities, drifts, etc. (and/or) arbitrary-order polynomial maps

Apertures Collective (single and multiple bunches)

Space charge (3D, 2.5D, semi-analytic, multiple boundary conditions)

Wakefields Can accommodate arbitrary wake functions


Synergia FeaturesSynergia Features

Python-driven simulations Pure C++ simulations are possible All computationally intensive calculations in C++ One example: arbitrarily complex ramping, including feedback, is

straightforward Checkpointing

Simulations can be stopped and restarted arbitrarily Parallel scalability

Has run efficiently on 2^17 = 131,072 cores(!), simulating over 10^10 particles.

Can be extended in either Python or C++ User-defined Diagnostics New collective operators Etc.


Plans for SynergiaPlans for Synergia

More collective effects Electron cloud

In progress Beam-beam?

Cern related Features Double RF implementation Time varying fields Acceleration Foils Observer Particles

• Parsers for more lattice formats

• Mad8 finished• MadX close to done• others as requested

• Full documentation• End-user, API and examples

• Fixed API• Very close


““Unfair” speed comparison between Unfair” speed comparison between PTC-ORBIT and SYNERGIAPTC-ORBIT and SYNERGIA

• I have made a speed comparison of the SIS18, which is just a linear lattice plus space charge.

• For the same number of space charge kicks, the same grid bins and the same number of macro-particles. I find Synergia ~5 times faster than PTC-ORBIT.

• To avoid to be “unfair” one has to make a more careful analysis for various cases and with non-linearities.

• PTC allows a very general treatment of space charge without any need to split elements. This might be inherently inefficient.

• It also could be that up to now PTC-ORBIT is not yet fully optimized. Presently, we have started an attempt to profile the code to understand where the time is being spent. To identify remedies will take some time.


H-DETUNING SYNERGIA H-DETUNING SYNERGIA 32 Turns32 Turns

0.1

Qx


H-DETUNING SYNERGIA H-DETUNING SYNERGIA 1024 Turns1024 Turns

0.1

Qx


Sudden shift in Tune SYNERGIASudden shift in Tune SYNERGIA

Turns

X[m

]


Evolution of Linear Invariant Evolution of Linear Invariant SYNERGIASYNERGIA

Co

ura

nt S

nyd

er

Inva

rian

t


Analysis of the situation with PIC Analysis of the situation with PIC codescodes

• Originally, I had planned to follow the complete benchmarking procedure for PTC-ORBIT as prescribed by Giuliano. In view of a possible upgrade of ORBIT to PTC-ORBIT at Fermilab.

•Following this surprising results I checked out with Leonid every possible source of error that I might have done. In fact, he repeated the same studies with plain ORBIT and we find the same results.

• The conclusion is: that for PTC_ORBIT the benchmarking can simply not be done.

• We then contacted the author Jeff Holmes and his team and let them be the first to be

informed about this problem.

• A couple of weeks later I have received an Email by Jeff with a confirmation of our finding: “I have been doing a bit of work on the individual particle tracking with space charge. I set up a uniform focusing channel and a KV distribution. I observe the same numerical diffusion behavior seen by you and Leonid. If I symmetrize the distribution in the x-y planes, the diffusion is much slower and delayed, but eventually it does occur (due to roundoff?).”

• Lastly, Synergia behaves much better but is also suffering from this instability.


Remedies and DiscussionsRemedies and Discussions

• During the discussions with Jeff we have agreed to investigate possible remedies within the next 6 months.

• Independently, Leonid is reviving his decade old very careful “clouds in PIC” analysis and he also has a much improved Poisson solver.

• The much smoother distribution (see for comparison next 2 slides) did not in itself lead to a substantial improvement. But we still are hopeful that the full replacement of the space charge classes in ORBIT will bring the desired improvement.

• To a lesser extend SYNERGIA suffers also from this problem and we should be looking into that as well.

• Leonid also has a radically different approach that is to use the so-called space charge templates (see reference above), i.e. NOT based on PIC. It remains to be seen how this will work out!

• Following all these findings and the excitement I am happy to invite you to a space charge workshop to be held in April 2013 at CERN where we will discuss these issues in depth and hopefully with a better understanding!

• For me the crucial question is if PIC codes can be reliably used for 1’000 - 10’000 turns or more despite this micro-scale instability?


Shapes of Distributions (Leonid)Shapes of Distributions (Leonid)


Original ORBIT DensityOriginal ORBIT Density


Reversed Parabolic Density Reversed Parabolic Density (#3 in Shape Slide)(#3 in Shape Slide)


• MADX-5sp is going to be used at CERN as our frozen model tool for space charge studies. Any developments, bugfixes etc will be provided to Fermilab. Most probable Valery Kapin and Yuri Alexahin will use it here and we will certainly help in case of problems and/or further developments.

• Optimizing the ORBIT code: Leonid Vorobiev has started to replace the space charge classes in ORBIT. Our plan is to continue our collaboration remotely. But there has been discussions with his future employer to invite Leonid for an extended period to CERN to finish this work in the most efficient way. In fact, since we are investigating a possible speed-up of PTC-ORBIT we could at the same time make his implementation compatible with MPI.

Planned Collaborations IPlanned Collaborations I


• ORBIT authors: Leonid and myself have approached Jeff Holmes and his team to present out findings about the unstable particle motion on the micro-scale level. We have agreed with them to find a solution or at least achieving a dramatic improvement within the next 6 months. It could very well be that the MPI version of Leonid’s implementation will be the best that can be done.

• SYNERGIA: We have prepared in writing a program for a long-term collaboration for a possible use of this code for the LIU (LHC Injector Upgrade) at CERN. Missing code features have been identified and have been discussed with the code developers. Special attractions are the modern structure with a nice PYTHON interface and the high performance compared to other codes. We have also agreed to perform the full benchmarking cycle that has been prepared by Giuliano Franchetti at GSI.

Planned Collaborations IIPlanned Collaborations II


ReserveReserve


Synergia PersonnelSynergia Personnel

Synergia is developed and maintained by the Computational Physics for Accelerators group in

Fermilab's Scientific Computing Division

James Amundson, Paul Lebrun, Qiming Lu, Alex Macridin, Leo Michelotti (CHEF), Chong Shik Park,

Panagiotis Spentzouris and Eric Stern

With development contributions from Tech-X:Steve Goldhaber


Physics Of SynergiaPhysics Of Synergia

Single-particle optics are provided by CHEF Direct symplectic tracking

Magnets, cavities, drifts, etc. (and/or) arbitrary-order polynomial maps

Apertures Collective (single and multiple bunches)

Space charge (3D, 2.5D, semi-analytic, multiple boundary conditions)

Wakefields Can accommodate arbitrary wake functions


Synergia Code structureSynergia Code structure

A simulation consists of propagating a Bunch (or bunches) through a Lattice.

Inputs: machine lattice, initial bunch parametersOutputs: user-selected Diagnostics.

6d means 6d std deviations 6x6 covariance matrix 6x6 correlation matrix Individual particle tracks Dump of all particles Losses with locations in lattice

Some available Diagnostics

Actions may specify Diagnostics every n steps, every m turns, specified sets of turns or sets of steps, etc. User-defined logic is also possible.


Synergia FeaturesSynergia Features

Python-driven simulations Pure C++ simulations are possible All computationally intensive calculations in C++ One example: arbitrarily complex ramping, including feedback, is

straightforward Checkpointing

Simulations can be saved and restarted arbitrarily Parallel scalability

Has run efficiently on 2^17 = 131,072 cores(!), simulating over 10^10 particles.

Can be extended in either Python or C++ User-defined Diagnostics New collective operators Etc.


Status of SynergiaStatus of Synergia

Synergia is in production use at Fermilab Main Injector space charge Mu2e resonant extraction Booster with SC and wakefields

End-user release in near future


Potential with zero Boundary Potential with zero Boundary Condition Assumption (Leonid)Condition Assumption (Leonid)

Frank SchmidtICE 5th of December 20121 1) Development of MAD-X with Space Charge and preliminary...

Documents

Transcript of Frank SchmidtICE 5th of December 20121 1) Development of MAD-X with Space Charge and preliminary...