R2E and Availability Workshop Interlock Systems I.Romera, M.Zerlauth, B.Todd, S.Gabourin, P.Dahlen,...

R2E and Availability Workshop

Interlock SystemsI.Romera, M.Zerlauth, B.Todd, S.Gabourin, P.Dahlen, R.Mompo,

Y.Bastian, C. von Siebenthal + many colleagues for discussions & contributions

R2E & Availability Workshop 15th October 2014 Interlock Systems – IRR & MZ 2

Outline

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System

Quench Protection System

Power Converters

Cryogenics Auxiliary Controllers

Warm Magnets

Experiments

Access System

Beam Loss Monitors (Arc)

Collimation System

Radio Frequency System

Injection Systems

Vacuum System

Access System

Beam Interlock System

Control System

Essential Controllers

General Emergency Stop

Uninterruptible Supplies

Discharge Circuits

Beam Loss Monitors (Aperture)

Beam Position Monitor

Beam Lifetime Monitor

Fast Magnet Current Changes

Beam Television

Control Room

Software Interlock System

TimingSystemPost Mortem

Safe Machine Parameters

• Beam and magnet interlock systems provide

connectivity and diagnostics of >10.000 interlock

conditions

• Systems designed to be fast and highly reliable

(<1 false dump/year/system)

• Radiation constraints a priori considered during

system design

• Availability not related to beam energy/intensity

• Run 1 experience and outlook to HL-LHC



PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System


Power Converters


Warm Magnets

Experiments

Access System


Collimation System


Injection Systems

Vacuum System

Access System


Control System




Discharge Circuits





Beam Television

Control Room





Beam Interlock SystemExample for LHC - Beam 2(Duplicated for Beam 1)

- 17 BICs all over the LHC, 2 for each point (right and left), plus 1 in the CCR- Similar architecture for SPS, tree architecture in SPS-LHC TL + LINAC4

LHC BIC in UA63


BIS system components

CIBM

D &

CIB

TD

Parti

ally

loca

ted

in

Radi

ation

env

ironm

ent


Dependability vs Configuration (FMECA)


BIC availability during Run1(seen by the beam/OP)

Freq

uenc

y to

Dat

eIntervention [Access]

Shadow Specific Several More

Week

Month

Year

Once

Communication lost with CCR BIC(bug in FESA class)

Spurious dump from OP button(contact issue)

Glitches on Optical Fibers(CIBO exchanged)


Failure of VME processor (RIO3) in SPS (x8)

VME bus access failure in SPS (3x)

CIBU failures (1x critical NC in UJ33)

BIC availability 2006-2012(seen by the equipment expert)

not enough data…


BIC availability during Run1(seen by the beam/OP)

Freq

uenc

y to

Dat

e

Intervention [Access]


Week

Month

Year

Once

Communication lost with CCR BIC(bug in FESA class)

Spurious dump from OP button(contact issue)

Glitches on Optical Fibers(CIBO exchanged)


CIBU PS (redundant)

Accelerator Fault Tracking Project (AFT) will be an asset also for equipment teams!

Failure of VME processor (RIO3) in SPS (x8)

CIBU failures (1x critical NC in UJ33, 4x test & monitor)

VME PS (redundant)

VME bus access failure in SPS (3x)

BIC• Radiation

• BIS hardware not rad-tolerant and hence installed in protected areas (UAs, US,..), apart from client boxes (CIBUs)

• Equipment relocated from UJ56 to USC55 in LS1, expect no major gain with further relocation

• Some 10 oo 250 CIBUs remain in (low) radiation areas as installed in client racks• CIBUs located in UX85, UX45, RR73, RR77 can be affected during Run3 and HL-LHC

according to predictions• Most critical component (CPLD XC95144) radiation tested

• Optical fibers for beam permit loop (non-intrusive monitoring system currently under test)

• Obsolescence • Chassis VME (ELMA replacement?)• CIBM (BIC Master) and CIBU: Xilinx (CPLD 95xx and FPGA Spartan 2 & 3)• CIBO (optical boards): ELEDs


BIC• Improvements / Maintenance

• System design is sound and highly dependable = available, exchange of RIO3 -> MENA20 expected to bring the 20% availability increase

• Maintenance to remain roughly as is, monitoring of fiber optics loop as preventive measure

• No increased radiation effects expected up to LS3 (SEU and TID), profit from further client relocation/new layout in e.g. IR1/5

• Aging of components will result in more electrical failures of components (today well < expectations)

• Outlook to HL-LHC / new inventory• BIC V2.0 foreseen for (latest) HL-LHC era to address new requirements and

obsolescence of components, would profit from high-reliability FECs • New installations foreseen in PSB (LINAC4 connection) and PS

Low

stati

stics

!


Powering Interlock System

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System


Power Converters


Warm Magnets

Experiments

Access System


Collimation System


Injection Systems

Vacuum System

Access System


Control System




Discharge Circuits





Beam Television

Control Room




• Distributed system (2-6 controllers /

insertion region) providing interlocking

between power converters, QPS,

cryogenics, UPS, AUG and Access

• Interfacing magnet powering with BIS

•Vital role in diagnostics of powering events


Powering Interlock System36x SIEMENS PLC modules + redundant supplies

36x Remote IO modules 120x Profibus slaves, 36x CPLDs, 2000x Optocouplers

36x Power modules (5V/5V/24V) 220x AC/DC modules

150x Patch Panels + daughterboards1800x Current sources, 1400x OptocouplersPa

rtial

ly lo

cate

d in

Rad

iatio

n en

viro

nmen

t


PIC availability during Run1

Freq

uenc

y to

Dat



Week

Month

Year

Once

SEU in PLC (UJ14, UJ16, UJ56)

PSUOptocoupler(damaged during TS)

Current loop spurious triggers (not fully understood ?!)


Powering Interlocks • Radiation

• None of the COTS components is considered rad-tolerant and hence installed in protected areas (UAs, US,..) or redundant (power supplies)

• During LS1 9 systems fully relocated as part of R2E efforts (UJ56, UJ14, UJ16) – PLCs already relocated during TS end 2011

• Few remote I/O modules remain in low radiation areas (RRs in point 1/5/7)

• Components radiation tested in Louvaine la Neuve (60MeV, <5E8p/cm2s) for up to ~ 300 Gy, optocouplers & CPLDs to 150Gy

• Obsolescence • CPLD• Remote I/O modules• SIEMENS PLCs (319 series)


Powering Interlocks • Improvements / Maintenance

• Maintenance to remain roughly as is, might need at some point campaign to replace opto-couplers in most exposed areas

• Today no obvious handle to improve availability (no HW failure seen in 6 years)?!

• Spares might become an issue if we see increased electrical failures

• Outlook to HL-LHC / new inventory• No (increased) radiation effects expected up to LS3, HL-LHC probably

brings removal of last components in RRs (PC and QPS)• No new installations planned (sc magnets), apart IR1 & IR5 for HL-LHC• PIC V2.0 foreseen for (latest) HL-LHC era to address new requirements and

obsolescence of components

Low

stati

stics

!


Warm Magnet Interlock System

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System


Power Converters


Warm Magnets

Experiments

Access System


Collimation System


Injection Systems

Vacuum System

Access System


Control System




Discharge Circuits





Beam Television

Control Room




• Distributed system (8 controllers in LHC)

providing interlocking between power

converters, water cooling and magnets

• Interfacing magnet powering with BIS

•Vital role in diagnostics of powering events


~ 30 Controllers ~ 500 I/O modules~ 2500 magnets

LHC (2007)

TT60 & Ti2 (2005)

TT66 (2010)

4 systems

TT40 & Ti8 (2004)

2 systems

TT41

LEIR (2005)

1 system

8 systems

LINAC3 (2009)

(2005)

WIC installationsSPS(2014)

7 systems

1 system

1 systemPSB(2013)

4 systems

LINAC 4 (2014)

2 systems


SIEMENS PLC modulesCPUs, memory card, Ethernet module, F 24DI

Power Supply crate Redundant 24V/10A power supplies, redundancy module

WIC renovation: Cost estimation HW

Remote I/O crate Safety DI and redundant DO modules

Magnet interlock boxesMechanics, PCB, electronic components, connectors

Parti

ally

loca

ted

in

Radi

ation

env

ironm

ent


WIC availability during Run1

Freq

uenc

y to

Dat



Week

Month

Year

Once

Network glitches (PC trip)

SEU in TI8 remote I/O crate (beam shot on close-by collimator)


Warm Magnet Interlocks • Radiation

• COTS components not considered rad-tolerant and hence installed in protected areas (UAs, US,..)

• Exception is remote I/Os installed in TI2/TI8• Radiation tests in TCC2 & TT60 (2002&2003) and PSI (2003) up to 250

Gy for chosen type with one single SEU, Magnet boxes to 1MGy• One PLC relocated from US85 to UA83 during Run 1• Main concern is new deployments in injectors (PS, North Area…)• Investigating new remote IOs to stay closer to clients and optimize cable

lengths

• Obsolescence • Entirely based on COTS, compatible replacements can be found (but

need revalidation)


Radiation Tests @ CHARM• Preparing radiation test of low cost low density I/O module• Robust, IP56• Would allow cost efficient architectures for PS, TT2,…


Warm Magnet Interlocks • Improvements / Maintenance

• Maintenance to remain ~ as is• Today no obvious handle to further improve availability • Spares might become an issue for modules in ‘radiation areas’

• Outlook to HL-LHC / new inventory• No (increased) radiation effects expected up to LS3 in LHC• >> new installations/renovations in the pipeline in injectors (PS, TL L4-PSB,

TL PSB-PS, TT2, TT20, North Area, East Area, AD,..) • New architectures possible if radiation tolerant I/O module can be found

Low

stati

stics

!


Fast Magnet Current Change Monitors

PowerInterlock

Controllers

BeamInterlockSystem

Beam Dumping System


Power Converters


Warm Magnets

Experiments

Access System


Collimation System


Injection Systems

Vacuum System

Access System


Control System




Discharge Circuits





Beam Television

Control Room




• 12 devices installed in LHC, 14 in SPS-

LHC Transfer lines to protect from

beam excursion due to powering

failures in nc magnets

• Dedicated electronic development in

collaboration with DESY


Fast Magnet Current Change Monitors

26 FMCM modulesFPGA, memory, analogue circuity, power supply

26 Voltage dividers & isolation amplifiersIsolation amplifier

Parti

ally

loca

ted

in

Radi

ation

env

ironm

ent


FMCM

Freq

uenc

y to

Dat

e

Intervention [Access]


Week

Month

Year

Once

Defective earth connection in power converter RD34.LR7 (end 2011/ beg 2012)

Network glitches(thresholds vs PC sensitivity)

Isolation amplifier in SR3


FMCM vs electrical perturbations

RD1 RD34 RQ5 RQ4 RBXWTV RMSD0%

10%

20%

30%

40%

50%

60%

70%63%

56%

15%10%

13% 13%

Warm magnet trips

EXP magnets, several sectors, RF,…tripped

No beam, no powering (CRYO recovery)

No beam in SPS/LHC, PS affected

Majority of perturbations < 100ms/-20%

0 100 200 300 400 500 600 700

-50%

-40%

-30%

-20%

-10%

0%

10%

Duration [ms]

Vari

ation

[%]

• Thyristor converters revealed poor rejection of network perturbations • Used for powering normal conducting magnets in high beta regions of the LHC,

dump septas,… (RD1, RD34, RQ4/5 in IR7, RMSD,…)• 24 preventive beam dumps during 2012 operation, mainly during summer months• RD1 and RD34 most affected due to network topology and rated power• EPC started development of switched mode converter, deployment for RD1

envisaged 2015/16


Fast Magnet Current Change Monitors• Radiation

• Not considered radiation tolerant, installed in radiation free areas• Sole (potentially) exposed unit relocated from UJ56 to USC55 during

LS1

• Obsolescence • YES! RAMs, Shift registers,…


Fast Magnet Current Change Monitors• Improvements / Maintenance

• Maintenance to remain roughly as is• New switched mode power converters will greatly reduce number of

FMCM dumps • Spares might/will become an issue if we see increased electrical failures

• Outlook to HL-LHC / new inventory• Will have to remain outside radiation areas• Additional deployments possible or HL-LHC (Crab cavities?!), additional TL

elements,…• 2nd production batch or new development foreseen for next year to

address obsolescence of components


Summary• Interlock Systems have performed up to (or better) than initial estimates,

however statistic is still (too) low

• Few (unexpected) weak points affecting availability have been or are being addressed (R2E, MENA20, optical loops, power converter re-design,…)

• System performance expected to be stable up to LS3, after that new designs mainly due to component obsolescence (reaching end-of life of components) and partially new operational requirements

• Upgrades would profit from common developments towards radiation tolerant / high dependability FECs and data links

• Should maintain active exchange platform (between equipment groups, experiments, industry,..) concerning reliability engineering (AWG) / radiation tolerant designs (R2E)


We need to know the inventory of electronics that need to be considered for radiation effects…1) Remaining Inventory:· What options are there for system relocation and what are the advantages/disadvantages?· What equipment will remain in affected environments?· Do you expect the amount of equipment exposed to radiation to be lower, equal, or higher than Run1?· Do you expect problems in the injectors? 2) New Inventory:· What new developments are planned?· What are they motivated by? [e.g. operational requirements, obsolescence requirements, …] We need to know whether cumulative effects due to radiation and electrical reliability are known…3) Cumulative Effects· What are the predictions for cumulative radiation effects and electrical reliability?· How is your long term supply of system components ensured? [e.g. WorldFIP] We need to know if access to the machine can be expected to be lower, stay the same, or higher, and whether it will be due to radiation….4) Maintenance Expectations· Do you expect the maintenance of your system to be lower, stay the same, or higher?· Do you have a preventive maintenance strategy?· Do you expect the ratio of access : remote interventions to be lower, stay the same or higher? We need to know what engineering effort would be needed to improve the system level availability…5) Improving Availability· What effort would be needed to make a 20% improvement in your system availability We need to know whether there is common ground to develop a new system, designed specifically for the LHC machine radiation environment…6) Signaling and Communications· At the interface to the machine, what are the signals and bandwidth requirements for your system?· What is the communications mechanism between your system and controls? [Ethernet, WorldFip, …]

R2E and Availability Workshop Interlock Systems I.Romera, M.Zerlauth, B.Todd, S.Gabourin, P.Dahlen,...

Documents

Transcript of R2E and Availability Workshop Interlock Systems I.Romera, M.Zerlauth, B.Todd, S.Gabourin, P.Dahlen,...