TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE LS1 The upgrade of 600 A ENERGY EXTRACTION SYSTEMS TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 600 A EE; what, where, how ? Reliability Hardware Failures to be cured Why upgrade? Failure causes Solutions LS1 actions Resources Planning Showstoppers Non LHC activities during LS1 Overview TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Energy Extraction 600 A circuit TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 600A Energy Extraction Systems 202 systems installed in the LHC tunnel in corrector circuits with stored energy between 2.2 and 150 kJ In 15 different locations; 8 x UA parallel service tunnel and 6 x RR and 1 x UJ tunnel caverns Systems developed in close collaboration between CERN and the Budker Institute of Nuclear Physics (BINP), Novosibirsk, Russia. TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Reliability Systems reliability from protection point of view: No opening failures during Fast Power Abort events or tests (> openings since 2005) 2 independent opening mechanisms/circuits per breaker; guaranteeing 6-fold redundancy Calculated by Antonio Vergara in December 2003: System Failure probability in 20 years of operation < 0.01% (Preventive maintenance!!) TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Reliability (2) Systems reliability from hardware failure point of view: 6 hardware failures in hardware failure in * hardware failures in 2012 For operation: 2 failures per year not in shadow Mean Time To Recovery : 4 hours (including preparation, access and travel) All failures were fail-safe for the s.c. circuits reducing the availability of the system. Changes with respect to 2010 operational year : Preventive maintenance with help of automatic analysis and tracking software tool. TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Hardware Failures treated during LS1 5-10/year - 95% of all failures: Breaker refuses to close Loosening of holding coil mechanism LHC re-start delayed 1/year - Breaker is physically open but indicates CLOSED micro-switch not actuated / main axle LHC re-start delayed 1/year - Breaker main contact re-bounces on micro-switch main axle LHC re-start delayed TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Why do we upgrade? Correcting measures are relatively easy < 100 kCHF This upgrade will improve the availability of 600 A EE systems for LHC operation by 4-8 hours per year. Less preventive maintenance TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Failure causes 1- Loosening of holding coil mechanism Weak design of U-shape 2- Micro-switch not actuated Tolerances in dimensions of housing 3- Main axle fixation Weakness during production not an increasing fault TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 1- Loosening of holding coil mechanism Weak design of U-shape and dimensional tolerances Solutions Needs to achieve Plate Glue Action in all 606 breakers Open all breakers (anyway foreseen for regular maintenance) Plate gluing Once open: 2-3 minutes per breaker TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 2- Micro-switch not actuated (temporarily) Tolerances in dimensions of housing Already tracked by automatic analysis software tool Leading to an unscheduled but still preventive maintenance intervention (in the shadow) On-going but to be extended with advice in form of logbook entries (MPE-MS) Solutions cntd. TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 3- Main axle fixation A loose main axle means no current in 2 oo 3 phases in the CB leading (> 200A circuit current) to overheating of the 3 rd phase. Weakness during production no increasing numbers Manufacturer has improved this process for new breakers For existing breakers no solution.. But.. We can track it because of known early symptoms Solutions cntd. Already tracked by the same automatic analysis software tool but with very few cycles left we can measure the System Resistance! Leading to an unscheduled corrective or preventive maintenance intervention TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 3- Main axle fixation tracking improvement - Measurement Solutions cntd. Needs to achieve Improvement of existing Voltage Measurement Channel Replacement of Voltage Divider by low impedance divider and noise reduction components (270 pcs MPE-EM) Replacement of Measurement Board by low noise separation amplifier (270 pcs: MPE-EM) Replacement of DQAMS by 12 bit to 24 bit ADC (MPE-EP & EM: 220 pcs) Voltage measurement taps to be improved Hi-level SW for on-line analysis (MPE-MS) TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE 3- Main axle fixation (and loose bolts) Save the EE system in case we dont track (or too late) it with software tools Overcurrent in one or two phases of the breakers will lead to over- temperature on the Equalizing Resistor and eventually destroy all breakers in one system Installation of a Thermostat on each Equalizing Resistor will trip the system early before hardware damage is done A spare internal interlock channel is available and wired up to the power board. Solutions cntd. Needs to achieve Installation of Thermostats Reprogramming of Firmware in all 404 interface cards (MPE-EP) Allow trip on over temperature of Equalizing Resistor Profit to limit cycles of the breakers TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE LS1 Actions Breakers - Regular maintenance and Consolidation Equalizing Resistors Installation/wiring of Thermostats Voltage measurement-channel improvement Visual inspection of main axles Endoscope IST SCT HWC TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Resources MaintenanceIST Installation man-hours T/TEman-hours E/TE per point Even point - 2 UAs10030 Odd point - 2 RRs4010 P7 right 3010 P7 left 3010 UJ Total LHC approx. : 630 h 160 h 130 man-days 6 man-months TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Planning (not incl. SCT and HWC) ANYTIME ANYWHERE April 2013 April 2014 6 man-months Team of 3 persons could come 2 x 1 month Man-power Calculations based on one team of Specialists from BINP Perfect preparation before start of activity One Engineer or TE as team leader plus two Technicians Working in a train Start to Finish one point, including IST But.. TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Planning SCT & HWC SCT & HWC 2014 SCT was presented as not needed for 600 A EE but latest tests show.. Yes, if we could we should 1 day per UA or RR 2 TE or E 15 working-days - NO recourses allocated yet HWC like SOF test PLI3.b1 remotely from CCC Automatic Analysis of PM PNO.b1 1 day per UA or RR 2 TE or E 15 working-days - NO recourses allocated yet TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Showstoppers Preparation for LS1 Planning Tests Tooling Purchasing (thermostat-study still to be finished) A LONG WAY TO GO!!! No reference contract with Russia If done by non-experts More CERN staff needed for training, the actual job, follow up, debugging of collateral damage etc. Might delay re-start after LS1. Spare Circuit Breakers: DZNVA Visit in March Manufacturers premises in very bad state and only 20% occupied - might stop delivering (special) breakers Currently: 28 spares (13 operational 15 to be qualified before TS4) Consumption average: 8-10 per year (including preventive changes) TE-MPE LS1 workshop November 22-23, 2012 G.J. Coelingh TE-MPE-EE Non LHC activities during LS1 Currently low in spare systems (