Review of Hardware Commissioning May 11-13, 2005 Hardware Commissioning Co-ordination Organisation...

Review of Hardware CommissioningMay 11-13, 2005

Hardware Commissioning Co-ordination

Organisation

Roberto Saban the logic day to day organization safety decision making failure scenarios/fall-back solutions my worries

Roberto Saban TS/HDO HC Review 050511

subsystems ready

the logic

all systems ready for machine check out

individualsystem tests

hardwarecommissioning

InjectionSystem

RF CollimatorSystem

SuperconductingCircuits

ShortCircuitTests

we assume everything is installed before the Hardware Commissioning starts

two types of documents : as designedas tested

impact on equipment safetyimpact on equipment/system performanceimpact on machine performance


preparation

the scopeall the accelerator systems in the LHC tunnel with particular emphasis to superconducting circuits

documentationdocumentation the programthe program the schedulethe schedule the manpowerthe manpower

AB/CO

...

AB/PO

AB/BT

TS/LEA

TS/EL

TS/CV

TS/CSE

AB/ATB

AT/VAC

AT/ACR

AT/MEL

HCC

individual system tests

commissioning

failure scenarios

x 48


preparation : software

requirementsrequirements prototypesprototypes

procedures

commissioning scenario

x 10

sacec : software applications for the commissioning of the electrical circuits

AB/PO

AB/CO HCC

AT/ACR

AT/MEL

AB/CO

QPS Power Converters Powering Interlock


subsystems ready

the as designed documentation

all systems ready for machine check out

defines the pre-requisites to start the tests in terms of conditions and availability of systems

describes in what conditions the system should be delivered to the next step

specifies the objective of the test and either describes each step or summarizes the test procedure

... although special meetings were devoted by the HC WG, many documents describing the individual system tests are missing


preparation : essential time savers

user friendly application suite tool to aid the interpretation of the

test results programmed procedures for battery

tests but also all the classical tools like the

logging system, the alarm system, the post-mortem system, the electronic logbook, etc.

optimization of the MTF configuration for hardware commissioning

exchange of data between the central database and the specialist owned databases

support from the Controls Group

support from the MTF and Database Team


day to day organization: the field

RH87

RE78

RE82

RE12RE88

Point 7

RR73

RR77

UJ 76

PM76

TZ76

RA83

UA83

UJ 83

UJ 84

PM85

PX84

PZ85

UX85

TX84

UL86

UA87

RA87 UJ 86

UJ 87

UW85

US85

UL84

TI 8UJ 88

PGC8

TJ 8

RR13

UJ 13

RT12

UJ 14

US15

TI 12

PM15

PX14

UX15

UL14

UJ 12

LSS4

UJ 17

UJ 18

UJ 16TI 18

RR17

PX16 PX15

USA15

UL16

LHCb

ATLAS

SPS Point 8

Point 1

minor control centres

field control room

arc and even matching section

inner triplets and odd matching section

warm magnets

~210 cold circuits 190 orbit corrector circuits~10 warm circuits

~210 cold circuits 190 orbit corrector circuits~10 warm circuits


day to day organization: the field control room

DFBs

fieldcontrolroom

UA 83 – Left of Point 8

PowerConverters

EnergyExtractionSystems

+ wi-fi coverage of the UA+ printer

+ social corner + water fountain

+ desks+ meeting table+ lockable closets

cryogenics powering tests interlocks post-mortem


day to day organization: the tunnel and the odd extremities

+ wi-fi coverage of the UJ and RR+ desk+ lockable closets

106 m

mobile wi-fi coverage of the tunnel

will it work with the magnet assemblies in place ?


the sequence of tests around an even pointPowering Tests


organisation

communicationcommunication

follow up ofdata entry

follow up ofdata entry

analysisanalysis

commissioningof warm

components

commissioningof warm

components


and cool down of sector B’


and cool down of sector B’


and cool down of sector A’


and cool down of sector A’

commissioning of sector B

commissioning of sector B

commissioning of sector A

commissioning of sector A

presence in the fieldpresence in the field

coordinationcoordination


day to day ... coordination : the human resources

x 6

x 9x 9

x 1x 1x 1x 1x 1

operationsupport to operationdata recordingreporting

Field Control Room

Sector A1 Engineer1 Technician

1 AB Operator

Sector B1 Engineer1 Technician

5 people during two shifts

Progress TrackingPlanningRe-schedulingMeetingsCrisis ManagementMTFDatabaseAnalysisInformation processingPublications

Analysis

Project Management

1 Project Leader1 Mr.Circuit & DPL1 Expert Data Analysis1 Database/Web Expert1 Administrative Assistant

5 people during one shift

hardware orientedremoval of FCRsmall repairmissing detailsaccess/balisage

Field Preparation

on request

LHC Safety Coordinationadvise on safety mattersenforce safety in the field


Job Descriptions


resource loading : Hardware Commissioning Coordination

team 1

team 2

team 1

team 2

team 1

team 2

team 1

team 2

team 3

team 3

team 3

team 3

1

2

3

4

[team

s]


personnel safety: assumptions

1. the safety systems (fire and smoke detectors, ODH system, evacuation sirens, emergency lighting, arrêts d’urgence, etc.) are operational

2. posts, signs, flashing lights are in place to announce the ongoing hardware commissioning activity in the tunnel sector or sub sector

3. the access system at the surface allows in only personnel involved in hardware commissioning

4. the panels at the entrance indicating tests in progress will not be installed on time

5. the tunnel and underground areas are ventilated in nominal conditions


personnel safety: the field control room

the issue of the danger related to electricity was discussed on two occasions at the HC WG meetings, but failed to obtain clear directives on the training level of the people present in the FCR; many followed an ad hoc course with dissatisfaction

other issues related to helium venting in the tunnel, fire, number of people present undeground, etc. are known to worry some who are expected to spend a significant amount of their time in the UAs, UJ and RRs.


safety: zonage & balisage

RH87

RE78

RE82

RE12RE88

Point 7

RR73

RR77

UJ 76

PM76

TZ76

RA83

UA83

UJ 83

UJ 84

PM85

PX84

PZ85

UX85

TX84

UL86

UA87

RA87 UJ 86

UJ 87

UW85

US85

UL84

TI 8UJ 88

PGC8

TJ 8

RR13

UJ 13

RT12

UJ 14

US15

TI 12

PM15

PX14

UX15

UL14

UJ 12

LSS4

UJ 17

UJ 18

UJ 16TI 18

RR17

PX16 PX15

USA15

UL16

LHCb

ATLAS

SPS Point 8

Point 1

+ riot barriers+ red & white plastic chains

HC Coordination delimits the areas where particular safety conditions must be enforced during the tests in conjunction with LHC Safety Coordination


safety : personnel underground

Front 1Front 2 Front 1 Front 2

Left Right

6.6 km

in the field on call2t + 2o 1e+1o cryogenics operation1e+1t cryogenics

instrumentation1x+2e+2t quench protection2x+2t 1x+1t+ ... machine interlocks1x+2t 2e+1t+ ... power converters2e+2t coordination1t 1e controls1o operation

4x+5e+10t+3o 1x+4e+2t+1o

o operatorx experte engineert technician

22 present in the field

Front 1 arc and the matching section on the even side

Front 2 arc, the matching section on the odd side and the inner triplets on both sides

two shifts = 15 hour days resources deployed for the powering tests


safety : access during hardware commissioning 1/2

assumptions 1. the sectorization (into access sectors) of the machine is not possible

2. installation is ongoing in other sectors3. the access system will not be operational

states 1. access is not allowed2. access is regulated

• the activity is stopped during an access - if required ...

• for the passage of a convoy, the equipment for the tests is folded so that it is not in the way

regulatedregulated


safety: access during hardware commissioning 2/2

activityaccess

restrictionsarea concerned

first cool down of a sector from room temperature to 80K

no access whole sector

electrical quality assurance (hi-pot)

regulated arc, subsector

short circuits tests no access subsector (RA, RR, UJ)

powering tests1 no access whole sector

instrument checks regulated arc, subsector

equipment checks regulated arc, subsector

validation of systems regulated arc, subsector

(1) above minimum current as during normal operation of the collider


decisions

the commissioning must

be carried-out in safety for the personnel and the hardware,

be according to the approved procedures,

be carried-out in the time expected,

be traceable,

remain within budget.

schedulingimpact on commissioning time

impact on other systems budget, resources

go/no-gorules personnel safety, procedures, traceability

equipment integritycircuit integrity

HCC

HCC + HC Teamowners

HCC

operation & organisationimpact on commissioning time

HCC


communication

electronic media web pages progress reports (logbook, MTF, etc.) e-mails for major milestones/events

regular meetings daily operations meeting in the FCR where every system is

represented including the experiments via LEA weekly reports to GLMs, Team Meeting weekly meetings of HC monthly TCC

extraordinary meetings in case of crisis with concerned Groups, the Project and Department Leaders


failure scenarios

following the suggestion of the MAC in December

2003 and the endorsement of MARIC, failure scenarios during the commissioning - both for Individual System Tests and Hardware Commissioning - were studied

the objective was to define strategies for at least a few case studies per major system in order to gain in reactivity at the time of failures

evaluate the impact on the program and

suggest alternatives to carry-on

six sessions of the HCWG were devoted to this study:

Superconducting Electrical Circuits Cooling and Ventilation System Cryogenic System Vacuum System & Leak Tests Electrical Power Distribution Power Interlock Controllers

can it becircumvented

do itaccordingto definedprocedures

cleardiagnostic

applydiagnosticprotocols

decisioncrash

programme

failure

yes

yes

decision

applydiagnosticprotocols


failure scenarios

Superconducting Electrical Circuits earth fault diagnostic short circuit configuration demineralised water circuit air cooling of the 600 A cables

in the ducts loss of quench detection

instrumentation bad splices start-up with missing circuits

Cooling and Ventilation System redundancy of components blowing of water cooled cables

Cryogenic System redundancy of cooling through

the five feedpoints higher heat loads redundant instrumentation but

no redundant cabling contamination of the helium

inventory helium loss in the tunnel via

safety valves impact of a utility (water,

electricity) failure

Vacuum System & Leak Tests overhead for leaks detected

after the pressure test Electrical Power Distribution

over-voltages (40 to 50% amplitude) for some tens of ms

short spikes Power Interlock Controllers

electro-magnetic compatibility the download of a wrong

configuration file to a controller or a software bug

swapping cables failures at the level of SCADA

failure scenario document in preparation

*

*

*

*


impact of a fault requiring a repair

any failure requiring a repair of a cold part of the collider takes between 10 and 35 days [1]

qualification tests (e.g. elqa) will have to be repeated

commissioning program will have to be rerun: how much depends on where we were when the fault was encountered

during most of this time, the commissioning of the warm systems can continue

5 w

eeks

warm

-up

repair

cool-

dow

n

qu

alif

y

between 15 and 40 days out of ~72

[1] LHC-PM-ES-0002 General Parameters for Equipment installed in the LHC, April 1999


consequence of a failure


... and then there is an exception !

a core team made of a few specialists/experts will meet on short notice to study the problem recommend additional

diagnostics take a decision

go no-go & crisis meeting with

Project and Department Management

something the specialists or the hardware commissioning team do not understand.


can it be circumvented/can it be accepted as is ?

what is the impact of a non-conformity on the performance of

the equipment ? the system ? the machine ?

what is the budget ? have we reached the limit of the budget ?

AB-ABP

LHCOP

Equipment Groups

Hardware Commissioning Core Team


my worries

coordination missing documentation transport & logistics access restrictions not respected safety of tunnel, of field control room, of the other

areas availability of components

major components is not the problem of HC sub-components like instrumentation is

operation thefts noisy environment system testing infrastructure

hardware (e.g. field busses) software (e.g. post-mortem system)

applications utilities (e.g. e-logbook)

LSS8L tests not given the importance it deserves accelerator component failure during the

commissioning

Review of Hardware Commissioning May 11-13, 2005 Hardware Commissioning Co-ordination Organisation...

Documents

Transcript of Review of Hardware Commissioning May 11-13, 2005 Hardware Commissioning Co-ordination Organisation...