G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 1 ECAL Detector Control System G....

4.11.2004ECAL Cooling Review - DCS

1G. DissertoriETH Zürich

ECAL Detector Control System

G. Dissertori ECAL Cooling Review, Nov 4, 2004

Status report

Trigger/DAQ project DCS

CMS DCS

ECAL DCS



Outline

Overview

Precision Temperature Monitoring (PTM)

ECAL Safety System (ESS)

PVSS Monitoring (and related issues)

Note : For details on the PTM and ESS hardware

(sensors, readout) I refer to the DCS ESR, May 04!



Overview

ECAL moduleECAL module

crystalcrystal

PCBs for VFE and FEPCBs for VFE and FE

coolingcooling

LV, HV LV, HV

Laser systemLaser system

Offline, DAQOffline, DAQ

Data BasesData Bases

PVSSPVSS

prec. temp. sensor

Monitoring

ESS temp. sensor + WLDESS temp. sensor + WLD

MonitoringMonitoring Safety, PLCSafety, PLCinterlockinterlock

cooling, faults

humidity sensorhumidity sensor



PTM

Precision Temperature Monitoring - PTM

S. Zelepoukine, IHEP Protvino and ETHZ



PTM

Requirements:

very high precision temp. monitoring : check stability of water cooling system, to stay within 0.02 - 0.05 deg C.

no hardwired feedback to cooling

relative precision : 0.01 deg C

one sensor/module on thermal screen and grid main water IN, main water OUT



Patch panel

Patch panel

Thermal screen

Grid

Sensor probes location inside SM:

Probes of type 1 (N = 4 + 4) Probes of type 2(N = 2)

Number of probes (per SM): 4 + 4 + 2 = 10Each probe has its own signal cable – twisted pair

M1 M2 M3 M4

PTM : Sensor location



PTM : Sensor location (SM0)

M3

probe M3

cable from M3 probe

cables from M1 and M2 enter M3 via different windows

Patch panellocation



55 mm

St.steel;outer / inner dia = 4 / 3 mm

by F.Mossiere

PTM : immersion probes



Location of immersion probes

Turned out during electronicsintegration for SM10:

Location/mounting of these probesis critical, in the sense that operationson the PP can lead to damaging them.

Alternatives?



SM – top viewPTM

immersion probes

Cooling manifolds

Current probelocation is provenfor probe to beeasily damagedwhile people areworking at the patch panel…

PP

Better locationwould be:

Now…

Location of immersion probes



PTM/HM:intermediateconnection

SM – top view

PPPP

SM10 in H4:2 sensors from 14 (1 PTM and 1 HM )are lost due to the poor intermediate connection quality…

SM production (from now on):All PTM/HM sensors areto be installed in B27without an intermediateconnection –sensor cables aredirectly connectedto the PP connector.

SM production (from now on):All PTM/HM sensors areto be installed in B27without an intermediateconnection –sensor cables aredirectly connectedto the PP connector.

Intermediate Connections



PTM : Readout - Layout

ELMBELMB

Cur.srcCur.srcPC / PVSSPC / PVSS

ECALUXC55 balcony:

Counting room

20x2 STP cables

CANbus cables

~ 500 sensors(NTC thermistor Betatherm;100K @ 25C; indiv.calibrated;better than 0.01C rel.accuracy)

Readout electronics:ELMB based;ELMB reads voltages andconverts to digital data, thensends data via CANbus.DC-DC converters

DC Power supplies

DC Power supplies

Monitoring application:PVSS based;data storage/archiving,visualization/trending,warnings/alarms.

Network access

Galvanicisolation



Recent Results

First results from SM10 at H4:

Sensors, sensor probes, read-out practically final

Reading 4 sensors on grid, 3 sensors on thermal screen, and Water IN/OUT

One sensor lost, most likely due to bad intermediate connection inside SM before PP

Since about one week : installed 2 sensors on backplate Above M1 and M4



Cooling Stability

Zoom



M4

M3, M2

M1

Grid

Thermal screen

Water - IN

0.01C

0.01C

0.01CM1

M4, M3

Cooling stability = 0.01C (incl. day/night oscillations)

Zoom :



Just to showuncompressed(in time)data…

10 min

Grid

Thermal screen

Water - IN

AmbientDiv = 1.0C

Div = 0.01C

Div = 0.01C

Div = 0.01C



M2M2

M1, M3M1, M3

M4M4

M1 M2 M3 M4

28.5 30.6 28.2 25.7

M1 M2 M3 M4

28.5 30.6 28.2 25.7

Inside the electronics compartmentFrom ESS, see later…



Grid side

Water – out (blue)

Th.screen sideWater – in (red)

Grid side(zoom)

Th.screen side(zoom)

Water – in (black)

Ambient

4x sensors are locatedon the SM10 external surfaces:

On Backplate (Grid side): M4, M1Thermal screen side: M4, M1

4x sensors are locatedon the SM10 external surfaces:

On Backplate (Grid side): M4, M1Thermal screen side: M4, M1

Th.screen side – roughly equal to ambient (18.2C);

Grid side – 23.4C (and much heat is dissipated bythe massive metallic support!).

Th.screen side – roughly equal to ambient (18.2C);

Grid side – 23.4C (and much heat is dissipated bythe massive metallic support!).



Just to showuncompressed(in time)data…

Grid side

Water – out (blue)

Water – in (red)Th.screen side

Grid side(metallic surface)

Th.screen side(plastic surface)

Water – in (black)

ambient

Div = 1.0C

Div = 1.0C

Div = 0.2C

Div = 0.2C



18.01 C

23.4 C

18.01 C

Air conditioner ~18 C

SM10 in H4

18.2 C heatheat

~30 C

Heat flow out of ECAL?



heatheat

heat heat

heat

heat heat

heat

??

??

Heat flow out of ECAL?



Conclusions on PTM

Cooling stability: Achieved 0.01 C in the long-term, including day/night variations Have a monitoring application, Serguei is working on a more “final”

version…

Temperatures on outer surfaces Recent measurements performed by Serguei Is heating towards outside (or inside, depending on orientation) an issue?

For other detectors? For other SMs ?

As next step, Serguei will move external sensors to lateral surfaces…

Hardware Resolve issue of immersion probes Intermediate connections will not be there any more



ESS

ECAL Safety System - ESSformerly : TSS (Temperature Safety System)

Belgrade group, ETHZ

See also CMS NOTE 2004-013



• Full system autonomy in all aspects;

• Independent, continuous temp. monitoring of the ECAL VFE + FE environment in

both ECAL SM + EE; Precision : 0.5 deg C

• Archiving of temp. data and system status information for analysis of system and

detector performance;

• Reliable hardwired interlocks with ECAL HV and LV Power Supply systems;

• External Alarm Interfaces with ECAL Cooling system, as well as interface with

general CMS DSS;

• Also with the newly proposed Water leakage detection system!

• Prompt reaction on any external alarm or critical change of temperature inside the

ECAL by issuing, in a proper time sequence, Warnings and Alarms to:

1. HV System Crates (hardwired interlocks),

2. LV System Crates (hardwired interlocks),

3. System Operator (soft PVSS Warnings and Alarms);

• Maximum possible level of robustness, reliability, safety and maintainability;

ESS : Requirements



Three interconnected system layers:

• Temperature conversion and channel multiplexing - ESS FE LayerESS FE Layer,• Data acquisition, data processing and interlock generating - ESS PLC LayerESS PLC Layer,• System monitoring and system control - ESS Soft LayerESS Soft Layer;

Schematic layout

RS485

RS485

?



There are in total 288+64 NTC 470 Ohms 288+64 NTC 470 Ohms thermistors (EPCOS)positioned in pairs at each measurement point ( “Twin” sensors ).

2x twisted pair cableSCEM = 04.21.51.704.4SCEM = 04.21.51.704.4, 2x2x0.05mm2,outside diameter 2.9mm, PETP insulationPETP insulation

ESS : Sensor Location

40-90m long 9xSTP cable sectionsSCEM = 04.21.51.405.2SCEM = 04.21.51.405.2 – 9x2x0.008 mm2outside diameter 7.2 mm, MA 18 P



We got 2 complete setups

needed for a redundant system

Siemens S7- 400 Family PLC System for ESS:

ESS : PLC



Principle Controls Actions

See also Xavier’s alarm/action matrix…

Critical Reading : Temperatures (TSS) Software warning. If persistent: Switch OFF relevant HV/LV power supplies via ECAL-PLC

Critical Reading : Water Leak Software warning. If persistent: Switch OFF relevant HV/LV power supplies via ECAL-PLC Message to Cooling PLC : close SM n safety lines

Critical Reading : Sniffers (on CMS-DSS PLC) Switch OFF power on ECAL racks (to be implemented by CMS-DSS) ? See also Ch. Schaefer’s talk…




DSS Watch Dog : ECAL-PLC not running Switch OFF power of the ECAL racks Note : ECAL-PLC checks that Cooling-PLC is running!

Critical Reading on General Cooling PLC (TS) : Message to the CMS-DSS PLC and the Cooling PLC Actions (?) : Stop water, switch OFF power of ECAL racks

Critical Reading : DCU temperatures (on PCBs/Capsules) Software warning. If persistent? I would say : no automatic action

Critical Reading : PTM temperatures Software warning. I would say : no automatic action




In general, for ESS - DSS interactions:

ESS should take principle actions

Only when DSS sees that ESS is “dead”, DSS should take action!

Exceptions : Sniffers (go directly to DSS) Others?



Experience up to now

Excellent performance of ESS

Sent relevant interlocks during SM0 tests

No problem during SM10 running up to now

See ESR May04 review talk for details on SM0



PVSS Monitoring and Control

S. Zelepoukine, P. Milenovic, R. Ofierzynski, F. Beaudette, A. Lister, R. Gomez-ReinoAlso : Thanks to S. Schmeling / JCOP



Standard Controls

Continuous Monitoring (display/storage via ECAL-PVSS)

Temperatures from ESS Temperatures from PTM Temperatures from DCUs (not implemented yet at H4) Water Leakage Detection System (via ESS) Humidity Cooling System (temperatures, flow, pressures) HV and LV Status of Laser System



PVSS at H4 : Prototype for ECAL

First complete set-up of PVSS DCS system At H4, includes: PTM - HM - ESS - LV - HV - Cooling - Laser + Supervisory system Every application implements a Finite State Machine Based on PVSS II, v.3, and the JCOP framework

Distributed system (Windows XP) on 3 PCs : 1 : Supervisor, Laser, Cooling 2 : HV, LV 3 : PTM, HM, ESS

Should be similar at P5 I see no reason to have dedicated PC just for Cooling

Monitoring



PVSS at H4 : Data Logging

For the moment, all data go to the PVSS archive (proprietary)

Not easy to get data out (apart trending, which is ok) not foreseen as final solution (!)

Done because of pressure to get the FSM at H4 etc running

On the long term : ORACLE

Some first tests last year with HV, but did not learn too much

A new expert is joining the efforts (Rick Egeland, Minnesota) He will try to help us setting up the commun. with ORACLE and to

develop scripts to easily retrieve data

We will not write all DCS data to conditions DB at high frequency. Rather at low freq. ( 1/ (1 - 10) min ?) or when major changes occur

Could write all DCS data to some dedicated DCS DB, which is archived at some stage (after a couple of years?). TBD

But not more to say for the moment…..



Supervisory System : 1



Cooling Application

See alsoXavier Pons’talk:

For ways ofchanging limits,OPC connection,etc.

Note:This applicationonly monitors

user can notoperate the cooling system from here !



ESS Application



Conclusions

Hardware PTM : performs according to specs Able to monitor temperature stability to 0.01 C To be resolved : issue of immersion probe location

ESS: Have purchased redundant Siemens PLC system All tests up to now very satisfactory To be finalized :

Water leak detection Hardware links with Cooling Exact protocol of further software communication with Cooling



Conclusions

Software

Have developed first (in CMS) complete distributed monitoring system based on PVSS

Was rather difficult and painful endeavour, have learned important lessons, also for other subdetectors

To be implemented : Monitoring of PCB/capsule temperatures from DCUs

Monitoring of cooling : ONLY monitoring, no action/operation by shifter

Have to identify man-power for the development of the P5 set-up, based on what we developed for H4

G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 1 ECAL Detector Control System G....

Documents

Transcript of G. Dissertori ETH Zürich 4.11.2004 ECAL Cooling Review - DCS 1 ECAL Detector Control System G....