1

ALICE Control SystemALICE Control Systemready for LHC operationready for LHC operation

ICALEPCS 16 Oct 2007L.Jirdén

On behalf of the ALICE Controls Team CERN Geneva

2

OutlineOutline About ALICE Tools & components Typical applications Data flow User Interface Integration, installation &

commissioning Conclusion

New approaches or trends in experiment controls

New !

3

IntroductionIntroductionATLAS

ALICECMS

LHCbcenter of Geneva

ALICE experiment 18 sub-detectors, 2 magnets 1000 members, 90 institutes, 30 countries Being installed & commissioned

Detectors not shown

FMD , V0, T0, ZDC,

EMC, CPV, ACO

HMPID

TOF

TRD

PM

DSPD

SDD

SSD

TPC

MUON - SPECTROMETER

PHOS ALICE Controls

Started 6 years ago Small central team collaborating with

Detector groups & LHC experiments (JCOP) In total ~100 people involved

4

ALICE ALICE environmentenvironment

~ 55m

~ 50m

BEAMBEAM

HLT

Control

DAQ

racks

racks

detectors

ACR

DetectorsStrong radiation

strong magnetic field

Experimental cavern Closed zone at runs

radiationmagnetic field

Counting roomscomputer farms

databases

ALICE Control Roomoperator consoles

5

18 detectors

~150 sub-systems

Controls context

SPD PHO FMD T0 V0 PMD MTI MTK ZDC ACOSDD SSD TPC TRD TOF HMP

Controls Back-end

ServicesGas

Electricity

Cooling

Magnets

Access

Infrastructure

B-field

Space Frame

Environment

Radiation

Beam Pipe

On-line

Trigger

DAQ

HLT

ECS

SafetyCSAMDSSExternal

SystemsAccelerator

Offline

6

ALICE FRAMEWORK

USER APPLICATIONS

Tools and components

New approach!

FRAMEWORK

SCADA SYSTEM PVSSII

• Modular, distributed,• equipment oriented• error handling• file & Dbase access• graphics user interface

TOOLS

ALARM

CONFIGURAT

ACCESS

ARCHIVAL

TRENDINGModeling operational behavior

Integration of commonly used devices

Standard communication & hardware access

CONNECTION: OPC, DIM, DIP

DEVICE COMPONENTS

FINITE STATE MACHINE TOOLS

7

ALICE DetectorsTime Projection Chamber

TPC

8

TPC sub-systems to TPC sub-systems to controlcontrol

TPC: 15 sub-systems ALICE: 150 sub-systems

Very High Voltage (100 kV)High Voltage (3 kV)

Low Voltage(60 kW)

Front-EndElectronics

Cooling

0.1 oC stability

Laser

Cameras

Mirrors

Gas (88 m3) Jan 07: TPC being lowered

down into the cavern

510 cm

9

counting room

cavern

ALICE Control Room

Systems layoutoperator

LV Devices

Worker Node

LV OPC

PVSS

Ethernet

worker node

DIM

PVSSworker node

DIM

PVSSWorker Node

DIM

PVSS

Ethernet

DETECTOR

Worker Node

HV OPC

PVSS

HV DevicesCANbus

FEE Devices

Operator Node

PVSS

Ethernet

D

D

DD

Ethernet massively used as field-bus: - 1200 devices on private LAN - devices with Single Board

Computers - large bandwidth required

New !

10

Worker NodeWorker Node

DIM

PVSS

TPC Front-End ElectronicsTPC Readout Partition

1 MB/sethernet

OFF-DETECTOR

Data busI2C bus

Board ControlFPGA

MEMORY

Comm. ControlLinux SBC

Bus Control

RCU

DETECTOR

ON-DETECTOR

FECVLSIVLSI

FECFPGAVLSI

128 ch

FECVLSIVLSI

FECFPGAVLSI

128 ch

FECVLSIVLSI

FECFPGAVLSI

128 ch

FPGA

Linux SBC

4500 Front End Cards0.6 million RO channels

Front End Cards

IROC

OROC

RO1

RO2

- Much control software located on detectors - Previously in VME now in custom systems - Harsh environment & no access - redundancy – Control work done by electronics engineers

New !

11

Operations layoutOperations layout

Control Units

DeviceUnits

Physicaldevices

TPC

FEELVOROCIROC

HV

DET

CH CH CH CH CH CH CH CH CH RCU RCU RCU

CH CH

…

HV Devices FEE DevicesLV Devices

RO …36xOFF

ERROR

ON

STANDBY

GO_STANDBY

GO_READY

GO_STANDBY

RESET

Alarm

GO_OFFCONFIG

RAMP_UP RAMP_DW

12

Operating Operating principleprinciple

CH

TPC

FEELV

RO

OROCIROC

HV

DET

CH CH CH CH CH CH CH CH RCU RCU RCU CH

operations plane

HV OPCPVSS

D

LV OPCPVSS

D

DIMPVSS

D

DIMPVSS

D

DIMPVSS

D

PVSSD

FEE DevicesLV DevicesHV Devices

systems plane

operator

• modular & flexible

• operational aspects considered early & ‘built-in’

• easy partitioning

commandsstatus, alarms

B

A

operator?

13

Operating Operating principleprinciple

CH

TPC

FEELV

RO

OROCIROC

HV

DET

CH CH CH CH CH CH CH CH RCU RCU RCU CH

operation plane

HV OPCPVSS

D

LV OPCPVSS

D

DIMPVSS

D

DIMPVSS

D

DIMPVSS

D

PVSSD

FEE DevicesLV DevicesHV Devices

systems plane

operator

commandsstatus, alarms

operator B

A

New approach!

• modular & flexible

• operational aspects considered early & ‘built-in’

• easy partitioning

14

Conditions Physics Data

offlineoffline

From Detectors

To GridLHC LHC

acceleratoraccelerator

DAQ & DAQ & TriggerTrigger

TechnicalTechnical services services

Main data flowsMain data flows

Archival database

Configuration database

operator

syn

chro

niz

atio

n

FEEFEEFEE

6 Gbyte

archival “on change”

100 000 change/s

processing

Control SystemControl SystemBack-endBack-end

800 Single Board computers ~108 registers in FPGA’s, VLSI’s

new trend !

Large amounts of data - more control channels - configurable electronics

15

ALICE User ALICE User InterfaceInterface

Why needed? Small shift crew, changing often, non-experts Operating large set of sub-detectors GUI’s designed by each sub-detector

Aim Same ‘Look and Feel’ Provide common utilities

Sub-sys status

General utilities

Monitoringzone

Login

Messenger

Controlzone

Global Control

ALICE UI Fixed screen layout System browser General utilities

16

LOGIN PRINTHELPALARMS

MESSENGER

PHONE

DETECTOR STATUSHIERARCHICAL BROWSER

ALICE CONTROLS USER INTERFACE

Select HMPID

SUB-SYSTEMS

CONTROLS TOP-NODE PAGE

17

ON

operator

TPC

Systems integrationSystems integration

central Operator Node

18 detectors, 150 applications

TRD TOF

100 PVSS systems:- a large large distributed system- of distributed systems

LV Devices

Worker Node

LV OPC

PVSS

Ethernet

worker node

DIM

PVSSworker node

DIM

PVSSWorker Node

DIM

PVSS

Ethernet

DETECTOR

Worker Node

HV OPC

PVSS

HV DevicesCANbus

FEE Devices

Operator Node

PVSS

Ethernet

D

D

DD

18

ALICE ONLINE

DAQ TRG HLT

ECS

Operations Operations integrationintegration

Detectors, infrastructure, services

LHC

operator

DCS

FSM top node

TPC

FEELVOROCIROC

HV

DET

CH CH CH CH CH CH CH CH CH RCU RCU RCU

CH CH

…

HV Devices FEE DevicesLV Devices

RO …36x

FSM is an excellent tool for integration !

TPC

19

Status todayStatus today Installation

Controls Back-end is operational 150 computers Dedicated network for 1400

devices User applications 90% ready, 50% installed

150 detector applications Originating from 20 groups in 30 countries Conformity and quality control

Commissioning & operation 13 of 18 detectors are installed being

commissioned 1st Cosmic run in December 2007 1st physics run in May 2008

20

ConclusionConclusion

ALICE Controls System is a new generation system incorporates new innovative approaches is being installed and is successfully used to

commission the sub-detectors fulfills all requirements will be ready for operation with LHC in 2008

New !

21

PostersPosters WPPB32 Cyber-security in ALICE

RPPB21 Finite State Machines for Integration and Control in ALICE

TPPA22 Standard Device Control via PVSS Object Libraries

RPPA36 Handling Large Amounts of Data in ALICE

New !

New !

New !

New !

22

BackupBackup

ALICE UI Fixed screen layout System browser General utilities

23

ConclusionConclusion

ALICE Controls is a new generation system Incorporates new innovative approaches

SCADA with common Framework Massive use of Ethernet Important “On-detector” controls Handles large amounts of data Operation based on Finite State Machines

Presently used for sub-detector commissioning

Fulfills all requirements Will be ready for operation with LHC in 2008

New !

24

Control & Device Control & Device unitsunits

PVSS

CU Behaviour(FSM)

configurationarchiving

PartitioningAlarmhandling

access, ownership

dB

operator

parent

child child

commands

states

commands

states

PVSS

FSMinterface

configurationarchiving

Alarmsdevicedriver

access, ownership

dB

operator

parent

settingsreading

s device

Partitioning

OFF

ERROR

ON

STANDBY

GO_STANDBY

GO_READY

GO_STANDBY

RESET

Alarm

GO_OFFCONFIG

RAMP_UP RAMP_DW

25

Detector Control

Back-End Systems

Tools, components

Networks

Computers

On-line

Services

Infrastructure

Integration

External Systems

Accelerator

Offline

18 detectors

~150 sub-systems

MAG

GAS

COO

ELE

ACC

CSADSSLHC

OFF

SafetyCSAMDSS

SPD PHO FMD T0 V0 PMD MTR MTK ZDC EMCSDD SSD TPC TRD TOF HMP ACOCPVTOFSPD SDD SSD TPC TRD FMD MTRPHOHMP CPV ZDCT0 MTKPMDV0 ACOEMC

Trigger

DAQ

HLT

ECS

Gas

Electricity

Cooling

Magnets

Access

B-field

Space Frame

Environment

Radiation

Beam Pipe

ENV

BFI

BPI

SFM

RAD

HLT

DAQ

TRG

ECS

DIP

DIP

DIP

DCS FSM top node

Data Interchange Protocol

26

New approaches in New approaches in experiment controlsexperiment controls

System based on SCADA and framework Limited resources Share developments between LHC experiments

Ethernet used as field bus devices offer Ethernet interface increasingly Becoming robust and less expensive Large bandwidth needed

Important dataflow Increasing number of control channels Configurable electronics

Much controls located on the detector Custom chips, single board computers

Operation based on Finite State Machine modeling Operational needs prepared in the basic system

27

Front-End Front-End ElectronicsElectronics

ALICE CONTROL ROOMPVSS

console

Common interface?Common interface?

18 different FEE architectures

ON DETECTOR

VME

SPD FEE

JTAG

HMPID FEE

DDL

TPC FEE

PROFIBUS

TRD FEE

ETHERNET

…

MXI …

Front End Device (FED)

DIM server

interfaces

FEE H/W

MXIDDLPROFIBUSETHERNETEtc.

FEE specific

FEE commo

n

DIM client

28

Architecture of SPD Architecture of SPD FEE ControlFEE Control

(PVSS) DIM Client

CA1CA2 MA1 MA2

Pixel Multi Chip Module

Router

VISA

PCI-MXI-VME

Database

DIM server Data

, flags

Multithread applications

Control Agent reacts to

commands received by server

Monitoring Agent publishes data as

DIM services

Control and

Monitoring

Server

ClientSoftware

JTAGC

om

mands