A New Vision for the Evolution of Controls J. Patrick, C. Briegel, B. Hendricks APT Seminar February...

A New Vision for the Evolution of Controls

J. Patrick, C. Briegel, B. Hendricks

APT Seminar

February 21, 2008

February 21, 2008 J. Patrick 2

Motivation• Recent Accelerator Initiatives

– HINS– NML– Accelerator NUMI Upgrade (current complex without Tevatron)– More recently Project X

• Bottom-up decisions made on control system choice:– Needed a specific feature or support for specific hardware– Support external collaboration– No explicit vision for an accelerator complex-wide controls beyond 2010

• Ended up with a myriad of different controls: – NUMI Upgrade - ACNET– HINS; NML: EPICS; EPICS/DOOCS + ACNET– A0: DOOCS + ACNET– Project X: ?

• How do we integrate and support these to meet the needs of one large operating complex?


One Control System

• Project X:– Substantial new facility that would be built here– Catalyst for thinking about the evolution of the accelerator

complex-wide Control System– Step back and understand WHAT a control system should do,

rather than how it should be done.

• Independent of Project X– NUMI upgrade supposed to operate beyond 2018

• Demanding operational requirements

– For maintainability, older hardware must be replaced• System released from those constraints

– Key architects may retire, maintenance becomes difficult and upgrades become attractive


Bottom Line

Fermilab is the premier HEP Accelerator Laboratory in the nation, our control system needs to evolve, and we have taken the first step to understand what is needed.


Requirements• What is the functionality of a control system? • Separate from how it will be done (which is the Design phase).• We did not decide on or even discuss specific technology choices

(EPICS, DOOCS, ACNET, or specific hardware, etc.)

• Requirements are:– Agreement between users and developers on needed functionality– If a requirement can not be met, it needs to be revisited and modified– A controlled process– An objective way to evaluate design and technology choices

• A chosen implementation has to meet the requirements.– A standard in industry– A proven way to build in quality upfront.

Software contributes 30% of the failure rate. [see High Availability: A Design Primer]


Process

• Series of discussions among Controls, Operations, HINS + Computing Division people

• Each section written by subgroup of experts• Coordinated by and final document edited by

Suzanne Gysin (CD)• Needs broader input

– From machine experts and operations– Hardware experts

• Not considered final, will evolve with more input


Controls Requirements Document

• Contents– 1 Introduction (J. Patrick, S. Gysin)– 2 Base Requirements (J. Patrick, S. Gysin)– 3 Low-Level Systems (R. Neswold)– 4 Central Services (A. Petrov)– 5 Application Infrastructure (A. Petrov)– 6 High Level Applications (B. Hendricks)– 7 Controls In A Box (S.Gysin)– 8 Beam-Based Feedback (J. Patrick)– 9 Machine Protection System (B. Hendricks)– 10 Software Development Environment (C. Schumann)– 11 Hardware/Operating Systems (D. Finstrom)– 12 Networks (D. Stenman)

• Document is available: https://beamdocs.fnal.gov/AD-private/DocDB/ShowDocument?docid=2934

• Internal Review (January), i.e. we agree that this is what is needed in an evolved control system.

TCP/IP communication services


CENTRAL OPERATORCONSOLES

LOCAL OPERATORCONSOLES

FIXEDDISPLAYS

GIG

AB

IT

ET

HE

RN

ET

T

EC

HN

ICA

L N

ET

WO

RK

FILE SERVERSAPPLICATION SERVERS SCADA SERVERS

VME Front Ends Front Ends

PLCs

MACHINE


PUBLIC ETHERNET NETWORK

TIMING GENERATION

MACHINE

ACTUATORS AND SENSORSCRYOGENICS, VACUUM, ETC…

QUENCH PROTECTION AGENTS,POWER SUPPLIES, …

BEAM POSITION MONITORS,BEAM LOSS MONITORS,BEAM INTERLOCKS,RF SYSTEMS, ETC…

Wor

ldF

IP S

EG

ME

NT

(1,

2.5

MB

its/s

ec)

PLCs

PR

OF

IBU

S

FIP

/IO

OP

TIC

AL

FIB

ER

S

Front Ends

ANALOGSIGNALSYSTEM

T T T T

TT

TT

TT

LHC controls architecture diagram

Hardware Tier

Application Tier

Service Tier


Challenges

• Support large multi-accelerator complex• Many simultaneous modes of operation

– Automation, ease of configuration

• Large number of hardware devices– Configuration management, software support– Easy access to information

• Many simultaneous users– Scalable data acquisition

• Application development• Reliability - robust against single component failure• Better stability as beam power increases

– Diagnostics, Fast Feedback, Data Logging


Today’s Presentation

• Present the highlights in each section.

• Invite user feedback:– To give feedback, please send us email

([email protected], [email protected])


Document Highlights

• Clock system with more information

• Correlated data

• Front end/service framework consolidation

• Device locking/ownership

• Self-describing mixed type data format

• Improved server-side application environment


Document Highlights

• Information at your fingertips• Generic device search• Open applications with devices• Copy/paste data between applications• Controls in a box• Infrastructure for beam-based feedback• Bug tracking system• Supported graphical development environment• Development network

February 21, 2008 C. Briegel 13

Timing System

XClock

1. 1.2 usecs per event

2. 16 bit event

3. 32 bit event counter

4. Self-describing payload

5. 1 GHz phased to TClock


Front-End

Common Framework Available at all Tiers

1. Common Software Development

2. Memory Protection

3. Maintain Data Connection

4. Device/System Ownership

5. Acquire Data via Multicast

6. Debugging, Post-mortem


Front-EndData Acquisition

1. Client Specification1. One-shot

2. Repetitive

3. Event+Delay

4. Event Sequence Number - Correlated Data

5. On-change+Periodic

6. Array on Event

7. Array on Alarm

8. Real-time (1 KHz)


Front-End

Data Acquisition

2. Returned Data1. Raw or Scaled

2. Status

3. Time Stamp

4. Event Sequence Number

5. Self-Describing


Front-End

Setting

1. Rapid

2. Forwarded

3. Archived

4. Atomic Set/Read


Front-End

Alarms

1. Events

2. Exceptions

3. Default Scan

4. Extensible Scan

5. Bypass Devices or Groups of Devices


Network

1. General

DMZ/Firewall

2. Operational

3. Development/Test

Network

February 21, 2008 B. Hendricks 21

Server-Side Application Environment

• Server-side components run continuously, on a schedule, or on-demand

• Run on machines maintained by Controls Dept• Server-side components in a control system:

– Central Services(e.g., data acquisition, data logging)

– Server-side parts of multi-tier applications(e.g., Autotune)

– Web applications(e.g., Supertable, Synoptic Display)



• Run unattended– Require an automated tool to start, stop, and monitor the

operation

• Do not have user interface– Require a means to report the status to users (logging)

• Multi-component– Components (Data Acquisition Engine, OAC, servlets) are

dynamically linked to each other

• Need a server-side application environment to perform these common tasks



• Several separate “environments”:– Data Acquisition Engine with OACs and data loggers– Tomcat for servlets– Custom shell scripts

• Limited number of supported component types– It's hard to maintain custom components other than

OACs, data loggers, or servlets.

• Complicated development and deployment process



• Use a standard application server for server-side components (including shell scripts)

• Customize the server for our needs• Provide common services, such as

authentication • Facilitate development and deployment for

server-side components• Eventually have a cluster of application servers

for redundancy and load balancing


Information at Your Fingertips

• Information is available but scattered through applications

• Inconsistent availability of information

• Inconsistent means for accessing information



• Need a generic solution

• Should be provided by the application framework

• Simple click and menu system

• Context sensitive menus– Device names– Node names– Error strings



• Device names– Device database attribute information– Device presence in standardized lists

• Datalogger, alarm, save lists• Save files• Saved plot setups• Families of devices

– Send device to another application (including plots)– Find related devices– View alarm and setting history



• Node names– Keeper name(s)– Other static attributes of the node– Devices belonging to the node– Current status of the node

• Error strings– Detailed description of the error


Bug Tracking System

• Provides communication path

• Provides persistence of information

• Maintains history

• Provides accountability

• Requires commitment of the user community

• Provides a mechanism for getting issues resolved


Bug Tracking System

• Fix List– Pros

• Web based• Easy to make an entry• Can add images• Easy to link to our electronic logs

– Cons• Passive system• No assignment of responsibility• Problems can remain unresolved• No standard removal mechanism when resolved


Bug Tracking System

• Remedy (Help Desk)– Pros

• Entry method is web based• Active system with reminders• Supports assignment of responsibility

– Cons• More complicated issue entry• Removal method is not straight forward• Doesn’t support attaching images• Limits who can be assigned to resolve an issue• Not managed by us


Bug Tracking System

• What we want in a new system– Open source or commercial product– Web based interface– Simple issue entry and removal– Ability to assign responsible parties– Notification upon entry, update, or removal– Ability to attach images– Easy linkage to our electronic logs– Managed by the Controls Department– Usable by other entities in the division/lab


Controls in a Box

• Control system provides many services

• Large footprint with expensive commercial software

• What about a local or remote system developer?

• What about small testing facilities or experiments?

• Need to be able to scale down


Controls in a Box

• Must be able to stand on its own

• Simple to download and install

• Easy configuration tool

• Shall not use expensive commercial software


Controls in a Box

• Shall provide a local clock (hardware or software)

• Shall provide for local data storage

• Shall include necessary central services including database

• Shall include basic applications for reading and setting data

• Self-contained


Next Steps

• Solicit more input– Meet with broader user base

• Select what ideas to implement according to the following criteria:– Benefit the current program, NUMI Upgrade,

and Project X– Incorporating user input– Prevent a divergence in Controls


Summary

• Realistic vision of a control system beyond 2010• Presented specific ideas and the selection

criteria to start implementing• Independent of Project X, Fermilab is the

premier Accelerator Laboratory in the nation, our control system needs to evolve

• Ready to provide an organized and systematic evolution of the control system for the FNAL accelerator complex

A New Vision for the Evolution of Controls J. Patrick, C. Briegel, B. Hendricks APT Seminar February...

Documents

Transcript of A New Vision for the Evolution of Controls J. Patrick, C. Briegel, B. Hendricks APT Seminar February...