A New Vision for the Evolution of Controls
J. Patrick, C. Briegel, B. Hendricks
APT Seminar
February 21, 2008
February 21, 2008 J. Patrick 2
Motivation• Recent Accelerator Initiatives
– HINS– NML– Accelerator NUMI Upgrade (current complex without Tevatron)– More recently Project X
• Bottom-up decisions made on control system choice:– Needed a specific feature or support for specific hardware– Support external collaboration– No explicit vision for an accelerator complex-wide controls beyond 2010
• Ended up with a myriad of different controls: – NUMI Upgrade - ACNET– HINS; NML: EPICS; EPICS/DOOCS + ACNET– A0: DOOCS + ACNET– Project X: ?
• How do we integrate and support these to meet the needs of one large operating complex?
February 21, 2008 J. Patrick 3
One Control System
• Project X:– Substantial new facility that would be built here– Catalyst for thinking about the evolution of the accelerator
complex-wide Control System– Step back and understand WHAT a control system should do,
rather than how it should be done.
• Independent of Project X– NUMI upgrade supposed to operate beyond 2018
• Demanding operational requirements
– For maintainability, older hardware must be replaced• System released from those constraints
– Key architects may retire, maintenance becomes difficult and upgrades become attractive
February 21, 2008 J. Patrick 4
Bottom Line
Fermilab is the premier HEP Accelerator Laboratory in the nation, our control system needs to evolve, and we have taken the first step to understand what is needed.
February 21, 2008 J. Patrick 5
Requirements• What is the functionality of a control system? • Separate from how it will be done (which is the Design phase).• We did not decide on or even discuss specific technology choices
(EPICS, DOOCS, ACNET, or specific hardware, etc.)
• Requirements are:– Agreement between users and developers on needed functionality– If a requirement can not be met, it needs to be revisited and modified– A controlled process– An objective way to evaluate design and technology choices
• A chosen implementation has to meet the requirements.– A standard in industry– A proven way to build in quality upfront.
Software contributes 30% of the failure rate. [see High Availability: A Design Primer]
February 21, 2008 J. Patrick 6
Process
• Series of discussions among Controls, Operations, HINS + Computing Division people
• Each section written by subgroup of experts• Coordinated by and final document edited by
Suzanne Gysin (CD)• Needs broader input
– From machine experts and operations– Hardware experts
• Not considered final, will evolve with more input
February 21, 2008 J. Patrick 7
Controls Requirements Document
• Contents– 1 Introduction (J. Patrick, S. Gysin)– 2 Base Requirements (J. Patrick, S. Gysin)– 3 Low-Level Systems (R. Neswold)– 4 Central Services (A. Petrov)– 5 Application Infrastructure (A. Petrov)– 6 High Level Applications (B. Hendricks)– 7 Controls In A Box (S.Gysin)– 8 Beam-Based Feedback (J. Patrick)– 9 Machine Protection System (B. Hendricks)– 10 Software Development Environment (C. Schumann)– 11 Hardware/Operating Systems (D. Finstrom)– 12 Networks (D. Stenman)
• Document is available: https://beamdocs.fnal.gov/AD-private/DocDB/ShowDocument?docid=2934
• Internal Review (January), i.e. we agree that this is what is needed in an evolved control system.
TCP/IP communication services
TCP/IP communication services
CENTRAL OPERATORCONSOLES
LOCAL OPERATORCONSOLES
FIXEDDISPLAYS
GIG
AB
IT
ET
HE
RN
ET
T
EC
HN
ICA
L N
ET
WO
RK
FILE SERVERSAPPLICATION SERVERS SCADA SERVERS
VME Front Ends Front Ends
PLCs
MACHINE
TCP/IP communication services
PUBLIC ETHERNET NETWORK
TIMING GENERATION
MACHINE
ACTUATORS AND SENSORSCRYOGENICS, VACUUM, ETC…
QUENCH PROTECTION AGENTS,POWER SUPPLIES, …
BEAM POSITION MONITORS,BEAM LOSS MONITORS,BEAM INTERLOCKS,RF SYSTEMS, ETC…
Wor
ldF
IP S
EG
ME
NT
(1,
2.5
MB
its/s
ec)
PLCs
PR
OF
IBU
S
FIP
/IO
OP
TIC
AL
FIB
ER
S
Front Ends
ANALOGSIGNALSYSTEM
T T T T
TT
TT
TT
LHC controls architecture diagram
Hardware Tier
Application Tier
Service Tier
February 21, 2008 J. Patrick 9
Challenges
• Support large multi-accelerator complex• Many simultaneous modes of operation
– Automation, ease of configuration
• Large number of hardware devices– Configuration management, software support– Easy access to information
• Many simultaneous users– Scalable data acquisition
• Application development• Reliability - robust against single component failure• Better stability as beam power increases
– Diagnostics, Fast Feedback, Data Logging
February 21, 2008 J. Patrick 10
Today’s Presentation
• Present the highlights in each section.
• Invite user feedback:– To give feedback, please send us email
February 21, 2008 J. Patrick 11
Document Highlights
• Clock system with more information
• Correlated data
• Front end/service framework consolidation
• Device locking/ownership
• Self-describing mixed type data format
• Improved server-side application environment
February 21, 2008 J. Patrick 12
Document Highlights
• Information at your fingertips• Generic device search• Open applications with devices• Copy/paste data between applications• Controls in a box• Infrastructure for beam-based feedback• Bug tracking system• Supported graphical development environment• Development network
February 21, 2008 C. Briegel 13
Timing System
XClock
1. 1.2 usecs per event
2. 16 bit event
3. 32 bit event counter
4. Self-describing payload
5. 1 GHz phased to TClock
February 21, 2008 C. Briegel 14
Front-End
Common Framework Available at all Tiers
1. Common Software Development
2. Memory Protection
3. Maintain Data Connection
4. Device/System Ownership
5. Acquire Data via Multicast
6. Debugging, Post-mortem
February 21, 2008 C. Briegel 15
Front-EndData Acquisition
1. Client Specification1. One-shot
2. Repetitive
3. Event+Delay
4. Event Sequence Number - Correlated Data
5. On-change+Periodic
6. Array on Event
7. Array on Alarm
8. Real-time (1 KHz)
February 21, 2008 C. Briegel 16
Front-End
Data Acquisition
2. Returned Data1. Raw or Scaled
2. Status
3. Time Stamp
4. Event Sequence Number
5. Self-Describing
February 21, 2008 C. Briegel 17
Front-End
Setting
1. Rapid
2. Forwarded
3. Archived
4. Atomic Set/Read
February 21, 2008 C. Briegel 18
Front-End
Alarms
1. Events
2. Exceptions
3. Default Scan
4. Extensible Scan
5. Bypass Devices or Groups of Devices
February 21, 2008 C. Briegel 19
Network
1. General
DMZ/Firewall
2. Operational
3. Development/Test
Network
February 21, 2008 B. Hendricks 21
Server-Side Application Environment
• Server-side components run continuously, on a schedule, or on-demand
• Run on machines maintained by Controls Dept• Server-side components in a control system:
– Central Services(e.g., data acquisition, data logging)
– Server-side parts of multi-tier applications(e.g., Autotune)
– Web applications(e.g., Supertable, Synoptic Display)
February 21, 2008 B. Hendricks 22
Server-Side Application Environment
• Run unattended– Require an automated tool to start, stop, and monitor the
operation
• Do not have user interface– Require a means to report the status to users (logging)
• Multi-component– Components (Data Acquisition Engine, OAC, servlets) are
dynamically linked to each other
• Need a server-side application environment to perform these common tasks
February 21, 2008 B. Hendricks 23
Server-Side Application Environment
• Several separate “environments”:– Data Acquisition Engine with OACs and data loggers– Tomcat for servlets– Custom shell scripts
• Limited number of supported component types– It's hard to maintain custom components other than
OACs, data loggers, or servlets.
• Complicated development and deployment process
February 21, 2008 B. Hendricks 24
Server-Side Application Environment
• Use a standard application server for server-side components (including shell scripts)
• Customize the server for our needs• Provide common services, such as
authentication • Facilitate development and deployment for
server-side components• Eventually have a cluster of application servers
for redundancy and load balancing
February 21, 2008 B. Hendricks 25
Information at Your Fingertips
• Information is available but scattered through applications
• Inconsistent availability of information
• Inconsistent means for accessing information
February 21, 2008 B. Hendricks 26
Information at Your Fingertips
• Need a generic solution
• Should be provided by the application framework
• Simple click and menu system
• Context sensitive menus– Device names– Node names– Error strings
February 21, 2008 B. Hendricks 27
Information at Your Fingertips
• Device names– Device database attribute information– Device presence in standardized lists
• Datalogger, alarm, save lists• Save files• Saved plot setups• Families of devices
– Send device to another application (including plots)– Find related devices– View alarm and setting history
February 21, 2008 B. Hendricks 28
Information at Your Fingertips
• Node names– Keeper name(s)– Other static attributes of the node– Devices belonging to the node– Current status of the node
• Error strings– Detailed description of the error
February 21, 2008 B. Hendricks 29
Bug Tracking System
• Provides communication path
• Provides persistence of information
• Maintains history
• Provides accountability
• Requires commitment of the user community
• Provides a mechanism for getting issues resolved
February 21, 2008 B. Hendricks 30
Bug Tracking System
• Fix List– Pros
• Web based• Easy to make an entry• Can add images• Easy to link to our electronic logs
– Cons• Passive system• No assignment of responsibility• Problems can remain unresolved• No standard removal mechanism when resolved
February 21, 2008 B. Hendricks 31
Bug Tracking System
• Remedy (Help Desk)– Pros
• Entry method is web based• Active system with reminders• Supports assignment of responsibility
– Cons• More complicated issue entry• Removal method is not straight forward• Doesn’t support attaching images• Limits who can be assigned to resolve an issue• Not managed by us
February 21, 2008 B. Hendricks 32
Bug Tracking System
• What we want in a new system– Open source or commercial product– Web based interface– Simple issue entry and removal– Ability to assign responsible parties– Notification upon entry, update, or removal– Ability to attach images– Easy linkage to our electronic logs– Managed by the Controls Department– Usable by other entities in the division/lab
February 21, 2008 B. Hendricks 33
Controls in a Box
• Control system provides many services
• Large footprint with expensive commercial software
• What about a local or remote system developer?
• What about small testing facilities or experiments?
• Need to be able to scale down
February 21, 2008 B. Hendricks 34
Controls in a Box
• Must be able to stand on its own
• Simple to download and install
• Easy configuration tool
• Shall not use expensive commercial software
February 21, 2008 B. Hendricks 35
Controls in a Box
• Shall provide a local clock (hardware or software)
• Shall provide for local data storage
• Shall include necessary central services including database
• Shall include basic applications for reading and setting data
• Self-contained
February 21, 2008 J. Patrick 36
Next Steps
• Solicit more input– Meet with broader user base
• Select what ideas to implement according to the following criteria:– Benefit the current program, NUMI Upgrade,
and Project X– Incorporating user input– Prevent a divergence in Controls
February 21, 2008 J. Patrick 37
Summary
• Realistic vision of a control system beyond 2010• Presented specific ideas and the selection
criteria to start implementing• Independent of Project X, Fermilab is the
premier Accelerator Laboratory in the nation, our control system needs to evolve
• Ready to provide an organized and systematic evolution of the control system for the FNAL accelerator complex
Top Related