ATCA based LLRF system design review
3.12.2007 DESY
Control servers for ATCA based LLRF Control servers for ATCA based LLRF system system
Piotr Pucyk - DESY, Warsaw University of Technology
Jaroslaw Szewinski – Warsaw University of Technology
2ATCA based LLRF system design review
3.12.2007 DESY
Agenda
• Requirements• LLRF servers classification• ATCA computation power for LLRF servers• Servers topology in ATCA based LLRF system • Possible control systems• Development environment• Time schedule and summary
3ATCA based LLRF system design review
3.12.2007 DESY
Requirements
• What servers should do ?
– Should make possible remote access to the applications (update parameters, readout data, etc.)
– Should support hardware in computation with given timing constraints (recalculate tables between pules, etc.)
– Servers should guarantee reliable data transfers to the history storage
– Should provide, for local applications, interface and interaction with external systems (other servers, foreign hardware, etc.)
– Should provide management, control and synchronization of multiple subsystems (procedures, FSMs, automation)
4ATCA based LLRF system design review
3.12.2007 DESY
What control software do we need?
• Finite state machines for automation• Front-end servers for hardware
maintenance, configuration and diagnostics
• Front-end servers for controller and low level applications (execution nodes for state machines)
• Middle layer servers for high level applications
• DAQ servers or interfaces to DAQ• GUI panels, interfaces to Matlab, C,
etc.
Diagnostics maintenance
Diagnostics maintenance
End-node for FSM, controller server
End-node for FSM, controller server
High level apps, FSM
High level apps, FSM
DAQ
GUI, external apps
GUI, external apps
5ATCA based LLRF system design review
3.12.2007 DESY
Different CPUs in ATCA based LLRF systemEmbedded processors
AMC module processors
ATCA blades mainframes
Computation power
• Huge processing power
• DAQ, storage
• Post processing
• GbE
+ server class processing power
- No PCIe until now
+ serious processing power
- uses AMC slot on carrier
+ PCIe, GbE
- Low processing power
- FPGA resources
+ close to the hardware
6ATCA based LLRF system design review
3.12.2007 DESY
FPGA
Possible topology of LLRF servers
AMC CPU
IO AMC (FPGA)
FPGA embedded
CPU
ATCA Carrier board
CPU
ATCA CPU blade
CPU
Mainframe blades
CPU CPU CPU CPU
CPU CPU CPU CPU
CPU CPU CPU CPU
CPU CPU CPU CPU
PCIe
GbE
GbE
Front-end middle layerSimple FSM
Front-end middle layerSimple FSM
Front-end
Front-end
Middle layer FSMFront-end
Middle layer FSMFront-end
FSM, DAQ
FSM, DAQ
7ATCA based LLRF system design review
3.12.2007 DESY
Possible control systems
• DOOCS
– Used for the FLASH control
• EPICS
– Widely used, low hardware requirements
• TINE
– Heterogeneous/portable, runs on different platforms (DOS, Windows, various UNIXes, VxWorks, etc), can use different network protocols (TCP/IP, IPX/SPX, etc.)
• Dedicated software
– custom servers, often embedded
• Hybrid configurations
– Standard control system server (on ATCA CPU board) as gateway for small, lightweight custom embedded servers in different ATCA facilities (AMC, Carrier, etc.)
8ATCA based LLRF system design review
3.12.2007 DESY
What we can reuse, what we have to develop
• Reuse
– Old communication scheme (one software interface – many hardware drivers), memory description map
– Tools for debugging and configuration
– Some existing server’s source code
• New development
– Diagnostics, maintenance interfaces (inc. IPMI, crate management)
– FSM framework for high level apps
– New communication channels
9ATCA based LLRF system design review
3.12.2007 DESY
Development environment & tools
• Platforms - preferred free, open systems like Linux or FreeBSD. Linux is widely used on various systems (from embedded to mainframe and cluster – can run in different facilities of ATCA). It has good hardware support - excellent platform for development.
• Usage of real time (commercial) systems (like VxWorks) – only where Linux can not be used.
• Technologies - 'keep it simple when possible ' - preferred well known, widely portable technologies like:
– C/C++
– BSD sockets + TCP/IP
– Pthreads
– etc.
10ATCA based LLRF system design review
3.12.2007 DESY
Schedule, manpower
• Before we start:– Development environment, control system, communication
libraries, FSM framework.• Development schedule strongly depends on other tasks
– Configuration and maintenance servers (when carrier board and at least one AMC is debugged and ready for firmware implementation - 3-5.2008 ?)
– Finite state machines and procedures – starting from 1.2008 ? – Controller server, and low level applications interface servers
(parallel to controller development and low level applications)• Minimum Manpower
– 1 fulltime programmer for front-ends, 1 fulltime for FSM and high level apps, 1-2 students for help
– A lot of support from MCS group !!!
12ATCA based LLRF system design review
3.12.2007 DESY
Development & Design guidelines
• Usage open-source, free tools when possible • Projects organized by Makefiles, to enable automatic build of all sub-
components, including cross-builds for different targets (CPUs on AMCs, DSPs, PPCs in FPGA, ATCA CPU boards, etc.)
• Codes organized by version control tools (CVS, SVN)• Codes documented by tools like Doxygen• Common coding conventions & policy• Separated mechanism (platform dependent) and policy (platform
independent)• Avoid platform depended, hardly portable solutions• Avoid fancy external tools/libraries/technologies which development
or support my be stopped one day.
13ATCA based LLRF system design review
3.12.2007 DESY
Old System Scheme
FPGA
LPT
VME
???
Memory map descriptionFile / parser
TCP Server
Hardware Channels EngineUser
Applications
Inte
rnal In
terfa
ce
Eth
OtherApplication
s
TCP/IP
14ATCA based LLRF system design review
3.12.2007 DESY
Linux on PowerPC
User applications access hardware through the driver
Kernel mode driver has access to the FPGA
FPGA has defined hardware interface
Virtex II PRO
Linuxon PPC
Hardware
FPGA II Core
Client onremote
machine
Kernelmode
Internal Interface bus
driver
Usermode
TCPserver
TC
P/IP
15ATCA based LLRF system design review
3.12.2007 DESY
DOOCS patterns
http://flash.desy.de/sites/site_vuvfel/content/e403/e1644/e1136/e1137/infoboxContent1796/tesla2006-10.pdf
Top Related