Hardware Platforms for ESS ICS

Timo KorhonenICS

www.europeanspallationsource.seNov.. 5, 2014

2

Overview

• Scope of the hardware platform decision(s)• Goals • Development strategies• Platforms

• High-speed digital platform• Middle range I/O• Industrial I/O

• Conclusions

3

Scope of the decision(s)

• Control systems have a lot of interfaces• Need to cater for a spectrum of I/O requirements

• Fast, real-time signal processing• State-of-the-art technology, evolving fast• FPGA-based processing• MHz to GHz range of signal acquisition

• Middle-range I/O• requires synchronization, kHz range I/O

• Non-real time industrial I/O• Typically PLC-based

• Off-the-shelf devices• Serial, Ethernet or other fieldbus devices

• No single platform can cover this cost-efficiently!

4

Goals

• Sufficient performance for the task at hand• Data processing & transport capabilities• Time synchronization

• Cost-efficiency• Looking at the total cost, not only component cost

• Software (and firmware) dominates cost• Roadmap for evolution

• Beware of obsoleteness traps• Project timespan – prepare for change

• Efficiency of development• ICS provides a platform for system development• Division of work – development and integration

5

Development strategies

• Use standardized equipment whenever possible• Select platforms to support and promote them (via a

harmonization committee: E2H2C)• Benefit from experience in the community

• Prefer standards that are used in the community • Benefit from collaborations

• Open sharing of work – direct peer-to-peer• Use proven solutions whenever possible

• Sometimes new solutions are required, though• Use modular systems

• Parts can be replaced without redoing the whole system

• Interfaces need to be clearly defined

6

Platforms

• It is not reasonable to do everything with a single platform• In the past, VME and CAMAC have been such ones• There will be many industrial-type systems• We need to be flexible (one size does not fit all)

• High-end, digital processing platform• For beam instrumentation and LLRF applications• Acquisition in MHz and above range• Response times in microseconds

• Middle range I/O• Often forgotten but important• Not highest speed but real-time capable and flexible

• Industrial I/O• PLC is the standard way in industry

7

High-speed Digital Platform

• Performance requirements (very shortly)• Support 14 Hz operation

• Good integration with timing• Acquisition (ADC sampling) at 88 MSPS or faster

• 2.86 ms long pulses: a lot of data to be handled• Local pre-processing of data• Interfaces to machine protection/interlock• Post-mortem abilities

• In fact, any (packaging/bus) platform could provide this!• The real questions are elsewhere:

• Maintainability• Development and long-term sustainability• Collaborations and support for IKC partners

8

High-speed Digital Platform – the box

• Today, there is no clear single choice• cPCI, VME, custom (aka “pizza”) box, MTCA.4,...?

• Serial links have overtaken parallel buses• Intelligence is in the interfaces and protocols• The “box” supplies infrastructure (power, cooling,

monitoring, connectivity, some convenience features)• Take a proven solution or go for something new?

• Several labs are looking at MTCA.4• DESY (main promoter), SLAC, many others

• Packaging standard supports a modular approach • Different components can be mixed (interoperability!)• Serviceability, component upgrades, etc.• Potentially a long lifecycle (if enough deployment)

9

High-speed Digital Platform

• What to take, then?• Decide on one (until now, cPCI and MTCA)• (quite) some work has been done on MTCA

• Using a single card (SIS8300) + RTMs• MTCA.4 native timing receiver in preparation• Seems prudent to continue on the MTCA path

• Decided on a platform – fine. What then?• Platform alone does not provide any functionality• Systems and processes have to be built around

• Here lies the major investment!• We have some components developed but

• Deployment will bring up many issues• Systems need to be maintained for a long time

10

High-speed Digital Platform – and beyond

• Lifetime issues• ESS project timespan > 10 years

• Obsolecense issues will arise during the project• Need to be ready for change!

• Biggest cost factor is in integration• Software, firmware, maintenance processes• How could that be modularized?

• Development of a full lifecycle system is costly• Has somebody addressed the deployment issues?• How to do gradual upgrades?• How to secure maintainability?

• Deciding the packaging standard is not enough• A more comprehensive framework is needed

11

High-speed Digital Platform – blocks

• Processing in FPGA is the standard way today• Monolithic designs have been common

• Reuse of components is hard• Needs special skills

• FMC (FPGA Mezzanine Card), VITA-57 standard addresses these issues• Separate analog (application-specific) and digital parts

with a defined interface• One digital board can be used for several purposes

• Reduce hardware stock complexity• Re-use of firmware libraries, software, integration

• Modularization at the level of the major investment• Firmware, software, operational procedures• Separation of work (defined interfaces)

12

High-speed Digital Platform - framework

• Set up a collaboration with PSI (ESS partner lab)• In the last 3 years, PSI invested a lot into development

of a framework that addresses the issues of:• Performance and flexibility• Maintainability and integration• Lifecycle management and modularity

• Done with industry (small company) collaboration• We could benefit from that

• Even if the form factor is different – majority of effort is in the infrastructure for applications

• Over 10 person-years worth of work • Used in production (PSI LLRF, diagnostics, etc.)

• Interest also from other labs (Daresbury, DESY..)

13

Framework proposal - development

• Port the existing framework to MTCA.4 hardware• Hardware implementation by the company IOxOS SA

• Reuse existing designs, with upgrades• Rapid hardware development

• Soft- and firmware can be retained close to 100%• Open collaboration with PSI and IOxOS

• Full documentation and source code sharing• Most tricky problems have already been solved

• Interrupts, DMA handling, etc• Collaboration in the first line, can develop into IKC• In the mid-term, port previous work to this platform

• All previous work can be used in the short term• Minimize distruptive sudden changes

14

Middle-range I/O

• High-end platform is expensive and centralized• Use only where needed

• Some applications still need time synchronization• Especially in a pulsed machine!• PLCs are not ideal for this (asynchronous cycle times)

• EtherCAT is a good solution for this range• Uses Ethernet cabling, special protocol• Can run on a PC or a “hard” IOC• Loop times of several kHz possible• Distributed I/O (reduce cabling)• Cost effective• Many I/O modules, several manufacturers

• Gaining popularity in the EPICS community

15

Industrial I/O

• PLC is the standard choice for most industrial-type I/O• High reliability• Widely known and deployed in partner labs and

industries• A huge selection of I/O modules available

• Ideally, select one manufacturer• Reduce development and maintenance costs

• One skillset, reduced stock of spares• Select a manufacturer with domain expertise

• Local support for IKC partners• Procure via an open call for tender

• Not quite straightforward, though – but in progress

16

“Other” I/O

• There is still a large number of devices not covered yet• Serial devices

• Use MOXA – seems to be almost de-facto standard• Ethernet-based devices

• Ethernet as a fieldbus, use StreamDevice & ASYN• Cameras

• GigE, 10GBE, CameraLink(HS)• Very much provider-dependent – take what the

company gives• EPICS AreaDetector supports a large device base

• Motion control• Working together with our NSS colleagues

17

Summary

• A lot of discussion revolves around the high-end platform• But that is only a part of the story

• work-intensive and critical part, but still• Other areas tend to get too little attention

• try to set up a comprehensive set of solutions• We have selected MTCA.4 for the high-end

• Knowing that there will be issues to solve• But they can be solved

• We try to establish and leverage collaborations• Mutual help between institutes

• Other areas are equally, if not even more urgent• Need to keep going there, too• Start prototyping (in our new lab)

18

Questions?

19

Reserve slides

20

System interfaces (PSI board for IS&LEBT)

FPGA

FMC

Event Receiver

PCIeswitch

CPU

(signal interfaces) Timing distribution

Ethernet

(EPICS)

• PCI express for internal (and external) communication

• Timing (Event Receiver) on-board via PCI

• EPICS running locally

• Ethernet to the outside world

21

PSI board: a picture tells more than 1kWords

CPU PowerPC P2020•1.2 GHz dual core• runs real-time Linux / EPICS•Boots over LAN, uSD card, or on-board FLASH

FPGA Virtex-6 LX130T•TOSCA-II PCI-express Network on-chip IP•connects: 512 MB shared memory (DDR3)•connetcs: VME, user logic, FMC sites, VME_P2•User FPGA code

PCI-express GEN2 switch•central interconnect betweenCPU / FPGA / XMC / VME_P0

•contains Non-Transparent (NT) function