CHEP’04, 27th. Sep. – 1st. Oct., 2004
1
An Embedded Linux System An Embedded Linux System Based on PowerPCBased on PowerPC YE MeiYE [email protected]@ihep.ac.cnExperimental Physics Center Experimental Physics Center Institute of High Energy PhysicsInstitute of High Energy Physics
CHEP’04, 27th. Sep. – 1st. Oct., 2004
2
OutlineOutline
1.1. IntroductionIntroduction
2.2. The Readout SystemThe Readout System
3.3. Overview of the Overview of the TestbedTestbed
4.4. Design and ImplementationDesign and Implementation
5.5. ConclusionsConclusions
CHEP’04, 27th. Sep. – 1st. Oct., 2004
3
IntroductionIntroduction
•12 years since the commission of the BES at BEPC
• Both machine and detector have undergone the upgrade
•The peak luminosity of 1033cm-2sec-1
• The estimated event rate: about 4000Hz at J/ψpeak
• The new electronics will up to higher event rate and has much more noise environment.
The DAQ system will adopt embedded real time operation system(RTOS) and corresponding programming techniques to fit the readout requirement of the Front-End Electronics(FEE).
CHEP’04, 27th. Sep. – 1st. Oct., 2004
4
The Readout SystemThe Readout System
• Designed to adopt Motorola’s PowerPC
• Real-time operation system : Multi-level parallel data collecting and Real-time operation system : Multi-level parallel data collecting and processing schemesprocessing schemes
• Work with the readout electronics: performs initialization and Work with the readout electronics: performs initialization and calibrations of sub-systems; collect, pre-process and transfer datacalibrations of sub-systems; collect, pre-process and transfer data
• Several readout crates are connected to a readout PC through fast Ethernet, thus constituting a readout branch
• All the readout branches are connected to the online computer farm through gigabit switch
• The events are recorded in persistent media.
CHEP’04, 27th. Sep. – 1st. Oct., 2004
5
The Readout SystemThe Readout System
• The main tasks of the readout system are:The main tasks of the readout system are:
• collecting event data from the front-end electronics after collecting event data from the front-end electronics after
Level 1 triggerLevel 1 trigger
• pre-processing and assemblingpre-processing and assembling
• transferring data fragments from the VME readout crate transferring data fragments from the VME readout crate
to the readout PC to the readout PC
CHEP’04, 27th. Sep. – 1st. Oct., 2004
6
Benefit from Embedded LinuxBenefit from Embedded Linux
• High price of the commercial RTOS: VxWorks, LynxOS
• The possibility of using the embedded real-time Linux as the operating system of the readout system.
In comparison with VxWorks (Single Seat) : 1:3In comparison with VxWorks (Single Seat) : 1:3
CHEP’04, 27th. Sep. – 1st. Oct., 2004
7
Overview of the Development EnvironmentOverview of the Development Environment
TimeSys Analysis ToolsWindows
Online Software , Linux
Gigabit Ethernet
100M Ethernet
Readout PCRH 7.3 ~ 9.0
TimeSys Linux Target
FEEs
PowerPC:MVME5500
Debug
Host X86
TimeSys SDK & IDE Env.RH 7.3 ~ 9.0
Host X86
CHEP’04, 27th. Sep. – 1st. Oct., 2004
8
The Readout SystemThe Readout System
To accomplish above tasks, the testbed system must be designed to have the following functions:
• To collect data from the VME readout modules (ADC and TDC) at To collect data from the VME readout modules (ADC and TDC) at high speed and full utilize the VME bus bandwidth;high speed and full utilize the VME bus bandwidth;
• To set up the multi-level buffering techniques that solve the To set up the multi-level buffering techniques that solve the “bottleneck” problem caused by the VME data fragments assembling “bottleneck” problem caused by the VME data fragments assembling and the network transmission; and the network transmission;
• To use the necessary task synchronization strategy and software To use the necessary task synchronization strategy and software protocols that ensure the correct parallel processing;protocols that ensure the correct parallel processing;
• To build the effective channel between the kernel module and the To build the effective channel between the kernel module and the user module;user module;
• To simulate the Front-End Electronics modules to generate the To simulate the Front-End Electronics modules to generate the VMEbus trigger and Monte Carlo data;VMEbus trigger and Monte Carlo data;
• To establish the APIs between the readout code and the online To establish the APIs between the readout code and the online software.software.
CHEP’04, 27th. Sep. – 1st. Oct., 2004
9
Battery
NVRAMRTC
FLASH32 MB
(soldered)8MB
(sockets)
66 MHz PCI Local Bus -- A32/D64-bitA32/D64-bit 100 MHz PCI Bridge & Memory Controller
Discovery
Universe II
133MHz SDRAM
512MBPowerPC ISA
MPC7455@ 1 GHz
L3 Cache200MHz
2MB
SDRAMMezzanine
512MB
66 MHz PCI Local Bus
PMC FP I/O
PMCSlot 2
P1
IPMCSlot 1
P2
PCIBridge
33 MHz PCI Local Bus -- A32/D64-bit
10/100Ethernet
PHY
10/100 BaseTX F/P I/O
10/100/1000BaseTXF/P I/O
PMCspanConnector
GigabitEthernet
Controller
PMC FP I/OJumperSelectable
Port 1
Port 2
A32/D64-bit
UART
Header
EIA232
FP I/OCom 1 Com 2
PowerPC Intro PowerPC Intro —— MVME5500 MVME5500
Features:• MPC7455 microprocessor at 1 GHz• 2MB of L3 cache • 100 MHz front-side bus • 256/512MB of on-board ECC SDRAM —
expandable up to 1GB with optional
memory expansion module• 40MB Flash memory • Dual IEEE P1386.1 compatible 32/64-bit
33/66 MHz PMC expansion slots • 64-bit PCI expansion mezzanine
connector allowing up to four more
PMCs • Dual serial ports • 10/100 and Gb Ethernet interfaces• Single VME slot when configured with
PMC modules and memory mezzanine• MVME51xx I/O compatibility• MOTLod firmware
CHEP’04, 27th. Sep. – 1st. Oct., 2004
10
PowerPC Intro PowerPC Intro —— MVME2431 MVME2431
• MotorolaMotorola 2400 2400 Series:MVME2431Series:MVME2431
• CPU:MPC750 PowerPC(350MHZ) CPU:MPC750 PowerPC(350MHZ)
• Up to 1MB of backside L2 cache Up to 1MB of backside L2 cache
• 32MB SDRAM 32MB SDRAM
• 8MB of on-board Flash(bankB), 8MB of on-board Flash(bankB),
1MB of socketed Flash(bankA),1MB of socketed Flash(bankA),
• 10/100BaseT Ethernet10/100BaseT Ethernet
• One serial portOne serial port
• PPCBug:evaluation and debugging PPCBug:evaluation and debugging
tooltool
CHEP’04, 27th. Sep. – 1st. Oct., 2004
11
Fully Preemptible Linux GPL Kernel
Standard Linux/POSIX APIs
TimeSys Linux RTOS OverviewTimeSys Linux RTOS Overview
PrioritizableInterruptHandler
Constant Time Scheduler
RTOS Modules• POSIX Real-Time• Priority Inversion
Control KLM• High Resolution
Timer KLM
Reservations Add-on
PrioritizableSoft-IRQ(aka Bottom
Half)
Linux App Linux App JTime Add-on…
CHEP’04, 27th. Sep. – 1st. Oct., 2004
12
TimeSys Linux RTOS:TimeSys Linux RTOS:Software Development Kit (SDK)Software Development Kit (SDK)
TimeSys Linux RTOS SDKs
2.4.21 Linux Kernel• GPL Performance Improvements• RTOS Kernel Loadable Modules• Certified drivers for hardware• CGL and CE Linux features
Root File System• CGL & CE Linux Packages• 100+ Packages & 196 Tools
Reservations Add-on• Loadable kernel module• CPU and Network
resource management
JTime Add-on• RTSJ compliant Java
Virtual Machine
GUI-Based Tools• TimeStorm Linux Development
Suite • TimeStorm IDE• TimeTrace
Certified Toolchains• gcc/g++ 3.2, gdb 5.3, glibc 2.2.5,
binutils 2.13, KGDB 5.2.1• Windows or Linux hosting
CHEP’04, 27th. Sep. – 1st. Oct., 2004
13
Professional & Designer Editions: Integrated Professional & Designer Editions: Integrated with TimeStorm IDEwith TimeStorm IDE
• Full-featured IDE for any Linux
• Kernel, driver and application development, debug, and deployment
• Linux & Windows based IDE
• Utilizes standard tools
• GCC, GDB, Make, ……
• Foundation for all TimeStorm Tools
• Built on Eclipse framework
• Extensibility: 350+ open-source and commercial plug-ins available
• Eclipse v2.1 and CDT v1.0.1 Easily create, download and
debug C/C++/Java code
CHEP’04, 27th. Sep. – 1st. Oct., 2004
14
Professional & Designer Editions: Integrated Professional & Designer Editions: Integrated with TimeTracewith TimeTrace
• Graphically represent exactly what is happening within a system.
• Critical events unfold within an application
• Context switches
• Mutex locks/unlocks
• Scheduling events
• Timer events
• Many others including user defined
• Dynamically instrumented on “production” code!
CHEP’04, 27th. Sep. – 1st. Oct., 2004
15
Measurement TechniqueMeasurement Technique
• Test the performance of the system such as:Test the performance of the system such as:
• The time spend on VME read and write operationsThe time spend on VME read and write operations
• The speed of DMA read operationThe speed of DMA read operation
• System latency when receive interruptSystem latency when receive interrupt
• Network throughput and the CPU rate on network Network throughput and the CPU rate on network
transferstransfers
• Integrated Test: Block/SignalIntegrated Test: Block/Signal
• Response JitterResponse Jitter
CHEP’04, 27th. Sep. – 1st. Oct., 2004
16
System LatencySystem Latency
Ti meSys VxWorksItem MVME5500->2431 MVME5100->2431
One VME Read 1942(ns) 1330(ns)
One VME Write 407(ns) 440(ns) DMA(A32/D32/BLT) 18. 6 MBps 18. 9 MBpssi ze=4096(Bytes)
DMA(A32/D64/MBLT) 35. 9 MBps 37. 9 MBpssi ze=4096(Bytes)
DMA Overhead(K) 7. 1/ 17. 1(μ s) 8. 6(μ s)Interrupt Latency( H.S.) 9. 9/ 14.4(μ s) 10. 2(μ s)Network Speed(1024) 94. 26(Mbps) 90. 4(Mbps)Network CPU Rate 15% 52%Network Idle 85% 48%
CHEP’04, 27th. Sep. – 1st. Oct., 2004
17
Readout ModuleReadout Module
Front End(MVME2431)
Front End(MVME2431)
Online Software
MVME5500
DMADMA PACKPACK NETNET
DataSize
RngBuf RngBuf
DataSize
DA
TA
VM
EB
us
Shake Hand
DATA
DATA
Trig.
Shake Hand
Handler
Signal
CHEP’04, 27th. Sep. – 1st. Oct., 2004
18
Result with Block ModuleResult with Block Module
• Intr(blk) +DMA +Pack+NetIntr(blk) +DMA +Pack+Net• msg_q(2)+rngBuf(2)msg_q(2)+rngBuf(2)
• 1 Trig.1 Trig.
Event Rate : 10300Event Rate : 10300 CPU = 72.8%CPU = 72.8%
• 16 Trig.16 Trig.
Event Rate : 18157Event Rate : 18157 CPU = 47.4%CPU = 47.4%
• VxWorks Ref.VxWorks Ref. • 1 Trig.1 Trig. Event Rate : 11600Event Rate : 11600 CPU = 88%CPU = 88%
• 16 Trig. 16 Trig. Event Rate : 16000Event Rate : 16000 CPU = 84%CPU = 84%
CHEP’04, 27th. Sep. – 1st. Oct., 2004
19
The Result with Signal ModuleThe Result with Signal Module
• Intr(sig)+Pack+DMA+NetIntr(sig)+Pack+DMA+Net• msg_q(2)+rngBuf(2)msg_q(2)+rngBuf(2)
• 1 Trig.1 Trig.
Event Rate : 8190Event Rate : 8190 CPU = 68.3%CPU = 68.3%
• 16 Trig.16 Trig.
Event Rate : 18186Event Rate : 18186 CPU = 51.4%CPU = 51.4%
CHEP’04, 27th. Sep. – 1st. Oct., 2004
20
The Result with IPC Message QueueThe Result with IPC Message Queue
• Intr(sig)+Pack+DMA+NetIntr(sig)+Pack+DMA+Net• Msg_IPC(2)+rngBuf(2)Msg_IPC(2)+rngBuf(2)
• 1 Trig.1 Trig.
Event Rate : 8287Event Rate : 8287 CPU = 68.0%CPU = 68.0%
• 16 Trig.16 Trig.
Event Rate : 18100Event Rate : 18100 CPU = 53.0%CPU = 53.0%
CHEP’04, 27th. Sep. – 1st. Oct., 2004
21
Jitter Tests of Real-Time OS PerformanceJitter Tests of Real-Time OS Performance
(Under Heavy Idle Task and Network Load) (Under Heavy Idle Task and Network Load)• Intr(sig)+DMA+Net+rngBuf(1)Intr(sig)+DMA+Net+rngBuf(1)• Intr(blk)+DMA+Net+rngBuf(1)Intr(blk)+DMA+Net+rngBuf(1)
Exit When the rngBuf fullExit When the rngBuf full
11
1 1 1 1 1
0
0. 2
0. 4
0. 6
0. 8
1
1. 2
256/247267(TB)512/159466(TB)1024/153629(TB)2048/74679(TB)4096/63183(TB)8192/52724(TB)16384/52049(TB)
CHEP’04, 27th. Sep. – 1st. Oct., 2004
22
TimeSys Linux/Real-Time
Embedded Linux
Real-Time Unix
Desktop OSCommercial Unix
Widely-Used RTOS
Benchmark from TimeSys/Linux RTOSBenchmark from TimeSys/Linux RTOS
CHEP’04, 27th. Sep. – 1st. Oct., 2004
23
Long Time Test RunLong Time Test Run
80008000Hz,Hz,2020HrsHrs
1000010000Hz,Hz,1717HrsHrs 1800018000Hz,Hz,6262HrsHrs
Net: 12MB/sNet: 12MB/s
CHEP’04, 27th. Sep. – 1st. Oct., 2004
24
ConclusionsConclusions
• As experiment discussed above the strategy using the As experiment discussed above the strategy using the embedded embedded real-time Linuxreal-time Linux as the operating system of the as the operating system of the PowerPC PowerPC is able to fitis able to fit the essential requirement of the the essential requirement of the readout system. readout system.
• In addition to the task of readout and net transmission In addition to the task of readout and net transmission there is still there is still enough CPU idleenough CPU idle to ensure the stabilization of to ensure the stabilization of the system. the system.
• Considering the nice price and performance of the Considering the nice price and performance of the TimeSys Linux/real-time the TimeSys Linux/real-time the DAQ could take into accountDAQ could take into account the possibility of adopting the the possibility of adopting the real-time Linuxreal-time Linux..
CHEP’04, 27th. Sep. – 1st. Oct., 2004
25
ThanksThanks
27th Sep. 200427th Sep. 2004Congress CentreCongress CentreInterlaken, SwitzerlandInterlaken, Switzerland
Top Related