PAL EPICS Meeting Oct. 2012
17
Archive Systems What you always wanted to know but were afraid to ask: What’s available? Who’s doing what? PAL EPICS Meeting Oct. 2012
description
Archive Systems What you always wanted to know but were afraid to ask: What’s available? Who’s doing what?. PAL EPICS Meeting Oct. 2012. 1. Performance?. How many samples/second write & read?. 2. How to get at the data?. Easy web reports?. CSS archive plugin?. 3. All the Rest?. - PowerPoint PPT Presentation
Transcript of PAL EPICS Meeting Oct. 2012
Archive Systems
What you always wanted to know but were afraid to ask:
What’s available?
Who’s doing what?
PAL EPICS MeetingOct. 2012
1. Performance?
How many samples/second write & read?
2
2. How to get at the data?• CSS archive plugin?
3
• Easy web reports?
3. All the Rest?• How to configure?• How to
– start/stop sample engines?– Monitor if they’re
• running?• connected to all channels?• write without errors?
• Impact of disk failure, engine crash?• Limits when data grows?• How to spread data across disks, servers?
– How to delete certain channels or time ranges?– Data backup?
• Info:– Which channel has the most samples in last 24 hours?– Which subsystem generates the most samples?
4
System
• Language, Platform: Java, C++, ??• Data StoreAdvantageGreat Stuff- Disadvantage- ProblemUser, user, user
6
Channel Archiver• C++• Custom Data/Index FilesFast: Write 60k, 70k samples/sec- Need library to read data- Trouble when engine crashed- File size limits- Maintenance nightmareSNS until 2009, KEK, J-PARC,
SLAC, KSTAR, ..
7
Channel Archiver, 64 bit
.. As before, butExtended file size limit-Even Bigger Trouble when engine crashed?-Even Bigger Data maintenance nightmare?Diamond
8
RDB-Based Channel Archiver (BEAUTY)
• Java (Eclipse, CSS)• RDB: MySQL, PostgreSQL, Oracle, … Any data maintenance possible
No size limits; partitions, cluster, …Any type of reporting on dataAnybody can reach data from any programming language
Engines don’t crash, and if they do: Just no new samples for that engine
- Slow: 8000 samples/sec write- ~15k for special MySQL setup
SNS since 2009, KEK, ITER?
9
DESY MySQL Archiver• Java (started with BEAUTY code)• RDB: MySQL with special setup– Writing original sample plus a few averages
Faster writing because of some MySQL optimization
Faster retrieval of long time rangesMost of the usual RDB benefits- Only MySQLDESY?
10
Fermilab (not EPICS)• Java• MySQL ‘BLOBs’ used as ring bufferFasterAutomatically replace oldest data, not running
out of space- Need library to write and read, encode and
decode content of BLOBs- None of the usual RDB advantagesFermilab
11
MyA (Jlab)
• C++?• Homebrew MySQL ‘cluster’– Several MySQL Servers. One table per channel.
FasterMost of the usual RDB benefits- Need library to read from correct MySQL
server and tableJLab
12
Archive Appliance
• Java• Google Protocol BuffersFaster“Appliance” Idea
Submit channel name. Rest Auto-configured.- Need library to access data?SLAC, BNL, IHEP, PSI, ITER?
13
CERN
• Language, Platform: Java, C++, ??• OracleFast because they tuned it well- not EPICSCERN
14
SDDS
• C• SDDS FilesIntegrate4d w/ SDDS tools- Needs library to access- Data managementAPS, …
15
Fast Acqu. Archive
• C• Raw diskFast, for specific purpose- Need lib.- Data managementDiamond
16
SLC History Buffers
• Fortran• Binary filesWorked for decades- SlowSLAC
17
