Sequencing your poo with a usb stick - Linux.conf.au 2016 miniconf - mon 1 feb 2016
Design Examples of Distributed Applications Based on the...
Transcript of Design Examples of Distributed Applications Based on the...
1
4/3/2001 1
Design Examples of DistributedApplications Based on the Use of
Synchronized Clocks
Randy Coverstone and John C. EidsonAgilent Laboratories
Ptolemy Conference, Berkeley,March 22-23, 2001
4/3/2001 4/3/2001 2
Agenda
• Measurement and control based on explicittime
• Techniques for synchronizing clocks• Example systems based on synchronized
clocks• Simulation of distributed, synchronized
clock based systems
2
4/3/2001 4/3/2001 3
Centralized SystemsPC
Network
• Master/slave protocol
•Usually message based design
• Slaves are sequentially polledin a loop.
Scalability of such a system is poor:as the number of channels isincreased,
• Processing load on the controllerincreases,• Complexity of the control programincreases,• Maximum channel sampling ratedecreases,• Ability to sample channels“simultaneously” degrades, and• The worst-case network loadincreases.
[Reading]
Externaltriggerinput
Externaltriggerinput
Externaltriggerinput
BUT: We are familiar withthis style!
4/3/2001 4/3/2001 4
Distributed Systems With ClocksPC
Network
• Master/slave protocol permitted, but not required.
• Increased flexibility in sampling schedule.
• Network messages may now contain accurate time-stamps generated at the point of sensing/actuation
Improved scalability:
• Complexity of the control programremains the same,• Maximum rate at which each channelcan be sampled depends on each node,not the controller,• The minimum time to sample all thechannels is reduced to the amount oftime to sample one channel.• To add more channels you simplyadd more nodes,• Simultaneous acquisition is possible.
[Reading, timestamp,variable, physical units,uncertainty,ID, location, calibrationinfo]
3
4/3/2001 4/3/2001 5
Use of Explicit Time inMeasurement and Control
• Basic uses of time (relative or absolute)– Time stamping system events for debugging– Time stamping data to allow correlation with
other data or events• Coordination of measurement: (sampling and
triggering)• Coordination of action (time based behaviors)
– Timed execution scripts– Temporal mutual exclusion: (time slots)
4/3/2001 4/3/2001 6
Sampling/triggering OptionsCentral clock With distributed clocks
Sequential
SimultaneousRequires simultaneous sample and hold or ext trigger wires.
Differentrates for eachnode
Requires oversampling and decimation.
4
4/3/2001 4/3/2001 7
Timed Execution Scriptnetwork message: start_experiment(time stamp = a)
Script_1:When(t= =a){Initialize_node( );Sample(10khz,a);}When(t= =a+0.0005){Setpoint_1 = 27;}
Script_2:Sample(100khz,now);Start_circular_buffer( );When(t= =a+0.001){Freeze_circular_buffer( );Send_data(a-
0.001,a+0.001);}
4/3/2001 4/3/2001 8
A start into the literature• Atul Adya, Robert Gruber, Barbara Liskov, Umesh
Maheshwari: Efficient Optimistic Concurrency ControlUsing Loosely Synchronized Clocks. SIGMODConference 1995: 23-34
• Barbara Liskov: Practical Uses of Synchronized Clocksin Distributed Systems. Distributed Computing 6(4):211-219 (1993)
• Barbara Liskov, Liuba Shrira, John Wroclawski: EfficientAt-Most-Once Messages Based on Synchronized Clocks.TOCS 9(2):125-142 (1991)
• Real-Time Systems: Design Principles for DistributedEmbedded Applications, Hermann Kopetz, KluwerAcademic Publishers, 1997.
5
4/3/2001 4/3/2001 9
Use of explicit timerepresentation requires:
• A real time clock and associated applicationdevices in each distributed device
• The local time of all clocks to agree towithin some system defined uncertainty
• Clock time is monotonic and ‘continuous• The process synchronizing the clocks to be
a provided system service
4/3/2001 4/3/2001 10
Node Block DiagramNetwork
1451.2driver
STIM dataacquisition
TEDS
STIM
NACKNTRIG
• CHANNEL_ZERO correction timing• Calibration factors• Calibration date
[Reading, timestamp, variable, physical units, uncertainty, ID, location, calibration info]
• NTRIG requests a measurement• NACK indicates request is done
Reading is corrected and stored in buffer. 1451.2
driver
Correctionengine
Data buffer
Real-timeclock
Timestampadjustment
Timestamp
Sample timetriggerregister
Clock syncengine
Node firmwareNCAP
ApplicationCode
6
4/3/2001 4/3/2001 11
Degrees of Time Synchronization• Day• Hour• Minute• Second• 10 msec• millisecond• microsecond• < microsecond
Requires hardware assiste.g. Agilent PTP, TTP/C,GPS,IRIG
No hardware assist requirede.g. Network Time Protocol (NTP)
4/3/2001 4/3/2001 12
Removing jitter
Application layer
Network protocolstack
msecs of delay andjitter
Time packetrecognizer
Application layer
Network protocolstack
Time packetrecognizer
Physical layer Physical layer
< 100 nsecs of delayand jitter
7
4/3/2001 4/3/2001 13
PTP Time Synchronization
Step 1:• Time sync packet from
master to slaves• All TPR’s note the time
this packet appears• Step 2:• Follow-up packet from
master to slaves• All slaves compute
offset and correct theslave clock
Micro-A 10:00AM
Micro-B 10:05AM
TPR-A 10:00AM
TPR-B 10:05AM
Time sync packet
Master Clock
Slave Clock
Micro-A10:00AM
Micro-B10:05AM10:00AM
TPR-A TPR-B
Follow-up packetSend time = 10:00AM
Clockcorrection
circuits
Clock
0:05
4/3/2001 4/3/2001 14
Selecting best master clockSelection criteria• Tracability to UTC• Precision/jitter• Path length• Preferred status
GPS
Example system #1
Example system #2
Repeater
Repeater
Router
Spider
Repeater Repeater
8
4/3/2001 4/3/2001 15
Seconds tick deviations between the clocks of two system instruments
0
20
40
60
80
100
120
140
160
-215 -185 -155 -125 -95 -65 -35 -5 25 55 85 115 145 175 205 235 265 295 325 355
Deviation- nanoseconds
Rel
ativ
e fr
equ
ency
Master
Slave
HP J2610A
HP J2610B
HP J2610B
HP J2610B
σ =68nsmean = 37ns
Seconds tick deviations between theclocks of two system instruments
4/3/2001 4/3/2001 16
Example: Fermentation Control
9
4/3/2001 4/3/2001 17
Fermentation Control
4/3/2001 4/3/2001 18
Gigabit Switch
BuildEvent PC
BuildEvent PC
BuildEvent PC
BuildEvent PC
16 Front End unit
optionalPC
switch10 / 100 MbpsE
ther
net L
ink
Optional PC for local
Processing
16 Front End unit
optionalPC
switch10 / 100 MbpsE
ther
net L
ink
Optional PC for local
Processing
16 Front End unit
OptionalPC
switch10 / 100 MbpsE
ther
net L
ink
Optional PC for local
Processing
24 planes tracker
Load perFront-EndUnit
Switch10/100(Mbits/s)
GigabitSwitchload
102 KBps
(10 Mbps)
1.6 Mbps
(100 Mbps)
38 Mbps
LoadperPC (4)
10 Mbps
1st
leve
l
2
nd l
evel
3
trd
leve
l
Ethernet Network-based DAQ and smart sensorsfor the neutrino long base line OPERA experiment.S. Gardien, C. Girerd, S. Katsanevas, J. Marteau.-CERN
72000 channels120Mbits/sec aggregate
10
4/3/2001 4/3/2001 19
Fig. 2 General block diagram of an Ethernet Capable Frontend Module.
Analog Front End ADC
ControlLogic Time
Stamping
FIFO
Ethernet CapableDevice
Ethernet
PPS
0
63
Fro
m P
MT
or
HP
DOr trigger
Readout logic signals
OptionalHard reset
CalibrationMonitoring ...
CERN-front end
4/3/2001 4/3/2001 20
Example: Data Acquisition
11
4/3/2001 4/3/2001 21
Sample Triggers From Several Instruments
100 triggers/sec10 msec/division
12 Hz sine wave
Clock master
Clock slave
Clock slave
4/3/2001 4/3/2001 22
Sample Triggers From Several Instruments
0.48 triggers/sec200 nsec/division
Clock master
Clock slave
Clock slave
1 usec
12
4/3/2001 4/3/2001 23
Example: Motor Controlcourtesy Indramat
4/3/2001 4/3/2001 24CCD-Box with CLC-Motion control card
HMIConsol
PLC
LWL-Ring with SERCOSInterface
Data-Bus
I / O
InfeedPrintunit 2 Outfeed
Chillrolls
ABRFB
3FU
3FL
Decentrale drive concept
Printunit 3
Printunit 4
Printunit 1
Folder
Commercial printing press(slide courtesy of Indramat)
Decentralized DriveConcept
13
4/3/2001 4/3/2001 25
Flexo Printing Press(Unit Construction)(slide courtesy of Indramat)
60mph =
1 inch/millisecond =
1 mil/microsecond
4/3/2001 4/3/2001 26
Simulation of Time Based SystemsDevice Under Test ( real world time)
Measurement andControl Device
CLOCK-1
Measurement andControl Device
CLOCK-2
Measurement andControl Device
CLOCK-3
Sampling basedon clocks 1, 2, 3
Messaging based on network timing