CS 136, Advanced Architecture Storage Performance Measurement.
Measurement Flow Architecture in OML
description
Transcript of Measurement Flow Architecture in OML
Jolyon WhiteGEC9, 4th November 2010
Measurement Flow Architecture in OML
OML = Measurement Flows
2
Rutgers University, New Jersey
NICTA, Sydney
Deutsche Telekom Labs@ TU Berlin
BOWL Testbed
National Broadband Network100Mbs FTTH
VoD Trial
IREELNetwork EducationTeaching Platform
Rail Bridge Monitoring Sensors
NSW Road Traffic Authority
Parking DiscoveryRutgers
Marco Gruteser
Current OML data pipeline
Application or
Service
Measurement points Filters Measurement streams
OML Server
Database
(SQL)
Database tables
File
OML client library3
Schemas
• Schemas enable:– Provenance– Processing in the pipeline (data crunching)
• Measurement Stream schema == Combination of schemas of filter outputs
• Each MS stored in its own DB table
4
MP (A, B, C) A
B
C
(S, T)
(U, V, W)
(X, Y)
(S, T, U, V, W, X, Y)
MS Schema
Schemas
• Example: app name is “otr2”
• SQL issued to the database:
• Schema names + metadata define provenance5
avg avg : DOUBLEmax : DOUBLEmin : DOUBLE
ts : DOUBLEflow_id : INT32seq_no : UINT32pkt_length : UINT32src_host : STRINGsrc_port : STRING
MP udp_in:
CREATE TABLE otr2_udp_in ([METADATA COLS], pkt_length_avg REAL, pkt_length_max REAL, pkt_length_min REAL);
Measurement Collection Graph
• Modularize producers + consumers• Measurement Point (MP) – data source• Processing Point (PP) – buffer, select, filter, join,
forward• Termination Point (TP) – persistent storage
6
MP
MP
MP
PP
PP
TP
TP
TP
PP Metadata Store
ServicesAPI
MDA(Measurement Data Archive)
Resource provisioning
• OML has no concept of resource provisioning• Experimenter obtains resources for I&M identically
to experimental resources– i.e. no distinction between I&M and experiment resources
• User has full control over how resources used• Useful defaults, but allow more if experimenter
wants it• Can’t always cleanly separate I&M from
experiment– Mobile wireless testbeds where I&M must share compute
+ network with experiment– E.g. Parknet
• Almost all wireless traffic was measurement flows7
Transports
• OML currently supports two custom procotols– Text version– Binary version
• Standard transports are good!• We like IPFIX, aiming to support it (near future)• Why? Several reasons but:
– Template support self-describing measurement streams
8
Metadata headers(schemas) Measurement flow
Metadata headers(schemas) Measurement flow
Processing Point Applications
9
Proxy Server
• Buffer measurements on command– Don’t transmit to remote server
• Same protocol as server– Transparent to client applications
Proxy server OML ServerApplication
CMD_BUFFERCMD_REPLAY
10
Hierarchical Measurement Collection
• High-resolution measurements lose value over time
• Local storage may be limited• Measuring at different granularities• Inspired by existing research in Streaming
Databases– Numerous VC-backed startups in financial data feed
processing space
11
Context-Driven Experiment Steering
• Dynamic experiments need measured context feedback
• E.g. Geographic trip lines, link state feedback
12
Context-Driven Measurement
• Environment feedback can be used to influence the measurement process itself
13