Statim, time series interface for Perl.
-
Upload
thiago-rondon -
Category
Technology
-
view
901 -
download
0
description
Transcript of Statim, time series interface for Perl.
![Page 1: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/1.jpg)
Statimtime series interface for Perl
Thiago RondonYAPC::Brasil 2011
![Page 2: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/2.jpg)
“Science progresses throught observation”-- Isaac Newton
![Page 3: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/3.jpg)
“The greatest value of a picture is that is forces us to notice what we never expected to see”-- John Tukey
![Page 4: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/4.jpg)
Where ?• Astronomy
• Medicine
• Stock Market
• Meteorology
• Biometrics
• Geology
• Apps
• Operating System
• Networks
• Robotics
• Events
• ....
![Page 5: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/5.jpg)
Wait! let’s see Chart::Cliker examples.
![Page 6: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/6.jpg)
Line
![Page 7: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/7.jpg)
StackedLine
![Page 8: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/8.jpg)
Bar
![Page 9: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/9.jpg)
StackedBar
![Page 10: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/10.jpg)
Area
![Page 11: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/11.jpg)
StackedArea
![Page 12: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/12.jpg)
Bubble
![Page 13: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/13.jpg)
CandleStick
![Page 14: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/14.jpg)
Pie
![Page 15: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/15.jpg)
PolarArea
![Page 16: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/16.jpg)
What ?
• time series is a sequence of data points, measured typically at successive times spaced at uniform time intervals.
• interface define a set of methods and conventions that provides a consistent time series interface.
![Page 17: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/17.jpg)
Why ?
• real time statistics.
• simple way to deploy.
• support openstandards
• distributed input
• provides default methods, functions, tools and etc.
![Page 18: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/18.jpg)
Motivation
• independent
• many writers
• abstract layer on top of other databases to map and reduce.
• Portability
• easy integration
![Page 19: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/19.jpg)
But, we need more.
• 3D, Visualizing data.
• Euclidean distance
• Find similar patterns
• Anomaly detection
• Rule discovery
• Frequent pattern discovery
![Page 20: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/20.jpg)
Architecture
TCPServer
Perl Client
TCP Client
REST Client
engineschemastorage
RESTServer
![Page 21: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/21.jpg)
Conventions
• data collection
• period (time units)
• step and timestamp.
• type of aggregation
• fields
![Page 22: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/22.jpg)
Engine
• AnyEvent
• TCP
• REST
![Page 23: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/23.jpg)
Storage• Key Interface
• Redis
• Memcached
• Table Interface
• DBI
• File
• RRD Interface
• RRD
![Page 24: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/24.jpg)
Schema{ "collection" : {
"period" : "84600","aggregate" : "sum","fields" : {
"foo" : "count","bar" : "enum","jaz" : "enum"
} }}
![Page 25: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/25.jpg)
Command
• add, get and set
• period, step
• version, quit
![Page 26: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/26.jpg)
Commands• add collection foo:jaz bar:1
• OK 1
• add collection foo:jaz bar:1
• OK 2
• add collection foo:wow bar:10
• OK 10
• get collection foo:jaz bar
• OK 2
• get collection bar
• OK 12
• set collection foo:jaz bar:2
• OK 2
![Page 27: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/27.jpg)
Commands with “ts”
• You can pass timestamp with “ts”.
• add collection foo:bar ts:123467890 bar:1
• You can get a new series with “ts”
• get collection foo:bar ts:1234567880-1234567890 bar
![Page 28: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/28.jpg)
Commands with step
• You can pass step with “step”.
• add collection foo:bar step:-10 bar:1
• You can get a new series with “step”
• get collection foo:bar step:-10..-20 bar
![Page 29: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/29.jpg)
step vs timestamp
• You can get the step in server by:
• step 1234567890
• Or make the calc in the client.
![Page 30: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/30.jpg)
timestamp vs step
• Timestamp make easy life to insert data
• Step make easy life to get data
![Page 31: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/31.jpg)
New time series by periods.
• get collection bar:jaz ts:$ts1-$ts2 foo:max
• get collection bar:* ts:$ts1-$ts2 foo:max
• get collection bar:jaz ts:$ts1-$ts3 foo:min
• get collection ts:$ts1-$ts3 foo:sum
• get collection ts:$ts1-$ts3 foo:avg
• get collection ts:$ts1-$ts3 foo:distinct
![Page 32: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/32.jpg)
New time series by periods.
• get collection ts:$ts1-$ts2 foo:trend(5)
• get collection ts:$ts1-$ts2 foo:anomaly(100)
• get collection ts:$ts1-$ts3 foo:frequency(200)
![Page 33: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/33.jpg)
Chart !
• You can use any lib to do that.
• Server-side, eg. Chart::Cliker.
• Client-site, eg. html5 and javascript libs.
![Page 34: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/34.jpg)
REST
• You can extend to any protocol.
• statim-rest : /method/[args]
![Page 35: Statim, time series interface for Perl.](https://reader034.fdocuments.us/reader034/viewer/2022042606/547c30e0b4af9fc3158b5095/html5/thumbnails/35.jpg)
Thanks.
• http://github.com/maluco/Statim
• http://search.cpan.org/dist/Statim
• @thiagorondon