"An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx...
-
Upload
dataconomy-media -
Category
Data & Analytics
-
view
91 -
download
1
Transcript of "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx...
![Page 1: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/1.jpg)
An Introduction to Kx Technology: A Big Data Solution
8th December 2016
Chris Leckey Kx Systems
![Page 2: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/2.jpg)
Est. 1993, California, USA
Founder's Concern: Volume
Widely adopted in financial services over two decades
Global company, Large user community
Other industries: Energy, Pharmaceuticals, Telecommunications, Sport
About Kx Systems
![Page 3: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/3.jpg)
Kx Clients & Industries
![Page 4: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/4.jpg)
About the Technology
• Integrated in-memory, columnar
database & programming system
• Map-Reduce built-in
• Parallelism
• Time as native
• Streaming, real-time and historical
data.
• Lightweight Executable (800kB)
• Low TCO: Low O&M costs Standard OS & Hardware
![Page 5: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/5.jpg)
q at a Glance
• Functional array processing language
• Vector based
• Descended from Ken Iverson's APL
• Right to Left Execution
• Low Latency:
Interpreted, not compiled
Runs inside database
• Terse - Get more from less code
Project Euler Problem 1 in qProject Euler Problem 1 in q
Project Euler Problem 1 in JavaProject Euler Problem 1 in Java
44 Characters44 Characters
233 Characters233 Characters
![Page 6: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/6.jpg)
q – Simple Query Syntax
6
sym price sizeJPM 34.51 105
GE 71.23 97BP 18.69 23
JPM 34.53 16
JPM 34.55 85BP 18.72 81
GE 71.15 48
sym price size
JPM 34.51 105
JPM 34.53 16
JPM 34.55 85
Trade Table Query
Result
![Page 7: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/7.jpg)
Kx Performance Comparison
• Dataset: NYSE TAQ 5000 days. 1.1 Trillion quotes. 65 Billion trades. 100TB raw text.
• Machine: 16 core 256GB. Date partitioned. Symbol indexed.
• All queries in RAM (kdb+ 3.x March 2015)
• Time to query for smallest day trades. 2003.09.10 35M quotes. 5M trades.
![Page 8: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/8.jpg)
Kx Performance
“Kx’s kdb+, running on Dell EMC DSSD D5 and PowerEdge servers, has established performance records while testing on the largest STAC-M3 data scale so far.”
Peter Lankford, STAC Director
![Page 9: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/9.jpg)
• The Independent Electricity System Operator (IESO)• Largest multi-tenant meter data processing service (4.7 million meters)• > 300 billion rows of data, growing by 120 million per day• Selected and applied Kx to support:
Analytics not previously possible On demand access to all of the data by utilities and their customers 24x7x365 availability and near real-time synchronization with transaction
system.
Kx for Utilities Case Study
![Page 10: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/10.jpg)
Service PreviousSolution
KxSolution
Web-services response time 30 sec 30 ms
Number of requests 30/sec 20,000/sec
Ad-hoc queries, aggregations, bulk extracts Hours-Days Second-Minutes
Service Availability 20 hours,6 days/week 24 hours,7 days/week
Kx for Utilities Case Study
![Page 11: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/11.jpg)
• Maintains investment in existing system.
• Rapid implementation of Kx technology.
• Low risk or impact on existing system.
• Functionality and availability improved.
Data Feed
Existing Systemof
Record
Time series & Master
Data in kdb+
Kx Technologies
Ad hocQueries
Analysis
Dashboards
Time series data
Real-time or scheduled replication
Query results
Kx for Utilities Case Study
![Page 12: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/12.jpg)
Dashboards for Kx
• Real-time streaming support.
• Easy to use, no programming
experience required.
• Drag-and-drop HTML 5 interface, with
over
35 ready-made components.
• Compatible with all modern browsers.
![Page 13: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/13.jpg)
Dashboards for Kx
![Page 14: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/14.jpg)
Dashboards for Kx
![Page 15: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/15.jpg)
Dashboards for Kx
![Page 16: "An Introduction to Kx Technology: A Big Data Solution" Chris Leckey, a Data Scientist from Kx System's Northern Ireland Office](https://reader031.fdocuments.us/reader031/viewer/2022030308/58ef86501a28abcb128b456d/html5/thumbnails/16.jpg)
Köszönöm!
Kx® and kdb+ are registered trademarks of Kx Systems, Inc., a subsidiary of First Derivatives plc.
Email: [email protected]
Free 32-bit Download: https://kx.com/download/