How to become a SQL Server 2012 DBA - Everything you need to know@
Optiq: a SQL front-end for everything
-
Upload
julian-hyde -
Category
Technology
-
view
6.357 -
download
2
description
Transcript of Optiq: a SQL front-end for everything
![Page 1: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/1.jpg)
Optiq: a SQL front-end for everything
Julian Hyde @julianhyde
http://github.com/julianhyde/optiqhttp://github.com/julianhyde/optiq-splunk
Pentaho Community MeetupAmsterdam, 2012
![Page 2: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/2.jpg)
http://www.flickr.com/photos/torkildr/3462606643
![Page 3: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/3.jpg)
http://www.flickr.com/photos/sylvar/31436961/
![Page 4: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/4.jpg)
“Big Data”
Right data, right time
Diverse data sources / Performance / Suitable format
![Page 5: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/5.jpg)
Use case: Splunk
NoSQL database Every log file in the enterprise A single “table” A record for every line in every log file A column for every field that exists in any log file No schema
SELECT “source”, “product_id”, “http_code”FROM “splunk”.”splunk”WHERE “action” = 'purchase'
![Page 6: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/6.jpg)
How do it (wrong)
Splunk Optiq
SELECT “source”, “product_id”FROM “splunk”.”splunk”WHERE “action” = 'purchase'
“search”
filter
action ='purchase'
![Page 7: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/7.jpg)
How do it (right)
Splunk Optiq
SELECT “source”, “product_id”FROM “splunk”.”splunk”WHERE “action” = 'purchase'
“searchaction=purchase”
![Page 8: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/8.jpg)
Example #2
Combining data from 2 sources (Splunk & MySQL)
Also possible: 3 or more sources; 3-way joins; unions
![Page 9: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/9.jpg)
MySQL
Splunk
Expression treeSELECT p.“product_name”, COUNT(*) AS cFROM “splunk”.”splunk” AS s JOIN “mysql”.”products” AS p ON s.”product_id” = p.”product_id”WHERE s.“action” = 'purchase'GROUP BY p.”product_name”ORDER BY c DESC
join
Key: product_id
group
Key: product_nameAgg: count
filter
Condition:action =
'purchase'
sort
Key: c DESC
scan
scan
Table: splunk
Table: products
![Page 10: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/10.jpg)
Splunk
Expression tree(optimized)
SELECT p.“product_name”, COUNT(*) AS cFROM “splunk”.”splunk” AS s JOIN “mysql”.”products” AS p ON s.”product_id” = p.”product_id”WHERE s.“action” = 'purchase'GROUP BY p.”product_name”ORDER BY c DESC
join
Key: product_id
group
Key: product_nameAgg: count
filter
Condition:action =
'purchase'
sort
Key: c DESC
scan
Table: splunk
MySQL
scan
Table: products
![Page 11: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/11.jpg)
Optiq is not a database.
![Page 12: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/12.jpg)
http://www.flickr.com/photos/torkildr/3462606643
![Page 13: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/13.jpg)
http://www.flickr.com/photos/telstra-corp/5069403309/
![Page 14: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/14.jpg)
Conventional database architecture
JDBC server
SQL parser /validatorQuery
optimizer
Metadata
DataData
Data-flowoperators
JDBC client
![Page 15: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/15.jpg)
Optiq architecture
JDBC server
SQL parser /validatorQuery
optimizer
3rd partydata
3rd partydata
JDBC client
3rd
partyops
3rd
partyops
Optional
Pluggable
Core
MetadataSPI
Pluggablerules
![Page 16: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/16.jpg)
What is Optiq?A really, really smart JDBC driver
Framework
Potential core of a data management system
![Page 17: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/17.jpg)
Writing an adapterDriver – if you want a vanity URL like “jdbc:splunk:”
Schema – describes what tables exist (Splunk has just one)
Table – what are the columns, and how to get the data. (Splunk's table has any column you like... just ask for it.)
Operators (optional) – non-relational operations
Rules (optional, but recommended) – improve efficiency by changing the question
Parser (optional) – to query via a language other than SQL
![Page 18: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/18.jpg)
http://www.flickr.com/photos/walkercarpenter/4697637143/
![Page 19: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/19.jpg)
Optiq roadmap ideas
Mondrian use Optiq to read from data sources such as Splunk & MongoDB, combine multiple data sources
Kettle integration: JDBC front-end; optimize jobs; push down filters & aggregations to data sources (e.g. SQL database)
Adapters: Cascading, MongoDB, Hbase, Apache Drill, …?
Front-ends: linq4j, Scala SLICK, Java8 streams
Contributions
![Page 20: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/20.jpg)
Conclusions
Liberate your data!
Optiq is a framework
Build & share Optiq adapters
![Page 21: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/21.jpg)
Questions?
@julianhyde
http://julianhyde.blogspot.com
http://github.com/julianhyde/optiq
http://github.com/julianhyde/optiq-splunk
![Page 22: Optiq: a SQL front-end for everything](https://reader033.fdocuments.us/reader033/viewer/2022052621/5586c10fd8b42aa6718b46d4/html5/thumbnails/22.jpg)
Additional material: The following queries were used in the demo
select s."source", s."sourcetype" from "splunk"."splunk" as s;
select s."source", s."sourcetype", s."action" from "splunk"."splunk" as s
where s."action" = 'purchase';
select s."source", s."sourcetype", s."action" from "splunk"."splunk" as s
where s."action" = 'purchase';
select s."action", count(*)
from "splunk"."splunk" as s
group by s."action";
select s."action", s."method", count(*)
from "splunk"."splunk" as s
group by s."action", s."method";
select * from "mysql"."products";
select p."product_name", s."action"
from "splunk"."splunk" as s
join "mysql"."products" as p
on s."product_id" = p."product_id";
select p."product_name", s."action", COUNT(*) AS c
from "splunk"."splunk" AS s
join "mysql"."products" AS p
on s."product_id" = p."product_id"
where s."action" = 'purchase'
group by p."product_name", s."action"
order by c desc;