Sadayuki FuruhashiFounder & Software Architect
Treasure Data, inc.
internalsPostgreSQL protocol gateway for Presto
A little about me...> Sadayuki Furuhashi
> github/twitter: @frsyuki
> Treasure Data, Inc. > Founder & Software Architect
> Open-source hacker > MessagePack - Efficient object serializer > Fluentd - An unified data collection tool > ServerEngine - A Ruby framework to build multiprocess servers > Prestogres - PostgreSQL protocol gateway for Presto > LS4 - A distributed object storage with cross-region replication > kumofs - A distributed strong-consistent key-value data store
Today’s talk
1. What’s Presto?
2. Prestogres design
3. Prestogres implementation
4. Prestogres hacks
5. Prestogres future works
What’s Presto?
A distributed SQL query engine for interactive data analisys against GBs to PBs of data.
What’s the problems to solve?> We couldn’t visualize data in HDFS directly using
dashboards or BI tools > because Hive is too slow (not interactive) > or ODBC connectivity is unavailable/unstable
> We needed to store daily-batch results to an interactive DB for quick response(PostgreSQL, Redshift, etc.) > Interactive DB costs more and less scalable by far
> Some data are not stored in HDFS > We need to copy the data into HDFS to analyze
HDFS
Hive
PostgreSQL, etc.
Daily/Hourly BatchInteractive query
CommercialBI Tools
Batch analysis platform Visualization platform
Dashboard
HDFS
Hive
PostgreSQL, etc.
Daily/Hourly BatchInteractive query
✓ Less scalable ✓ Extra cost
CommercialBI Tools
Dashboard
✓ Extra work to manage 2 platforms
✓ Can’t query against “live” data directly
Batch analysis platform Visualization platform
HDFS
Hive Dashboard
Presto
PostgreSQL, etc.
Daily/Hourly Batch
HDFS
HiveDashboard
Daily/Hourly Batch
Interactive query
Interactive query
Data analysis platform
Presto
HDFS
HiveDashboard
Daily/Hourly BatchInteractive query
Cassandra PostgreSQL Commertial DBs
SQL on any data sets
Data analysis platform
Presto
HDFS
HiveDashboard
Daily/Hourly BatchInteractive query
Cassandra PostgreSQL Commertial DBs
SQL on any data sets CommercialBI Tools
✓ IBM Cognos ✓ Tableau ✓ ...
Data analysis platform
Prestogres
Presto
HDFS
Dashboard
Interactive query
CommercialBI Tools
✓ IBM Cognos ✓ Tableau ✓ ...
Prestogres
Today’s topic!
What can Presto do?> Query interactively (in milli-seconds to minues)
> MapReduce and Hive are still necessary for ETL > Query using commercial BI tools or dashboards
> Reliable ODBC/JDBC connectivity through Prestogres > Query across multiple data sources such as
Hive, HBase, Cassandra, or even internal DBs > Plugin mechanism
> Integrate batch analisys + visualizationinto a single data analysis platform
Presto’s deployment
> Facebook > Multiple geographical regions > scaled to 1,000 nodes > actively used by 1,000+ employees > who run 30,000+ queries every day > processing 1PB/day
> Netflix, Dropbox, Treasure Data, Airbnb, Qubole > Presto as a Service
The problems to use Presto with BI tools
> BI tools need ODBC or JDBC connectivity > Tableau, IBM Cognos, QlickView, Chart.IO, ... > JasperSoft, Pentaho, MotionBoard, ...
> ODBC/JDBC is VERY COMPLICATED > Matured implementation needs LONG time
• psqlODBC: 58,000 lines • postgresql-jdbc: 62,000 lines • mysql-connctor-odbc: 27,000 lines • mysql-connector-j: 101,000 lines
Other possible designs were…
a) MySQL protocol + libdrizzle: > Drizzle provides a well-designed library to implement
MySQL protocol server. > Proof-of-concept worked well:
• trd-gateway - MySQL protocol gateway for Hive > Difficulties: clients assumes the server is MySQL but,
• syntax mismatches: MySQL uses `…` while Presto “…” • function mismatches: DAYOFMONTH(…) vs EXTRACT(day…)
b) PostgreSQL + Foreign Data Wrapper (FDW): > JOIN and aggregation pushdown is not available
Other possible designs were…
c) PostgreSQL + H2 database + patch: > H2 is an embedded database engine written in Java > H2 has a PostgreSQL protocol implementation in Java > Difficulties:
• System catalog implementation is incomplete(pg_class, pg_namespace, pg_proc, etc.)
d) Reusing PostgreSQL protocol impl.: > Difficulties:
• complete implementation of system catalogs was too difficult
Prestogres design
pgpool-II + PostgreSQL + PL/Python > pgpool-II is a PostgreSQL protocol middleware for
replication, failover, load-balancing, etc. > pgpool-II originally has some useful code
(parsing SQL, rewriting SQL, hacking system catalogs, …) > Basic idea:
• Rewrite queries at pgpool-II and run Presto queries using PL/Python
select count(1)from access
select * frompython_func(‘select count(1) from access’)
rewrite!
psql
pgpool-IIodbc
jdbc
PostgreSQL Presto
Authentication Create faked systemcatalogs for meta-queries
1. 2.
Rewriting queries Executing queries using PL/Python
3. 4.
Overview
Patched!
psql
pgpool-IIodbc
jdbc
PostgreSQL Presto
Authentication Create faked systemcatalogs for meta-queries
1. 2.
Rewriting queries Executing queries using PL/Python
3. 4.
Overview
Patched!
Prestogres
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
StartupPacket { database = “mydb”, user = “me”, … }
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
prestogres.conf
system_db_dbname = ‘postgres’ system_db_user = ‘prestogres’
prestogres_hba.conf
host mydb me 0.0.0.0/0 trust presto_server presto.local:8080, presto_catalog hive, pg_database hive
StartupPacket { database = “mydb”, user = “me”, … }
pgpool-IIpsql PostgreSQL Presto
StartupPacket { database = “mydb”, user = “me”, … }
$ psql -U me mydb
prestogres.conf
system_db_dbname = ‘postgres’ system_db_user = ‘prestogres’
> CREATE DATABASE hive; > CREATE ROLE me; > CREATE FUNCTION setup_system_catalog; > CREATE FUNCTION start_presto_query;
libpq host=‘localhost’, dbname=‘postgres’, user=‘prestogres’ (system_db)
prestogres_hba.conf
host mydb me 0.0.0.0/0 trust presto_server presto.local:8080, presto_catalog hive, pg_database hive
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
prestogres_hba.conf
host mydb me 0.0.0.0/0 trust presto_server presto.local:8080, presto_catalog hive, pg_database hive
prestogres.conf
system_db_dbname = ‘postgres’ system_db_user = ‘prestogres’
StartupPacket { database = “hive”, user = “me”, … }
StartupPacket { database = “mydb”, user = “me”, … }
system catalog!
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
“Q” SELECT * FROM pg_class;
"Query against a system catalog!”
Meta-query
system catalog!
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
SELECT setup_system_catalog(‘presto.local:8080’, ‘hive’)“Q” SELECT * FROM pg_class;
"Query against a system catalog!”
Meta-query
PL/Python functiondefined at prestogres.py
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
SELECT setup_system_catalog(‘presto.local:8080’, ‘hive’)“Q” SELECT * FROM pg_class;
> CREATE TABLE access_logs; > CREATE TABLE users; > CREATE TABLE events; …
Meta-query
SELECT * FROM information_schema.columns
"Query against a system catalog!”
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
“Q” SELECT * FROM pg_class; “Q” SELECT * FROM pg_class;
Meta-query"Query against a system catalog!”
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
“Q” select count(*) from access_logs;
regular table!
Presto Query"Query against a regular table!”
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
“Q” select count(*) from access_logs; SELECT start_presto_query(… ‘select count(*) from access_logs’)
regular table!
Presto Query"Query against a regular table!”
PL/Python functiondefined at prestogres.py
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
“Q” select count(*) from access_logs; SELECT start_presto_query(… ‘select count(*) from access_logs’)
> CREATE TYPE result_type (c0_ BIGINT); > CREATE FUNCTION fetch_results RETURNS SETOF result_type …
regular table!
Presto Query"Query against a regular table!”
1. start the query on Presto
2. define a function to fetch the result
pgpool-IIpsql PostgreSQL Presto
$ psql -U me mydb
“Q” select count(*) from access_logs; “Q” SELECT * FROM fetch_results();
Presto Query"Query against a regular table!”
PL/Python functiondefined by start_presto_query
“Q” RAISE EXCEPTION …
Faked current_database()
delete from pg_catalog.pg_proc where proname=‘current_database’;
create function pg_catalog.current_database()returns name as $$begin return ‘faked_name’::name;end$$ language plpgsql stable strict;
Check: www.treasuredata.com
Cloud service for the entire data pipeline, including Presto. We’re hiring!
Top Related