Apache Arrow and Python: The latest
-
Upload
wes-mckinney -
Category
Technology
-
view
2.302 -
download
2
Transcript of Apache Arrow and Python: The latest
1© Cloudera, Inc. All rights reserved.
Apache Arrow and Python in contextWes McKinney @wesmckinn
Data Science Summit 2016-07-12
2© Cloudera, Inc. All rights reserved.
Me
• Data Science Tools at Cloudera• Creator of pandas
• Wrote Python for Data Analysis 2012 (2nd ed coming 2017)
• Open source projects
• Python {pandas, Ibis, statsmodels}
• Apache {Arrow, Parquet, Kudu (incubating)}
• Mostly work in Python and Cython/C/C++
3© Cloudera, Inc. All rights reserved.
WrangleConf - July 28 in San Francisco
http://wrangleconf.comStorytelling from real-world data science
work (and BBQ, of course)
4© Cloudera, Inc. All rights reserved.
Python + Big Data: The State of things
• See “Python and Apache Hadoop: A State of the Union” from February 17
• Areas where much more work needed
• Binary file format read/write support (e.g. Parquet files)
• File system libraries (HDFS, S3, etc.)
• Client drivers (Spark, Hive, Impala, Kudu)
• Compute system integration (Spark, Impala, etc.)
5© Cloudera, Inc. All rights reserved.
Apache Arrow
Many slides here from my joint talk with Jacques Nadeau, VP Apache Arrow
6© Cloudera, Inc. All rights reserved.
Arrow in a Slide
• New Top-level Apache Software Foundation project• Announced Feb 17, 2016
• Focused on Columnar In-Memory Analytics1. 10-100x speedup on many workloads2. Common data layer enables companies to choose best of
breed systems 3. Designed to work with any programming language4. Support for both relational and complex data as-is
• Developers from 13+ major open source projects involved
Calcite
Cassandra
Deeplearning4j
Drill
Hadoop
HBase
Ibis
Impala
Kudu
Pandas
Parquet
Phoenix
Spark
Storm
R
7© Cloudera, Inc. All rights reserved.
High Performance Sharing & InterchangeToday With Arrow
• Each system has its own internal memory format
• 70-80% CPU wasted on serialization and deserialization
• Similar functionality implemented in multiple projects
• All systems utilize the same memory format
• No overhead for cross-system communication
• Projects can share functionality (eg, Parquet-to-Arrow reader)
8© Cloudera, Inc. All rights reserved.
Apache Arrow: What is it?
• http://arrow.apache.org
• Specification matters more than Implementation
• A standardized in-memory representation for columnar data
• Enables
• Suitable for implementing high-performance analytics in-memory (think like “pandas internals”)
• Cheap data interchange amongst systems, little or no serialization
• Flexible support for complex JSON-like data
• Targets: Impala, Kudu, Parquet, Spark
9© Cloudera, Inc. All rights reserved.
Focus on CPU Efficiency
TraditionalMemory Buffer
ArrowMemory Buffer
• Cache Locality
• Super-scalar & vectorized operation
• Minimal Structure Overhead
• Constant value access • With minimal structure overhead
• Operate directly on columnar compressed data
10© Cloudera, Inc. All rights reserved.
Example: Feather File Format for Python and R
•Problem: fast, language-agnostic binary data frame file format
•Written by Wes McKinney (Python) Hadley Wickham (R)
•Read speeds close to disk IO performance
11© Cloudera, Inc. All rights reserved.
Real World Example: Feather File Format for Python and R
library(feather) path <- "my_data.feather"write_feather(df, path) df <- read_feather(path)
import feather path = 'my_data.feather' feather.write_dataframe(df, path)df = feather.read_dataframe(path)
R Python
12© Cloudera, Inc. All rights reserved.
In progress: Parquet on HDFS for pandas users
pandas
pyarrow
libarrow libarrow_io
Parquet files in HDFS / filesystems
Arrow-Parquet adapter
Native libhdfs, otherfilesystem interfaces
C++ libraries
Python + C extensions
Data structures
parquet-cpp
Raw filesystem interface
Python wrapper classes
13© Cloudera, Inc. All rights reserved.
Language Bindings• Target Languages
• Java (beta)
• CPP (underway)
• Python & Pandas (underway)
• R
• Julia
• Initial Focus
• Read a structure
• Write a structure
• Manage Memory
14© Cloudera, Inc. All rights reserved.
RPC & IPC: Moving Data Between SystemsRPC
• Avoid Serialization & Deserialization
• Layer TBD: Focused on supporting vectored io
• Scatter/gather reads/writes against socket
IPC
• Alpha implementation using memory mapped files
• Moving data between Python and Drill
• Working on shared allocation approach
• Shared reference counting and well-defined ownership semantics
15© Cloudera, Inc. All rights reserved.
Executing data science languages in the compute layer
16© Cloudera, Inc. All rights reserved.
Real World Example: Python With Spark, Drill, Impala
17© Cloudera, Inc. All rights reserved.
What’s on the horizon• Parquet for Python & C++
• Using Arrow as intermediary
• IPC Implementation + Java/C++ interop
• Spark, Drill Integration
• Faster UDFs, Storage interfaces
18© Cloudera, Inc. All rights reserved.
Get Involved• Join the community
• Slack: https://apachearrowslackin.herokuapp.com/
• http://arrow.apache.org
• @ApacheArrow
19© Cloudera, Inc. All rights reserved.
Thank youWes McKinney @wesmckinn
Views are my own