Beating Pythons GIL to Max Out Your CPUs
-
Upload
andrew-montalenti -
Category
Software
-
view
1.462 -
download
0
Transcript of Beating Pythons GIL to Max Out Your CPUs
Beating Python's GIL!to
Max Out Your CPUsAndrew Montalenti!
CTO, Parse.ly @amontalenti
Scaling Python!to
3,000 CoresAndrew Montalenti!
CTO, Parse.ly @amontalenti
OR:
What happens when you have 153 TB of compressed customer data that may need to be reprocessed at any time,
and it’s now growing at 10-20TB per month?
@dabeaz = “the GIL guy”
Is the GIL a feature, not a bug?!
In one Python process,
at any one time,
only one Python bytecode instruction
is executing at once.
should we just rewrite it in Go?
fast functions!running in parallel
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
from urllib.parse import urlparse urls = ["http://arstechnica.com/", "http://ars.to/1234", "http://ars.to/5678", ...]
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
map(urlparse, urls)
from urllib.parse import urlparse urls = ["http://arstechnica.com/", "http://ars.to/1234", "http://ars.to/5678", ...]
Cythonspeeding up functions on a single core
concurrent.futuresgood map API, but odd implementation details
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
executor = ThreadPoolExecutor() executor.map(urlparse, urls)
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
executor = ProcessPoolExecutor() executor.map(urlparse, urls)
Python subprocess
State
Code
Python subprocess
State
Code
pickle.dumps()
os.fork()
joblibmap functions over local machine cores
by cleaning up stdlib facilities
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
par = Parallel(n_jobs=2) do_urlparse = delayed(urlparse) par(do_urlparse(url) for url in urls)
Python subprocess
State
Code
Python subprocess
State
Code
pickle.dumps()
os.fork()
ipyparallelmap functions over a pet compute cluster
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
rc = Client() rc[:].map_sync(urlparse, urls)
Python
State
Code
Python
State
Code
ipengine
Python
State
Code
Python
State
Code
Python
State
Code
ipengine
ipengine
ipcontroller
Python
State
Code
pickle.dumps()
pickle.dumps()
pykafkamap functions over a multi-consumer log
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
consumer = ... # balanced while True: msg = consumer.consume() msg = json.loads(msg) urlparse(msg["url"])
Python
State
Code
Python
State
CodePython
State
Code
Python
State
Code
Python
State
Code
pykafka.producer
Python
State
Code
pystormmap functions over a stream of inputs
to generate a stream of outputs
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
Python
State
Code
Python
State
CodePython
State
Code
Python
State
Code
pykafka.producer
Python
State
Code
multi-lang json protocol
class UrlParser(Topology): url_spout = UrlSpout.spec(p=1) url_bolt = UrlBolt.spec(p=4, input=url_spout)
pysparkmap functions over a dataset representation
to perform transformations and actions
Python
State
Code
Server 1
Core 2
Core 1
Server 2
Core 2
Core 1
Server 3
Core 2
Core 1
Python
State
Code
Python
State
CodePython
State
Code
Python
State
Code
pyspark.SparkContext
sc = SparkContext() file_rdd = sc.textFile(files) file_rdd.map(urlparse).take(1)
cloudpickle
py4j and binary pipes
"lambda architecture"
Parse.ly "Batch Layer" Topologies with Spark & S3
Parse.ly "Speed Layer" Topologies with Storm & Kafka
Parse.ly Dashboards and APIs with Elasticsearch & Cassandra
Parse.ly Raw Data Warehouse with Streaming & SQL Access
Technology Component Summary
parting thoughts
the free lunch is over, but not how we thought
multi-process, not multi-thread multi-node, not multi-core
message passing, not shared memory !
heaps of data and streams of data
GIL: it's a feature, not a bug.
help us!
pystorm pykafka
streamparse
Questions?tweet at @amontalenti