Asynchronous I/O in Python 3
-
Upload
feihong-hsu -
Category
Technology
-
view
11.263 -
download
5
description
Transcript of Asynchronous I/O in Python 3
![Page 1: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/1.jpg)
Asychronous I/O in Python 3
Feihong HsuChicago Python User Group
July 11, 2013
![Page 2: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/2.jpg)
What is PEP 3156?
• "Asynchronous IO Support Rebooted"
• Describes a new event loop API
• Aims to be compatible with current popular event loops (Twisted, Tornado, etc.)
• Should land in the standard library in 3.4 (early 2014)
![Page 3: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/3.jpg)
The big take-away
We're going to be able to write asynchronous code without threads or callbacks!
![Page 4: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/4.jpg)
Threads vs event loop
• I'm not going to talk about this
• Twisted being around for more than a decade kinda validates the event loop model
• For a detailed discussion see An Intro to Asynchronous Programming and Twisted
![Page 5: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/5.jpg)
Twisted vs Tulip
Ugh, how to explain this?
I could just give you a boring list of differences...
Or I could use an analogy...
![Page 6: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/6.jpg)
Programming with Twisted is like...
Being a mystical ninja warrior!
![Page 7: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/7.jpg)
Mystical Ninja Warrior
• Can defeat demons and demigods by deploying an army of shadow clones
• Mental gymnastics required to coordinate shadow clones requires years of brutal training
• Ninja magic comes with a terrible price: the more clones you make, the more insane you become
![Page 8: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/8.jpg)
Programming with Tulip is like...
Playing Plants Vs Zombies
![Page 9: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/9.jpg)
Playing Plants Vs Zombies
• Can defeat a zombie horde by deploying and managing a large number of horticultural specimens
• Not particularly mentally taxing
• Somehow end up playing for hours without eating or sleeping
![Page 10: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/10.jpg)
Alrighty
With the analogies out of the way, let's look at some examples.
![Page 11: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/11.jpg)
Fetching a web page in Twisted
https://github.com/feihong/tulip-talk/blob/master/examples/1-twisted-download.py
![Page 12: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/12.jpg)
Control flow diagram
I was going to draw one but then I got sleepy.
![Page 13: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/13.jpg)
Problems with callback-based asynch code
• Can be hard to understand control flow
• Error messages are not friendly
• Debugging is harder
![Page 14: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/14.jpg)
Fetching a web page in Tulip
https://github.com/feihong/tulip-talk/blob/master/examples/2-tulip-download.py
![Page 15: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/15.jpg)
Fetching a web page in Tulip w/ callbacks
https://github.com/feihong/tulip-talk/blob/master/examples/3-tulip-download-callbacks.py
![Page 16: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/16.jpg)
Fetching a web page in Twisted w/ inline
callbackshttps://github.com/feihong/tulip-talk/blob/master/examples/4-twisted-download-inline-callbacks.py
![Page 17: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/17.jpg)
The Tulip API
• Basics are documented in PEP 3156
• Not available in Python Package Index
• Includes sockets, file I/O, etc.
• Includes concurrency data structures like locks, queues, semaphores
• Includes SSL and HTTP (with support for websockets!)
![Page 18: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/18.jpg)
Coroutines vs tasks
When I first started, the most confusing thing about Tulip.
Coroutines are executed by "yield from".
Tasks are not executed by "yield from".
![Page 19: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/19.jpg)
Coroutine
• Basically a function that contains at least one "yield from" statement.
• Not the same thing as a generator function, which is a function that contains a yield expression.
• Tulip will barf if you try to make it execute a generator.
![Page 20: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/20.jpg)
Coroutine
@tulip.coroutine
def download(url):
response = yield from tulip.http.request('GET', url)
return yield from response.read()
Calling "download()" returns a generator object, but otherwise does nothing! You need to do "yield from download()" to run the body of a coroutine.
![Page 21: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/21.jpg)
Task
• An object that manages a coroutine
• Roughly equivalent to the Deferred object in Twisted
![Page 22: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/22.jpg)
def download(url):
response = yield from tulip.http.request('GET', url)
return yield from response.read()
Calling "download()" actually does run the body of the function. The "yield from" part is done implicitly for you.
![Page 23: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/23.jpg)
Task
In Tulip, there are two ways of creating tasks:
tulip.task is a decorator that produces a task-wrapping function
tulip.Task is a constructor that accepts a coroutine
![Page 24: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/24.jpg)
Why bother to use tasks?
• To interoperate with callback-based frameworks like Twisted
• To cancel an already-running coroutine
• To start a new coroutine from within another coroutine
![Page 25: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/25.jpg)
Wait, so what's a Future?
PEP 3156 makes frequent mention of Futures.
But talking about Futures is a little confusing, since there are two Future classes: tulip.Future and concurrent.futures.Future.
![Page 26: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/26.jpg)
Future is the superclass of Task
• Future don't necessarily manage a coroutine
• In practice you never create Future objects, only Task objects
• Futures are acceptable to yield from expressions
![Page 27: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/27.jpg)
Methods on Future
• cancel(), cancelled()
• running(), done()
• result(), set_result()
• add_done_callback(), remove_done_callback()
• ...
![Page 28: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/28.jpg)
Web development with Tulip
• All the classes you need are in the tulip.http module
• Make subclass of tulip.http.ServerHttpProtocol
• Override the handle_request() method
![Page 29: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/29.jpg)
Hello World!
https://github.com/feihong/tulip-talk/blob/master/examples/5-hello-world-http.py
![Page 30: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/30.jpg)
Some observationsThe HTTP API is fairly simple, but a bit low level for everyday web programming.
Expect a thousand microframeworks to bloom in the near future.
Speaking of microframeworks...
![Page 31: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/31.jpg)
Introducing viol
• Tiny web framework based on Tulip
• After initial page load, messages between client and server are exchanged via websockets
• Makes code demos a bit more visual
![Page 32: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/32.jpg)
Tulip API demos
Now for a bunch of demos...
![Page 33: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/33.jpg)
Tulip API demos
![Page 34: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/34.jpg)
Serial
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=1546
![Page 35: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/35.jpg)
Simultaneous
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=1666
![Page 36: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/36.jpg)
Wait for both coroutines to finish
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=1713
![Page 37: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/37.jpg)
Loop through coroutine results in order of finishing
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=1869
![Page 38: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/38.jpg)
Execute synchronous code in another thread
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=1930
![Page 39: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/39.jpg)
Talk to event loop from another thread
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=2060
![Page 40: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/40.jpg)
A regex debugging tool
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=2158
![Page 41: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/41.jpg)
Chat application
http://www.youtube.com/v/jMgRUI7V_mk?version=3&autoplay=1&&start=2236
![Page 42: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/42.jpg)
Tulip Links
• PEP 3156
• Guido's PyCon 2013 keynote
• Tulip project page on GoogleCode
![Page 43: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/43.jpg)
Other links
• An Introduction to Asynchronous Programming and Twisted
• Code for this talk on Github
• Viol project page on Github
![Page 44: Asynchronous I/O in Python 3](https://reader034.fdocuments.us/reader034/viewer/2022042607/554f8b76b4c905435d8b4de4/html5/thumbnails/44.jpg)
Questions?