Twisted Introduction

84
Twisted “An event-driven networking engine written in Python” Ying Li ([email protected] )

Transcript of Twisted Introduction

Page 1: Twisted Introduction

Twisted“An event-driven networking engine

written in Python”

Ying Li ([email protected])

Page 2: Twisted Introduction

Twisted“An event-driven networking engine

written in Python”

Page 3: Twisted Introduction
Page 4: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Runner / Daemonization

Page 5: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Runner / Daemonization

Page 6: Twisted Introduction

Transport

•manages socket or pipe

Page 7: Twisted Introduction

Transport

•manages socket or pipe•writes data

Page 8: Twisted Introduction

Transport

•manages socket or pipe•writes data•reads data•notifies protocol

Page 9: Twisted Introduction

Transport

•TCP•UDP

•AF_UNIX•Pipes

Page 10: Twisted Introduction

Transport

•TCP•UDP

•AF_UNIX•Pipes

•TLS •memory

Page 11: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Runner / Daemonization

Page 12: Twisted Introduction

Transport

Protocol

Incoming data

•handle data received

Page 13: Twisted Introduction

Transport

Protocol

Outgoing data Incoming data

•handle data received•write data to transport

Page 14: Twisted Introduction

Transport

Protocol

Outgoing data Incoming data

•handle data received•write data to transport•handle connections

Page 15: Twisted Introduction

Transport

Protocol

Page 16: Twisted Introduction

TCP

TLS

Page 17: Twisted Introduction

Process

TLS

Page 18: Twisted Introduction

Bluetooth?

TLS

Page 19: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Runner / Daemonization

Page 20: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Protocol

Transport

Protocol

Transport

Outgoing data Incoming data

...

Page 21: Twisted Introduction

TCP

HTTP

TLS (protocol)

TLS (transport)

Outgoing data Incoming data

Page 22: Twisted Introduction

UDP

HTTP

TCP (protocol)

TCP (transport)

TLS (protocol)

TLS (transport)

Outgoing data Incoming data

Page 23: Twisted Introduction

Protocol

•TLS•HTTP•IRC•XMPP•OSCAR

•IMAP•SMTP•POP•DNS•SSH

•SOCKS•TELNET•SIP (voip)•NMEA (gps)•...more

Page 24: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Runner / Daemonization

Page 25: Twisted Introduction

wait... that’s it?

Page 26: Twisted Introduction

Protocol

Application (e.g. server)

Page 27: Twisted Introduction

Application / Server

•IRC•SMTP•SSH•SFTP•HTTP

•NNTP•POP•Generic•...more

Page 28: Twisted Introduction

Application / Clients

•IRC•SMTP•SSH•SFTP•HTTP

•NNTP•POP•...more

Page 29: Twisted Introduction

server and/or client

Page 30: Twisted Introduction

Protocol

Application

•builds protocol•hooks up protocol

Page 31: Twisted Introduction

Protocol

Application

•builds multiple protocols•database•authentication•hooks up protocol

Page 32: Twisted Introduction

Authentication•unix user/password•ssh key•user/password list file•in-memory user/password

Page 33: Twisted Introduction

OS / System

Transport

Protocol

Application (e.g. server)

Runner / Daemonization

Page 34: Twisted Introduction

python myapplication.py

Application (e.g. server)

Runner / Daemonization

Page 35: Twisted Introduction

twistd [options] <myplugin>

Page 36: Twisted Introduction

twistd•daemonization•logging•privilege dropping•chroot•non-default reactor•profiling

Page 37: Twisted Introduction

twistd [options] <myplugin>[my_plugin_options]

Page 38: Twisted Introduction

twistd

twistd web --port 80 --path /srv/web

Page 39: Twisted Introduction

twistd

twistd web --port 8080 --path /srv/web

twistd telnet --port 4040

Page 40: Twisted Introduction

twistd

•twistd web•twistd telnet•twistd dns

•twistd ftp•twistd mail•twistd conch (ssh)

•...more (see `twistd --help`)

Page 41: Twisted Introduction

Twisted“An event-driven networking engine written in

Python”

Page 42: Twisted Introduction

Concurrency Models(A story of 3 HTTP requests)

Page 43: Twisted Introduction

Time

HTTP Request

Page 44: Twisted Introduction

Time

Await response

Page 45: Twisted Introduction

Time

Process response

Page 46: Twisted Introduction

Time

Request 2 Request 3

Page 47: Twisted Introduction

Time

InterpolateResponses

Page 48: Twisted Introduction

Blocking I/O Request 1Request 2Request 3

Time

Process 1

Process 2

Process 3

Multi-Process

Interpolation

Page 49: Twisted Introduction

Time

Thread 1

Thread 2

Thread 3

Blocking I/O Request 1Request 2Request 3

Interpolation

Multi-Threaded

Page 50: Twisted Introduction

Time

Thread 1

Thread 2

Thread 3

Blocking I/O Request 1Request 2Request 3

Interpolation

Multi-Threaded

Page 51: Twisted Introduction

Time

Thread 1

Thread 2

Thread 3

Blocking I/O Request 1Request 2Request 3

Interpolation

Multi-Threaded

Page 52: Twisted Introduction

Time

Thread 1

Thread 2

Thread 3

Blocking I/O Request 1Request 2Request 3

Interpolation

Multi-Threaded

Page 53: Twisted Introduction

Time

Thread 1

Thread 2

Thread 3

Blocking I/O Request 1Request 2Request 3

Interpolation

Multi-Threaded

Page 54: Twisted Introduction

Threads

• Locking

• Re-entrancy

• Debugging

Page 55: Twisted Introduction

Event-Driven(not to scale)

Page 56: Twisted Introduction

Callbacks:

Time

CB

Page 57: Twisted Introduction

Callbacks:

Time

CB CB

First HTTP Request

Page 58: Twisted Introduction

Time

Callbacks: CB

Second HTTP Request

CB CB

Page 59: Twisted Introduction

Time

Callbacks: CBCB CB

Event!

Page 60: Twisted Introduction

CB

Time

Callbacks:

Event!

CBCB

Page 61: Twisted Introduction

Time

Callbacks: CBCB

CB

Page 62: Twisted Introduction

Time

Callbacks: CB

CB

Event!

CBCB

Page 63: Twisted Introduction

Time

Callbacks: CBCB CB

CB

Event!

Page 64: Twisted Introduction

Time

Callbacks: CB

CB

CB

CB

Event!

Page 65: Twisted Introduction

CB

Time

Callbacks:

CBCBCB

Page 66: Twisted Introduction

CB

Time

Callbacks:

CBCBCB

Page 67: Twisted Introduction
Page 68: Twisted Introduction

Events Handlers

Page 69: Twisted Introduction

• file descriptors

• timed events

Events Handlers

Page 70: Twisted Introduction

• select

• poll

• epoll

• kqueue

• CoreFoundation

• IOCP

• ...more

Page 71: Twisted Introduction

Deferreds

Page 72: Twisted Introduction

Deferreds(promises of future results)

Page 73: Twisted Introduction

D

CBR1

R2

Page 74: Twisted Introduction

D

CBR1

R2

Page 75: Twisted Introduction

D

CB EBF1

R2

Page 76: Twisted Introduction

D

CB EB

CB

CB

1

2

3

R1

CB (CB (CB (R1)))3 2 1

Page 77: Twisted Introduction

D

CB EB

EB

EB

A

B

C

Page 78: Twisted Introduction

D

CB EBF1

try: ...

except: return R2

CB R2

1

2

Page 79: Twisted Introduction

D

CB EBF1=C

try: ...

except A: ...

EB

EB

A

B

C

except B: ...except C: return R2

1

Page 80: Twisted Introduction

D

CB EBF1=C

try: ...

except A: ...

EB

EB

A

B

C

except B: ...except C: return R2

CB R2

1

2

Page 81: Twisted Introduction

CB

D

CB

EB

EB

EBCB

D DD D

CB EB

Page 82: Twisted Introduction

fs.readdir(source, function(err, files) { if (err) { console.log('Error finding files: ' + err) } else { files.forEach(function(filename, fileIndex) { console.log(filename) gm(source + filename).size(function(err, values) { if (err) { console.log('Error identifying file size: ' + err) } else { console.log(filename + ' : ' + values) aspect = (values.width / values.height) widths.forEach(function(width, widthIndex) { height = Math.round(width / aspect) console.log('resizing ' + filename + 'to ' + height + 'x' + height) this.resize(width, height).write(destination + 'w' + width + '_' + filename, function(err) { if (err) console.log('Error writing file: ' + err) }) }.bind(this)) } }) }) }})

courtesy of http://callbackhell.com

Page 83: Twisted Introduction

• jQuery Deferreds

• MochiKit Deferreds

• Dojo Deferreds

• Google Closure Deferreds

Page 84: Twisted Introduction

Twisted•easier to reason about concurrency•lots of components•useful abstractions