Net::HTTP Async Programming2011.rubyworld-conf.org/files/slides/A-5.pdf · Features • Keepalive...

Net::HTTP and Async Programming

Sunday, September 4, 11

I Like Net::HTTP


Features

• Keepalive

• Transfer-Encoding: Chunked

• Streaming Requests

• GZip and Deflate

• Threading Support

• Socket Management

• form-data and urlencoded

• SSL and SSPI

• Proxies

• Basic Auth

• Pure Ruby


Bugs :(

>> http = Net::HTTP.new("www.engineyard.com", 80)=> #<Net::HTTP www.engineyard.com:80 open=false>

>> http.get("/javascripts/application.min.js") { }TypeError: can't convert Net::ReadAdapter into String


http://www.engineyard.com

http://www.engineyard.com

http://www.engineyard.com:80

http://www.engineyard.com:80

Net::HTTP.new(host, port)

Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk endend



Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk endend clean up response



Net::HTTP.start(host, port) do |http| http.get('/path') do |chunk| # do something with chunk endend

clean up socket


Manual

http = Net::HTTP.start(host, port) http.get('/path') do |chunk| # do something with chunkend # keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend


Manualhttp = Net::HTTP.start(host, port)

http.get('/path') do |chunk| # do something with chunkend # keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend

http.finish


Manualhttp = Net::HTTP.start(host, port)

http.get('/path') do |chunk| # do something with chunkend # keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend

http.finish clean up socket


Socket

http = Net::HTTP.start(host, port) http.get('/path') do |chunk| # do something with chunkend

# keepalive if HTTP 1.1 and no# Connection: closehttp.get('/another') do |chunk| # do something with chunkend

http.finish

socket = TCPSocket.new(host, port) socket.puts "GET / HTTP/1.1"socket.puts "Host: www.google.co.jp"socket.puts socket.read

socket.puts "GET / HTTP/1.1"socket.puts "Host: www.google.co.jp"socket.puts socket.read

socket.close


http://www.google.co.jp




Why?なぜ?


HTTP Proxy

Client WEBrick Google

1 Thread

stream


Parallel Requests


1 Thread

stream




Blocking Read


get

http = Net::HTTP.start(host, port)http.get('/path') # block


GenericRequest

http = Net::HTTP.start(host, port)http.get('/path') # block

req = Net::HTTPGenericRequest.new( 'GET', false, true, '/path')http.request(req) # block


transport_request

def transport_request(req) begin_transport req res = catch(:response) { req.exec @socket, @curr_http_version, edit_path(req.path) begin res = HTTPResponse.read_new(@socket) end while res.kind_of?(HTTPContinue) res.reading_body(@socket, req.response_body_permitted?) { yield res if block_given? } res } end_transport req, res resrescue => exception D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise exceptionend


transport_request

req.exec @socket, @curr_http_version, edit_path(req.path)

begin res = HTTPResponse.read_new(@socket)end while res.kind_of?(HTTPContinue)

body = req.response_body_permitted?

res.reading_body(@socket, body) { yield res if block_given?}


Socket Leak Impossible


Close Socket

def end_transport(req, res) @curr_http_version = res.http_version if @socket.closed? D 'Conn socket closed' elsif !res.body && @close_on_empty_response D 'Conn close' @socket.close elsif keep_alive?(req, res) D 'Conn keep-alive' else D 'Conn close' @socket.close endend


Not Flexible


HTTP Proxy


1 Thread

stream


Net2::HTTP


Net2::HTTPdef transport_request(req) begin_transport req req.exec @socket, @curr_http_version, edit_path(req.path) begin res = HTTPResponse.read_new(@socket) end while res.kind_of?(HTTPContinue) res.request = req if block_given? yield res res.close end

end_transport req, res, block_given? @current_response = resrescue => exception D "Conn close because of error #{exception}" @socket.close if @socket and not @socket.closed? raise exceptionend


Net2::HTTPreq.exec @socket, @curr_http_version, edit_path(req.path)

begin res = HTTPResponse.read_new(@socket)end while res.kind_of?(HTTPContinue) res.request = req if block_given? yield res res.closeend


Block Form

def end_transport(req, res, block_form) @curr_http_version = res.http_version if @socket.closed? D 'Conn socket closed' elsif @close_on_empty_response && !res.body D 'Conn close' @socket.close elsif keep_alive?(req, res) D 'Conn keep-alive' elsif block_form D 'Conn close' @socket.close endend


Parallel Requests


1 Thread

stream




"Async"


Blocking

foo = File.read("/foo")# blockbar = File.read("/bar")


Parallel I/O

foo, bar = nil

t1 = Thread.new do foo = File.read("/foo")end

t2 = Thread.new do bar = File.read("/bar")end

t1.joint2.join


One-Thread Parallel I/O

f1 = File.open("/usr/share/misc/operator", "r")f2 = File.open("/usr/share/misc/flowers", "r")foo, bar = "", "" until f1.eof? && f2.eof? begin IO.select [f1, f2] foo << f1.read_nonblock(1024) bar << f2.read_nonblock(1024) rescue IO::WaitReadable endend f1.closef2.close


IO.select read_ios

[ ]#<IO:fd 3>#<IO:fd 4>

until read_ios.any? { |io| i.ready? }


read_nonblock(size)

#<IO:fd 4>

bytes_available?

bytes[0...size] raise Errno::EWOULDBLOCK

yes? no?


General Purpose "Reactor"

f1 = File.open("/usr/share/misc/operator", "r")f2 = File.open("/usr/share/misc/flowers", "r")foo, bar = "", "" callbacks = {}

callbacks[f1] = proc { |chunk| foo << chunk }callbacks[f2] = proc { |chunk| bar << chunk }


General Purpose "Reactor"

until callbacks.empty? read, _ = IO.select callbacks.keys read.each do |io| begin callbacks[io].call io.read_nonblock(1024) rescue EOFError io.close callbacks.delete io end endend


Nonblocking Protocol

to_ioread_nonblock

closeeof?

Errno::EWOULDBLOCKEOFError


Nonblocking Protocol

to_ioread_nonblock

closeeof?

IO::WaitReadableEOFError


Demo


Proposal


Ruby Nonblocking Protocol

to_ioread_nonblock

closeeof?

Errno::EWOULDBLOCKErrno::EOFError


ありがとう！@wycats

profiles.google.com/wycatshttp://yehudakatz.com


Net::HTTP Async Programming2011.rubyworld-conf.org/files/slides/A-5.pdf · Features • Keepalive...

Documents

Transcript of Net::HTTP Async Programming2011.rubyworld-conf.org/files/slides/A-5.pdf · Features • Keepalive...