
Agenda

System callsFile descriptor semanticsBlocking, nonblocking and async I/OThe Reactor PatternPatterns and gotchas



Caveats

Can’t run before you walkPatterns are framework agnosticKISSread, write and connect only



Head count ?

Eventmachine (or any client libs)RedisNginxnode.jsZeroMQ



Reactor Pattern sweet spot ?



Soft realtime systems where throughput is more

important than processing



What does that mean ?

Lots of in flight I/OLittle CPU per request - proxiesImprove ON:OFF cpu timeNEVER about speed for a single client or request



"Upgraded to Thin, my app's flying!"



EDOINGITWRONG



To maintain acceptable response times for more clients,

with the same or less infrastructure


Client 1

BrokerClient 2

Client 3

Client 4


Example app



FIX

FIX clientFinancial Information eXchangeCompact protocol - encodes a lot of domain info



8=FIX.4.49=4535=049=AB56=CD34=352=20000426-12:05:0610=220

8=FIX.4.4 # FIX version 9=45 # body length 35=0 # Heartbeat msg 49=AB # sender 56=CD # receiver 34=3 # sequence number 52=20000426-12:05:06 # sent at 10=220 # checksum

FIX message



Mediates access to system resources :

I/O, memory and CPU

The kernel



Sandboxed execution for security and stability

requirements

User mode



System calls ( syscalls )

read(5, &buf, 4000)

How do we access resources ?



Syscalls

Kernel

Ruby process

Ruby process

disk

read(5, &buf, 4000)



A protected function call from user to kernel space with the intent to interact with a system resource

Definition



Characteristics

Uniform API on POSIX systemsDitto for behaviorLIES !Slow emulations



Syscall performance

MUCH slower than function calls20x and moreDefinite context switchCalls can block - slow upstream peers



Who doesn't know what file descriptors or file handles

are ?



fd = open(“/path/file”, O_READ)read(fd, &buf, 4000)

open syscall



Examples

local filesocketdirectorypipeConsistent API, semantics differ



Definition

Numeric handleReferences a kernel allocated resourceKernel bufferUser space buffer



A request to read a chunk of data from a descriptor may

block depending onbuffer state

Blocking I/O



Blocked buffer state

Kernel bufferUser buffer

2000 1000

read(5, &b, 4000)



fd = socket (PF_INET, SOCK_STREAM, 0); fcntl (fd, F_SETFL, O_NONBLOCK)

Nonblocking I/O



A read request would only be initiated if the system

call won’t block

Nonblocking I/O



EAGAIN or EWOULDBLOCK



Not a silver bullet

Guideline onlyNot free - invokes a syscallNot supported by all devices



Nonblocking I/O IS NOT asynchronous I/O

Async I/O myth



Async I/O

Windows I/O Completion PortsAIO: POSIX Realtime ExtensionAsync I/O often skip double bufferingSupports file I/O only



Recap

O_NONBLOCK is a guideline onlyInvoked in user spaceData transfer in user spaceBlocking and nonblocking I/O terms



Remember, the primary use case is increased throughput

Reactor Pattern



Challenges

Large number of file descriptorsRespond to descriptor state changesExecute domain logic



WHILE maintaining acceptable response times



while true do # shortened for brevity add_new_descriptors # attach new client descriptors modify_descriptors # update descriptor state break if terminate? # conditional loop breakend

The Reactor Loop



Registered descriptors

Reactor

5

6

78

9

10 R

RR

W

W

W



Who's familiar with select, poll, epoll or kqueue ?

Multiplexed I/O



Multiplexed I/O

Nonblocking I/O is inefficientRetry on EAGAIN is pollingMultiplexed I/O: concurrent blockingNotified of state changes on any registered fd



Multiplexed I/O

MultiplexerI/O bound

App

fd 5

User space Kernel space

fd 6fd 7

fd 8

fd 9fd 10fd 11

fd 12



1. Tell me when fds 1..200 are ready for a read or a

write

Multiplexed I/O



2. Do other work until one of them's ready

Multiplexed I/O



3. Get woken up by the multiplexer on state

changes

Multiplexed I/O



4. Which of fd set 1..200 are in a ready state ?

Multiplexed I/O



5. Handle all fds ready for I/O

Multiplexed I/O



These notifications, or rather state changes in

readiness for reading or writing, that's the Events in

Event Driven I/O

Event driven I/O



module MyFixApp def connection_completed end

def receive_data(buffer) endend

Event handlers / callbacks



Callbacks

Reactor provides I/O concurrencyNO concurrent handling of usercodeFire on the reactor threadRuns within the reactor loop



Ticks

A turn through the event loopThe reactor can do several thousand ticks per secondTime slice for doing workAim to use minimal CPU per tick



Best practices, patterns and gotchas



#1: Coupling Layers

App handler

I/O Multiplexer

Dispatch handler

Kernel



Dispatch Handler

Handles low level events from the multiplexed I/O frameworkeg. readiness for read or writeHide complexities from appsBufferingTransferConversion to and from Ruby strings



class EventableDescriptor def read; end # Tell don't ask interface def write; end def heartbeat; end def select_for_read; end # I/O multiplexer def select_for_write; end # agnosticend

Dispatch Handler



Application Handlers

Domain layerHigher level eventsConnection accepted, read, disconnectedEncode and decode protocolsPerform business logic



class FixConnection def receive_data(data); end # callbacks def connection_completed; end def unbind; endend

Application Handlers



class FixConnection def receive_data(buffer) # Encapsulate excess business logic in callbacks @queue.push *FIX::Parser.parse(buffer) endend

Glue



You don’t need a transport for testing

#2: Testing



class FixConnection def receive_data(data) @queue << data endend

def test_enqueue_on_receive conn.receive_data "stub data" assert_equal 1, conn.queue.sizeend

Testing



Events occur asynchronously, but they're

handled synchronously

#3: Confusing sync and async



App handler

I/O Multiplexer

Dispatch handler

Kernel



class FixConnection def receive_data(buffer) # enqueue to a thread pool EM.defer{ FIX::Parser.parse(buffer) } endend

Defer work to a thread pool



class FixConnection def receive_data(buffer) # allow the reactor to service other clients also EM.schedule{ FIX::Parser.parse(buffer) } endend

Schedule on the reactor thread



#4: Too much ON cpu time

Reactor is single threadedSingle core on SMP systemsPegging the core blocks the event loopTime sharing



[1..n].each do |i| process(i) # expensive workend

Tight loops



work = [1..n]EM.tick_loop do if work.empty? :stop else process(work.shift) # expensive work endend

Prefer a tick loop



Protocol parsers incur a cost for both encoding and decoding from buffers

Slow Protocol parsers



Prefer a fast C extension / parser to native Ruby code

Negate encoding / decoding costs



EM.connect("slow-broker.net", 2000, FixConnection)

#5: Name resolution and connects



d = EM::DNS::Resolver.resolve "slow-broker.net"d.callback do |addrs| # connect to broker API when resolved EM.connect(addrs.first, 2000, FixConnection) end

Async resolver



conn = EM::HttpRequest.new('http://api.a-broker.net') req1 = conn.get :path => “x”, :keepalive => truereq1.callback { req2 = conn.get :path => “y” req2.callback { # same connection as req1 }}

Async HTTP APIs


http://www.igvita.com/'

http://www.igvita.com/'


Not all file descriptors support O_NONBLOCK

#6: Blocking I/O



$ gem install eio

EIO.open(__FILE__) do |fd| EIO.read(fd, 1000) do |buf| p buf EIO.close(fd) endend

http://github.com/methodmissing/eio

Ruby wrapper for libeio





redis = EM::Protocols::Redis.connectredis.errback do |code| puts "Error code: #{code}"endredis.set "a", "foo" do |response| redis.get "a" do |response| puts response endend

#7: All libs have to be evented



client = Mysql2::EM::Client.newdefer = client.query "SELECT sleep(3) as query"defer.callback do |result| puts "Result: #{result.to_a.inspect}"end

Evented MySQL



The Garbage Collector could be invoked at any

time during request processing

#8: Memory



Be careful with sloppy allocation patterns

Minimize GC pressure



module ProxyConnection def initialize(client, request) @client, @request = client, request end def post_init EM::enable_proxy(self, @client) end def connection_completed send_data @request end def proxy_target_unbound close_connection # end # # def unbind # @client.close_connection_after_writing # end # end

Connection level proxy



class FixConnection def receive_data(inbound) # inbound.size can be a single protocol # message or a batch - stream of data end

def send_data(outbound) # enqueued on a write queue endend

#9: Buffer sizes



class FixConnection def receive_data(buffer) # buffer packs multiple FIX messages Parser.parse(buffer).each do |msg| EM.next_tick { process(msg) } end endend

Scheduled parsing



Takeaways

Time sharing for I/O bound servicesSystem call overheads and behaviorBlocking, nonblocking and async I/OSchedule through ticksOS, dispatch and app layersTransport last, interfaces first



Questions ?



Wildfire Interactive, Inc. is hiring



http://wildfireapp.com/buzz/jobsfollow @methodmissing

fork github.com/methodmissing

Thanks !


http://wildfireapp.com/buzz/jobs

http://wildfireapp.com/buzz/jobs

In the Loop - Lone Star Ruby Conference

Technology

Transcript of In the Loop - Lone Star Ruby Conference