Remote Procedure Remote Procedure CallCall

An Effective Primitive for An Effective Primitive for Distributed ComputingDistributed Computing

Seth James NielsonSeth James Nielson

What is RPC?What is RPC? Procedure calls Procedure calls

transfer control transfer control within local within local memorymemory

RPC’s transfer RPC’s transfer control to remote control to remote machinesmachines

Unused

Proc B

Proc A

Main

Why RPC?Why RPC?

Clean/Simple semanticsClean/Simple semantics Communication efficiencyCommunication efficiency GeneralityGenerality

RPC is an effective primitive for distributed systems because of -

How it WorksHow it Works(Idealized Example)(Idealized Example)

localCall()……

c = encrypt(msg)

CLIENT SERVERWith specialized hardware

and encryption key

wait…c = encrypt(msg)

localCall() … …

encrypt(msg)Implementation

Request

Response

Early History of RPCEarly History of RPC 1976: 1976: early reference in literatureearly reference in literature 1976-1984: 1976-1984: few full few full

implementationsimplementations Feb 1984:Feb 1984: Cedar RPCCedar RPC

– A. Birrell, B. Nelson at XeroxA. Birrell, B. Nelson at Xerox– ““Implementing Remote Procedure Implementing Remote Procedure

Calls” Calls”

Imagine our Surprise…Imagine our Surprise…

““In practice, … In practice, … several areas [of several areas [of RPC] were RPC] were inadequately inadequately understood”understood”

RPC Design IssuesRPC Design Issues

1.1. Machine/communication failuresMachine/communication failures2.2. Address-containing argumentsAddress-containing arguments3.3. Integration into existing systemsIntegration into existing systems4.4. BindingBinding5.5. Suitable protocolsSuitable protocols6.6. Data integrity/securityData integrity/security

Birrell and Nelson Birrell and Nelson AimsAims

Primary AimPrimary Aim– Easy distributed computationEasy distributed computation

Secondary AimsSecondary Aims– Efficient (with powerful semantics)Efficient (with powerful semantics)– SecureSecure

Fundamental DecisionsFundamental Decisions

1.1. No shared address space among No shared address space among computerscomputers

2.2. Semantics of remote procedure Semantics of remote procedure calls should be as close as calls should be as close as possible to local procedure callspossible to local procedure calls

Note that the first decision partially Note that the first decision partially violates the second…violates the second…

BindingBinding

Binds an importer to exporterBinds an importer to exporter Interface name:Interface name: type type//instanceinstance Uses Uses GrapevineGrapevine DB to locate DB to locate

appropriate exporterappropriate exporter Bindings (based on unique ID) break Bindings (based on unique ID) break

if exporter crashes and restartsif exporter crashes and restarts

Unique IDUnique ID At binding, importer learns of At binding, importer learns of

exported interface’s exported interface’s Unique ID Unique ID (UID)(UID)

The UID is initialized by a real-time The UID is initialized by a real-time clock on system start-upclock on system start-up

If the system crashes and restarts, If the system crashes and restarts, the UID will be a new unique numberthe UID will be a new unique number

The change in UID breaks existing The change in UID breaks existing connectionsconnections

How Cedar RPC worksHow Cedar RPC worksCaller Machine Grapevine Callee Machine

User User Stub RPCRun. RPCRun. Server Stub Server

record export exportupdate setConnect

update addmember

import import getConnect lookup

bind(A,B) lookup

return record

return

x=F(y) F=>3 transmit Check 3 3=>F F(y)

Packet-Level Transport Packet-Level Transport ProtocolProtocol

Primary goalPrimary goal: minimize time : minimize time between initiating the call and between initiating the call and getting results getting results

NOT general – designed for RPCNOT general – designed for RPC Why? possible 10X performance gainWhy? possible 10X performance gain No upper bound on waiting for results No upper bound on waiting for results Error Semantics: User does not know Error Semantics: User does not know

if machine crashed or network failedif machine crashed or network failed

Creating RPC-enabled Creating RPC-enabled SoftwareSoftwareUser Code

Server Code

InterfaceModules

Deve

lope

rLu

pine

User Stub

Server Stub

RPCRuntime

ServerProgram

RPCRuntime

ClientProgram

Client Machine

Server Machine

Making it FasterMaking it Faster Simple Calls (common case): all of Simple Calls (common case): all of

the arguments fit in a single the arguments fit in a single packetpacket

A server reply and a 2A server reply and a 2ndnd RPC RPC operates as an implicit ACKoperates as an implicit ACK

Explicit ACKs required if call lasts Explicit ACKs required if call lasts longer or there is a longer interval longer or there is a longer interval between callsbetween calls

Simple CallsSimple Calls

CLIENT SERVERCall

Response/ACK

Call/ACK

Response/ACK

Complex CallsComplex Calls

CLIENT SERVERCall (pkt 0)

ACK pkt 0

Data (pkt 2)

Response/ACK

Data (pkt 1)

ACK pkt 1

ACK or New Call

Keeping it LightKeeping it Light A A connectionconnection is just is just shared stateshared state Reduce process creation/swappingReduce process creation/swapping

– Maintain idle Maintain idle server processesserver processes– Each packet has a process identifier Each packet has a process identifier

to reduce swapto reduce swap– Full scheme results in no processes Full scheme results in no processes

created/four process swaps per callcreated/four process swaps per call RPC directly on top of EthernetRPC directly on top of Ethernet

Elapsed Time Elapsed Time PerformancePerformance

Number of Number of Args/ResultsArgs/Results

TimeTime

00 10971097µµ100100 12781278µµ

100 word array100 word array 29262926µµ

THE NEED THE NEED FOR SPEEDFOR SPEED

RPC performance cost is a barrier RPC performance cost is a barrier (Cedar RPC requires .1 sec for a 0 arg (Cedar RPC requires .1 sec for a 0 arg call!)call!)

Peregrine RPC (about nine years later) Peregrine RPC (about nine years later) manages a 0 arg call in .0573 seconds!manages a 0 arg call in .0573 seconds!

A Few DefinitionsA Few Definitions Hardware latencyHardware latency – Sum of – Sum of

call/result network penaltycall/result network penalty Network penalty – Network penalty – Time to Time to

transmit (greater than…)transmit (greater than…) Network transmission time Network transmission time

– – Raw Network SpeedRaw Network Speed Network RPCNetwork RPC – RPC – RPC

between two machinesbetween two machines Local RPC Local RPC – RPC between – RPC between

separate threadsseparate threads

Peregrine RPCPeregrine RPC Supports full functionality of RPCSupports full functionality of RPC Network RPC performance close Network RPC performance close

to HW latencyto HW latency Also supports efficient local RPCAlso supports efficient local RPC

Messing with the GutsMessing with the Guts

Three General OptimizationsThree General Optimizations Three RPC-Specific OptimizationsThree RPC-Specific Optimizations

General OptimizationGeneral Optimization

1.1. Transmitted arguments avoid Transmitted arguments avoid copiescopies

2.2. No conversion for client/server No conversion for client/server with the same data with the same data representationrepresentation

3.3. Use of packet header templates Use of packet header templates that avoid recomputation per callthat avoid recomputation per call

RPC Specific RPC Specific OptimizationsOptimizations

1.1. No thread-specific state is saved between No thread-specific state is saved between calls in the servercalls in the server

2.2. Server arguments are mapped (not copied)Server arguments are mapped (not copied)3.3. No copying in the critical path of multi-No copying in the critical path of multi-

packet argumentspacket arguments

I think this is I think this is COOLCOOL To avoid copying arguments from To avoid copying arguments from

a single-packet RPC, Peregrine a single-packet RPC, Peregrine arranges instead to use the arranges instead to use the packet buffer itself packet buffer itself as the server as the server thread’s stackthread’s stack

Any pointers are replaced with Any pointers are replaced with server-appropriate pointers server-appropriate pointers (Cedar RPC didn’t support this…)(Cedar RPC didn’t support this…)

This is cool tooThis is cool too

Multi-packet RPC’s use Multi-packet RPC’s use blastblast protocol protocol (selective retransmission)(selective retransmission)

Data is transmitted in parallel with Data is transmitted in parallel with data copydata copy

Last packet is mapped into placeLast packet is mapped into place

Data 0

Data 3

Data 1

Data 2 Data 1

Data 2

Data 0

Data 3Header0

Header 0

Header3

Header2

Header1

PageBoundary

Packets 1-3 dataare copied into bufferat server

Packet 0 buffer (sent last)Is remapped at server

Fast Multi-Packet Fast Multi-Packet ReceiveReceive

Peregrine 0-Arg Peregrine 0-Arg PerformancePerformance

SystemSystem LatencyLatency ThroughpuThroughputt

CedarCedar 10971097µsecµsec 2.0mbps2.0mbpsAmoeba**Amoeba** 11001100µsecµsec 6.4mbps6.4mbpsx-kernelx-kernel 17301730µsecµsec 7.1mbps7.1mbpsV-SystemV-System 25402540µsecµsec 4.4mbps4.4mbpsFirefly (5 Firefly (5 CPU)CPU)

26602660µsecµsec 4.6mbps4.6mbps

SpriteSprite 28002800µsecµsec 5.7mbps5.7mbpsFirefly (1 Firefly (1 CPU)CPU)

48004800µsecµsec 2.5mbps2.5mbps

SunRPC**SunRPC** 67006700µsecµsec 2.7mbps2.7mbpsPeregrinePeregrine 573573µsecµsec 8.9mbps8.9mbps

Peregrine Multi-Packet Peregrine Multi-Packet PerformancePerformance

ProcedureProcedure(Bytes)(Bytes)

Network Network Penalty Penalty

(ms)(ms)

LatencLatencyy

(ms)(ms)

ThrougThroughputhput

(mbps)(mbps)3000 byte in RPC3000 byte in RPC 2.712.71 3.203.20 7.507.50

3000 byte in-out 3000 byte in-out RPCRPC

5.165.16 6.046.04 7.957.95

48000 byte in RPC48000 byte in RPC 40.9640.96 43.3343.33 8.868.86

48000 byte in-out 48000 byte in-out RPCRPC

81.6681.66 86.2986.29 8.908.90

Cedar RPC SummaryCedar RPC Summary Cedar RPC introduced practical Cedar RPC introduced practical

RPCRPC Demonstrated easy semanticsDemonstrated easy semantics Identified major design issuesIdentified major design issues Established RPC as effective Established RPC as effective

primitiveprimitive

Peregrine RPC Peregrine RPC SummarySummary

Same RPC semantics (with Same RPC semantics (with addition of pointers)addition of pointers)

Significantly faster than Cedar Significantly faster than Cedar RPC and othersRPC and others

General optimizations (e.g., pre-General optimizations (e.g., pre-computed headers)computed headers)

RPC-Specific (e.g., no copying in RPC-Specific (e.g., no copying in multipacket critical path)multipacket critical path)

ObservationsObservations RPC is a very “transparent” RPC is a very “transparent”

mechanism – it acts like a local callmechanism – it acts like a local call However, RPC requires a deep However, RPC requires a deep

understanding of hardware to tuneunderstanding of hardware to tune In short, In short, RPC requires sophistication RPC requires sophistication

in its presentation as well as its in its presentation as well as its operation to be viableoperation to be viable