Cs 704 d rpc

CS 704DAdvanced Operating System

(RPC)

Debasis Das

2

Remote ProcedureCalls(RPC)

3

Remote Procedure Calls• A special case for building distributed applications• RPC became popular IPC mechanisms for distributed applications

o Simple call syntaxo Familiar semantics, similar to local procedure calls\o Well defined interface, compile time type checking, auto interface

generationo Ease of use, clean & Simpleo Generality & efficiencyo Same IPC mechanism can be used for procedure on different

machines as well as on the same machine

11/21/2009 IT703D: Dirstibuted Computing Debasis Das

4

The RPC Model• Based on procedure call model

o Make a procedure call, place parameters in a known location

o Transfer control to the procedureo Procedure executes with copies of the

parameters/argumentso Control is returned to calling procedure with result, if

any

11/21/2009 IT703D: Distributed Computing Debasis Das


Server Process

Request message & parametersSend reply with result, if any

CallerClient Process

RPC Model(Graphically)

6

RPC Characteristics• Server process is normally dormant. Works when

called• If concurrency required, calling processes can leave

messages asynchronously, server process keeps serving requests

• Server process can have threads. Starts a thread on request, server is free to field further calls if they come along


7

Transparency of RPC

• For a programmer it should be transparent whether

he is calling a local or a remote procedure. Requires

o Syntactic transparency

o Semantic transparency

Syntactic transparency is easy to achieve but not

semantic transparency11/21/2009 IT703D: Distributed Computing Debasis Das

8

Semantic Transparency• Nearly impossible to achieve, because

o No shared memory: passing addresses as arguments is meaningless, call by reference pointers are useless, structures referenced by pointers are difficult.

o More vulnerable to failures: two different processes, two different machines, intervening network. Processor and communication failures need to be taken care of.

o RPCs can take much more time(x100 to x1000) due to communication congestions. Long delays need to be handled.



Unpack Pack

Receive Send

PackUnpack

Receive Send

ClientReturn Call Call Execute Return

Server

Wait

Stubs

RPC Runtimes

Implementing RPC Mechanism

10

Stubs• Client stub

o On call request, packs specs of remote procedure and arguments

o Asks RPC Runtime to transmit the message to the server

• Server stubo After receipt, the RPC Runtime passes the message

to it. It then unpacks the message and makes a usual call to the procedure.

o On receipt of result, packs the result and asks the runtime to send it back to the client.


11

Stub & Transparency• Except for the stubs everything operates the same

transparent way for local or remote procedure call.• No need to dabble in send & receive primitives


Stub Generation• Manually

o Write code to implement translation functions provided by the RPC implementer

• Automaticallyo By using IDL. Procedures supported by the interface,

types of arguments and resultso IDL compiler will ensure compilation, type checking

etc.o Optimizes data to be sent over the network. Interface

definition usually will have information as to which arguments are input/output/both.


Automatic Stub Generation, IDL etc.

• Interface export: server that implements the procedures that can be used by others

• Interface import: calls procedures from the interface• Define an interface using the IDL, write client

program that imports the interface and server program that exports the interface

• IDL compilers generate client stub procedure, server stub procedure, header files. Compile & link the stubs.

• Interface code thus generated could be combined with client/server procedures done in different languages11/21/2009 IT703D: Distributed Computing Debasis Das

RPC Messages• Call messages

o Sent by client to server with request to execute a remote procedure

• Reply messageso Sent by server to the requesting client to indicate

completion of a request and related results if generated

• The messages follow RPC protocols and has nothing to do with network transport protocol


RPC Call Message• Basic fields of the message

o Id information for the remote procedure to be executedo Required arguments to execute the procedure

• Additional fieldso Message id field, essentially a sequence numbero Message type field identifying if a call/reply messageo Client id field, identifying the requesting client and for

authentication



Clientid

Messageid

Messagetype

Programnumber

Versionnumber

Procedurenumber

Arguments

Remote Procedure id

RPC Call Message Structure

RPC Reply Message• Call message not understood, call rejected• Client id not authorized, failure message• Procedure id not available on server, failure

message• Parameters not successfully decoded, failure

message• Exception(e.g. divide by zero) happens, failure

message• Procedure executes successfully, return result(s)



Messageid

Messageid

Messagetype

ReplyStatusfailure

Failurereason

Messagetype

ReplyStatus

success

Result(s)

RPC Reply Message Structure

19

Marshaling Arguments & Results• Data structures to be converted into stream form

and results decoded back• This is the marshalling process and involves

o Take the arguments/resulto Encoding of data on sender’s machineo Decoding of message data on receiver’s machineo Representation method(tagged or untagged) should

be known, type safety can be ensuredo Must reflect structure of all types; primitive types,

structured types and user defined types


Marshalling Procedure Types1.Provided as part of the RPC support. Procedures for

scalar types and compound ones built from Scalar types are included

2.Defined by users of RPC systems. Includes marshalling procedures for user defined data types


Server Management• Server Implementation

o Stateful serverso Stateless serverso Why stateless servers

• Server creation semanticso Instance-per-call serverso Instance-per-session serverso Persistent servers


Stateful Servers• When the server maintains state information from a

previous call to the server by the specific client• For example a Stateful file server will remember that

a particular client opened a file(given file id) previously and will let consecutive reads /writes and other operations on that file to be done properly

• Two consecutive reads such as Read(fid,100,buf) will read two consecutive 100 bytes of data from the open file, opened in an earlier procedure call.


Stateless Servers• When the server does not maintain any state

information• It is the responsibility of the server to ensure the

operations are carried out correctly• For a similar example; you may have to open the

file, seek to the correct location and then read to have a Read correctly.

• Read(filename, position, bytes, buffer) or Write(filename, position, bytes, buffer)


Why Stateless Servers?• Stateful servers allow easier programming and

typically they are more efficient, so why bother with a stateless server?

• Stateless servers protect against failureso Server crashes, state information is losto Client may be unaware of crash & restart creating

inconsistent results.o Client crashes and restarts, server has useless state

information that cannot be easily rolled back


Server Creation Semantics• Servers are independent processes compared to

client serverso Independent lifetimeso Independent locationso Independent address spaces

• Servers may be created prior to the client processes or on demand


Instance-per-call Servers• Server is created by RPC runtime when a call for it

arrives• Deleted when the call has been serviced• Not a very common method because

o Will have to be a stateless server and state information to be maintained by the client process or the OS. OS involvement is expensive.


Instance-per-session Servers• Server managers for each type of service• Server managers are registered with a binding agent• When a client needs a service, contacts binding agent• Binding agent sends address of server manager• Client sends request to server manager, who spawns a

server and sends its address• Client interacts with server until it no longer requires the

server• Server manager destroys the server

What should be type of server; stateful or stateless?


Persistent Servers• Indefinite lifetime• Shared by more than one client• Created and installed prior to clients• Servers register with binding agent• Client request binding agent for a server• Binding agent selects one randomly or as per some

policy• Client requests are interleaved and several sets of state

are maintained• Multiple servers improves load sharing/failure tolerance


Parameter Passing Semantics• Call by value

o Send value of arguments along with invocation message

o Can be complicated and wasteful for complex data structures

• Call by referenceo Does not make sense in separate address spaceso Systems with shared distributed memory would be oko Call by object reference

Call by move Call by visit


Call Semantics• Why are they needed?

o Call message can get losto Reply message can get losto Caller node can crash and get restartedo Callee node can crash and get restarted

• The issue is to use call semantics that can protect against this

• Application procedure is obviously not the place to handle these issues

• Best place would be the RPC runtime system


Call Semantics• Possibly or May-be call semantics• Last one call semantics• Last-of many call semantics• At-least-once call semantics• Exactly-once call semantics


Possibly or May be• Call and then wait for a time-out then continue• Call ack. or results are not guaranteed• Weakest semantic, may be ok for situations where

no response is really required.• May be within a LAN which has high probability of

transmission success


Last One• Call, wait for time-out, resend if result not received• Repeat until one call succeeds• This can work well only when only two nodes on

which the processes work are involved.o Nested calls to different nodes will create problemso Orphan processes create problemso Let the orphans complete or exterminate them


Last of Many• Similar to Last One semantics• Ignore orphans• Attach call ids to calls, response returns the same id• Calls repeated has new id• Ignore response if it does not match the latest call id


At least Once• Guarantees the call is executed at least once but

does not specify which results are returned• Implemented by the time-out based method• Takes the result of the first response without

bothering if the result is from an orphan


Exactly Once• With the earlier weak semantics server may get

executed more than once• Programmers will have to ensure indempotency by

creating appropriate interfaces• Whereas if the callee executes exactly once,

indempotency can be guaranteed• Implementation uses timeouts, re-transmissions, call

id, same id for repeats and reply cache for callee


Communication Protocols for RPCs

• Request protocol• Request/Reply protocol• Request/reply/Acknowledge-Reply protocol


Request Protocol( R Protocol)

• Caller process send a message and continues• Callee process does not send any reply• Also called an asynchronous protocol• Improves client and server performance• Example, remote X-window system


R Protocol with Unreliable Delivery• Asynchronous protocol with unreliable delivery

would mean loss of request• RPC runtime does not take responsibility of

retransmission• Some applications may still manage with this simple

protocol and unreliable delivery• Time update for example• Receivers may request specific update after fixed

period if connection lost for a long time


Request/Reply Protocol(RR Protocol)

• Can be used for simple RPCso A simple protocol is one I which the request,

arguments, results can all fit into a single packeto Duration of call and interval between calls are small,

smaller than transmission time of the packet• Implicit acknowledgements used

o A reply is considered as an ack. To the earlier requesto A request is considered as ack. For earlier reply

• Timeout & retires used for failure handling


RR with Retransmission Implications

• With duplicate requests hanging around it is like at least once semantics on client side (duplicate requests are not filtered out)

• Exactly once semantics on server possible with reply cache


Request/Reply/Acknowledge-Reply Protocol

(RRA Protocol)

• RR protocol, exactly once semantics requires storage of a lot of information in the server cache

• Request, reply followed by a reply ack. Completes the cycle

• The reply ack. Itself may get lost• Use sequence numbers, the ack. carries the

sequence number original request, as does the reply message


Complicated RPCs• RPCs involving long-duration calls or large gaps

between calls• RPCs involving arguments and/or results that

cannot be fitted into one datagram


Long Duration/Large Gap Calls• Periodic probing of the server by the client

o Request message to server, periodic probes with the same request number, server generates ack.

o If original request was not received, server says so and client is intimated of failure

• Periodic generation of acknowledgments by server o Server generates ack. Prior to timeout. So the client

knows the server is working on request.o If ack. or reply not received within timeout, failure of

node or comm. is assumed and user informed.


Too Large Arguments/Results• Example file read /write• Use multiple physical RPC for each logical RPC.

Each physical RPC handles data for one datagramo Inefficient due to fixed overheads with each physical

RPC• Use multi datagram messages, handle missing or

out of sequence datagrams as discussed in IPC


Client server Binding• Client must know address of server before a remote

call can be made• The process of associating a client & a server is

binding• Server “exports” services it is willing to undertake• Client imports these services via RPC runtime


Binding Issues• How does a client specify the server to which it

needs to be bound?• How does the binding process locate the required

server?• When is it proper to bind a client to a server?• Can the client change the binding during execution?• Can a client get bound to multiple server with similar

service?


Server Naming• Interface name: type, instance• Type includes a version number• When client is not bothered, it does not specify an

instance• User decides semantics of interfaces, RPC package

has no contribution

• Example, file_server is a type, which may be provided by many instances


Server Locating• When needed a serve has to be located first

o Broadcasting Client node broadcasts request to all First response received from a server is given the client

processo Binding Agent

Maintains a mapping table of interface name and location information

Servers register themselves, may deregister also Identification information and a handle Binding agent check periodically and deregister non

responding servers



Primitives used: register, deregister and lookup

Binding Agent

ServerProcess

ClientProcess

12

3

4

1. Server registers2. Client request for server3. Server information4. Client calls the server

Binding Process

Binding Time• Binding at compile time

o Server address is compiled into code at compile timeo Inflexible

• Binding at link timeo Server registers with binding agento Client requests import of service, binding agent return

handleo Handle retained by client for subsequent calls

• Binding at call timeo By indirect call



BindingAgent

ServerProcess

ClientProcess

1

23

4

5

1. Client passes server interface namename & arguments2. Agent does RPC call to server3.Server returns result4. Agent returns result & handle5.Client makes subsequent calls directly

Call Time Binding(Indirect call)

Changing Bindings• A client or a server may want to change binding• Client may want to change binding to another server

if the earlier call failed• Server may want to change binding if it has to

change to another node or upgrade to another version

• When a server wants to change binding, the current state data may have to be transferred to the alternate server


Multiple Simultaneous Binding• Useful in situations where multiple servers must act

on a request• For example update of multiple copies of a file at

different nodes• It is advantageous then to bind the client to multiple

servers• Bind the client to all the servers that has copies of

the file at different nodes


Exception Handling• In case of errors they need to be reported to a client• Define exception conditions and raise exceptions• Can be handled by programming languages that

have exception handling constructs• Alternatively(like in Unix) return a value -1 to

indicate error and a global variable errno that contains a number indicating error type( a legitimate result could be -1)


Security• Security may be needed

o Considerations are Is authentication of server required by the client? Is authentication of server required when result is

returned? Is it ok if results and arguments of RPC are required or

they could be shared by other users? (Other than caller & callee)


Special RPCs• Callback RPC

o Interactive programs may need several client actions before completing the original call

• Broadcast RPCo Client request broadcast on network, several server may respond to the

request• Batch Mode RPC

o Send a batch of RPC calls. High call rate(50-100 remote calls per second) applications can use this


To make Call back work• For call back facility to work

o Provide server with client’s handleo Making client process wait for callback RPCo Handling callback deadlock


Provide Server with Client’s Handle

• Client process uses a transient program number for the callback service and exports service by registering with the binding agent

• Program number sent to server as part of RPC request

• Server then can initiate a RPC to client• A handle also may be sent so that further calls can

be made directly


Making Client Process wait for Callback RPC

• Client has to wait for callbacks specifically so that callbacks are not treated as a reply of an earlier call

• To do that the client usually calls a svc-routine. The svc-routine waits for a callback and intimates client when a callback is received



Process P1

Process P2 Process P3

Waiting for replyFrom P1

Waiting for Reply from P3

Waiting for replyFrom P2

Handling Callback Deadlocks

Broadcast RPC• 1.Use a broadcast primitive to indicate it is a

broadcast call to binding agent• Agent sends it to servers registered for that service• The broadcast call will find only the ones registered

with the binding agento 2. Get a binding for broadcast porto Broadcast a RPCo o/one or m out of n may be adoptedo Servers with successful results respond, ones with

error may remain silent


Batch Mode RPC• Send a batch of requests in one RPC call• Reduces overheads in sending individual calls• Clients with high call rates can use it, needs reliable transport• Entire queue requests are sent to server when

o A predetermined interval elapseso Predetermined requests have been queuedo Amount of batched data exceeds buffer sizeo A call is made which needs a reply, one must be able to

distinguish between the batch calls and non-queueing RPC request


RPCs in Heterogeneous Environment

• Data representationo Participation machines may have different data

representation• Transport protocol

o RPC mechanism should be transparent of network protocol

• Control protocolo Should be transparent of network control protocols Delay the decision. Binding agent sends the necessary data

conversion and protocol routines to the client stub at binding time


Lightweight RPCs• Were developed for micro kernel OS architectures• Needs communications of two types

o Cross domaino Cross machine

• LPRC is about cross domain communication, techniques usedo Simple control transfero Simple data transfero Simple stubso Design for concurrency


Simple Control Transfer• Special threads scheduling: hands off scheduling• When calling, client supplies an argument stack and

the thread• Trap to kernel• Kernel validates caller, creates call linkage• Server starts executing the thread• When complete send the results and control back


Simple Data Transfer• Use a shared argument stack• One copy: copy from client stack to the shared

argument stack

• In contrast regular RPC has to do 4 copieso Client stack to RPC message buffero Message in client domain to kernel domaino Kernel domain to server domaino Server domain buffer to server stack


Simple Stubs• End to end, as per programming languages &

architecture• Stub to stub implemented by the stubs themselves• Domain to domain implemented by the kernel


Design for Concurrency• When a node has multiple processors

o Shared data structures minimized to reduce lock contention on domain transfer path

o Reduce context switching latency by caching domains on idle processors

LRPC increases throughput by a factor of three!


Optimizations for Performance• Concurrent access to multiple servers• Serving multiple requests simultaneously• Reducing per-call workload of servers• Reply caching of indempotent remote procedures• Proper selection of time out values• Proper design of RPC protocol specification


Concurrent access to multiple servers

• Use of threads, each thread can call a separate servero Addressing in the underlying protocol needs to be rich enough to provide

correct routing of results• Early reply method

o Split call, client sends parameters, server sends a tago Client call again with tag when convenient to get resulto Problem is server will have hold on to the result until required

• Call buffering (via a call buffer server)o Sends call with parameters to call buffer servero Gets the result when it is ready

• A variation of that methodo Use a variable promise(can be blocked or ready) operation ready and

claim on that


Serving multiple requests Simultaneously

• Common types of delayo Server may have to wait for a shared resourceo Server calls a remote process that takes time or

involves transmission delays• While waiting, server should be able to service other

callso Implement threads


Reducing per call Workload of Servers

• Server performance gets affected when there are too many calls and/or too much to be done per callo One way to reduce the load is to use stateless

servers. Clients can keep track of the state. It is responsible for the processing anyway


Reply caching of indempotent Remote Procedures

• As requests build up at a server, some calls will fail, prompting the client to send retransmissions making the problem worse.

• A cache helps


Proper Selection of Timeout Values

• Too small, too many retransmissions• Too large, reduction in throughput• Using retransmission time back off method is one

solution


Proper Design of RPC Design Specifications

• Minimize data to be sento Reduces encoding/decoding timeo Reduces transfer time

• One could use an existing general purpose protocol; in fact many RPC systems use TCP/IP and UDP/IP

• General purpose protocols may be wasteful• For example RPC does not need the checksum

field. It has to filled and takes computation time


Cs 704 d rpc

Documents

Transcript of Cs 704 d rpc