OpenAFS in a multihomed universe

Jeffrey Altman

Derrick Brashear

Topics

This is really 2 discussions:• Configuring and using multi-homed clients.

• Configuring and using multi-homed servers. It’s important to understand how we got

here. Some issues are still “on the table”.

Background

When AFS development began, multi-homed hosts were uncommon.

Assumptions made early did not accommodate multi-homed hosts.

As commercial (Transarc) customers began to require multi-homed support, changes were made over time to parts of AFS.

• Some offered better functionality than others.

• Full integration wasn’t always possible.

Architecture (abridged)

VLserver• Used to discover volume locations.

• Uses ubik distributed database technology to provide current copies of data, provided a fully-replicated service.

Fileserver• Registers its address with the VLserver.

• Used to provide data to clients.

• Must track clients to provide callback notifications.

Client• Uses VLserver to discover data location, then fetch it from

fileservers on behalf of users.

Issues: VLserver

How do clients find the VLserver in the first place?• Hardcoded address in CellServDB.

• IBM never solved this; you had to maintain copies of the file if you needed to address some or all database servers by different addresses on different groups of clients.

Solutions: VLserver

AFSDB

• Support for AFSDB DNS records added by OpenAFS volunteers.

• Use of split-horizon DNS allows different information to be distributed to different groups of clients, but still be managed in few places.

Issues: fileserver

Original VL RPCs returned a location which was server and partition.• Server meant “one IP address”.

If you had a different address space than the one in which the VLserver was providing answers, you were sad.

Solutions: fileserver

UUID• Fileservers generate a “universally-unique

identifier” which can be used in place of a single address.

• Included in the data structure with the actual UUID number is a list of addresses the fileserver registered with the VLserver.

• Revised VL RPCs use UUID instead of address; New clients and servers use the new RPCs when available.

Issues: client

The client itself can have more than one address.• Fileserver tracking of client by IP address only.

• A fixed client port of 7001 was assumed in some cases.

Worse, the client can have an alternate port!

Solutions: client

Clients also get a UUID.• Fileserver callback interfaces changed to

support UUIDs rather than addresses.

• Port assumptions removed.

11

Complications

It’s not that simple. NAT

• A client can be behind a NAT and not know its real address, precluding it from telling the server its actual address.

• A server can be behind a NAT! You may not want to advertise all of your

addresses!

11

12

Enter NetInfo and NetRestrict

In order to only publish some addresses, or mask some addresses, a new interface was created, for both clients and servers.

NetInfo is a list of addresses to publish. NetRestrict is a list of addresses to

mask. Typically you use one or the other.

13

NetInfo and NAT

Remember servers can also be behind NAT!• NetInfo supports fake addresses.

• The only gotcha here is that you need to know the “other” address of your server.• If it changes, you need to update NetInfo and restart.

For clients, it matters less.

13

Client addresses

Who can you trust?•Clients behind a NAT will not know their “real”

external address.

•Advertised internal addresses may be non-routable.

Fileserver updated to use the calling address and not use addresses advertised to it without first verifying they are the same host.

Forget your problems?

Just because a host has multiple addresses doesn’t mean you want to use them all.

Rx RPC library built on a single socket model.

• Allow socket to be bound to exactly one address. (“rxbind”)• Requires list of addresses as modified by NetInfo

and/or NetRestrict to be whittled to exactly one address to work.

Or at least make them manageable

Clients support the notion of server preferences.

• Server preferences are by address, not UUID!

• Preference can be given to a particular address for a server.• Sadly, the defaults are based on classful networking,

which is not as useful as it could be.

Except when you can’t

Ubik databases will have one primary address.• True multi-homing is not supported in the

internal ubik interfaces.

• Related, special powers are assigned always to the member of a ubik service pool with the lowest IP address.

What’s still needed

Truly multi-homed Ubik.A way to change what addresses a

fileserver is advertising, on the fly.A way to discover what addresses those

should be other than manual configuration.Server preference support in clients which

is not based on classful networks.Multi-socket Rx.

• Or maybe just finish Rx TCP and IPv6 support.

OpenAFS in a multi-homed universe

Questions?