OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

19
OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear

Transcript of OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Page 1: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

OpenAFS in a multihomed universe

Jeffrey Altman

Derrick Brashear

Page 2: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Topics

This is really 2 discussions:• Configuring and using multi-homed clients.

• Configuring and using multi-homed servers. It’s important to understand how we got

here. Some issues are still “on the table”.

Page 3: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Background

When AFS development began, multi-homed hosts were uncommon.

Assumptions made early did not accommodate multi-homed hosts.

As commercial (Transarc) customers began to require multi-homed support, changes were made over time to parts of AFS.

• Some offered better functionality than others.

• Full integration wasn’t always possible.

Page 4: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Architecture (abridged)

VLserver• Used to discover volume locations.

• Uses ubik distributed database technology to provide current copies of data, provided a fully-replicated service.

Fileserver• Registers its address with the VLserver.

• Used to provide data to clients.

• Must track clients to provide callback notifications.

Client• Uses VLserver to discover data location, then fetch it from

fileservers on behalf of users.

Page 5: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Issues: VLserver

How do clients find the VLserver in the first place?• Hardcoded address in CellServDB.

• IBM never solved this; you had to maintain copies of the file if you needed to address some or all database servers by different addresses on different groups of clients.

Page 6: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Solutions: VLserver

AFSDB

• Support for AFSDB DNS records added by OpenAFS volunteers.

• Use of split-horizon DNS allows different information to be distributed to different groups of clients, but still be managed in few places.

Page 7: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Issues: fileserver

Original VL RPCs returned a location which was server and partition.• Server meant “one IP address”.

If you had a different address space than the one in which the VLserver was providing answers, you were sad.

Page 8: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Solutions: fileserver

UUID• Fileservers generate a “universally-unique

identifier” which can be used in place of a single address.

• Included in the data structure with the actual UUID number is a list of addresses the fileserver registered with the VLserver.

• Revised VL RPCs use UUID instead of address; New clients and servers use the new RPCs when available.

Page 9: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Issues: client

The client itself can have more than one address.• Fileserver tracking of client by IP address only.

• A fixed client port of 7001 was assumed in some cases.

Worse, the client can have an alternate port!

Page 10: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Solutions: client

Clients also get a UUID.• Fileserver callback interfaces changed to

support UUIDs rather than addresses.

• Port assumptions removed.

Page 11: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

11

Complications

It’s not that simple. NAT

• A client can be behind a NAT and not know its real address, precluding it from telling the server its actual address.

• A server can be behind a NAT! You may not want to advertise all of your

addresses!

11

Page 12: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

12

Enter NetInfo and NetRestrict

In order to only publish some addresses, or mask some addresses, a new interface was created, for both clients and servers.

NetInfo is a list of addresses to publish. NetRestrict is a list of addresses to

mask. Typically you use one or the other.

Page 13: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

13

NetInfo and NAT

Remember servers can also be behind NAT!• NetInfo supports fake addresses.

• The only gotcha here is that you need to know the “other” address of your server.• If it changes, you need to update NetInfo and restart.

For clients, it matters less.

13

Page 14: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Client addresses

Who can you trust?•Clients behind a NAT will not know their “real”

external address.

•Advertised internal addresses may be non-routable.

Fileserver updated to use the calling address and not use addresses advertised to it without first verifying they are the same host.

Page 15: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Forget your problems?

Just because a host has multiple addresses doesn’t mean you want to use them all.

Rx RPC library built on a single socket model.

• Allow socket to be bound to exactly one address. (“rxbind”)• Requires list of addresses as modified by NetInfo

and/or NetRestrict to be whittled to exactly one address to work.

Page 16: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Or at least make them manageable

Clients support the notion of server preferences.

• Server preferences are by address, not UUID!

• Preference can be given to a particular address for a server.• Sadly, the defaults are based on classful networking,

which is not as useful as it could be.

Page 17: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

Except when you can’t

Ubik databases will have one primary address.• True multi-homing is not supported in the

internal ubik interfaces.

• Related, special powers are assigned always to the member of a ubik service pool with the lowest IP address.

Page 18: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

What’s still needed

Truly multi-homed Ubik.A way to change what addresses a

fileserver is advertising, on the fly.A way to discover what addresses those

should be other than manual configuration.Server preference support in clients which

is not based on classful networks.Multi-socket Rx.

• Or maybe just finish Rx TCP and IPv6 support.

Page 19: OpenAFS in a multihomed universe Jeffrey Altman Derrick Brashear.

OpenAFS in a multi-homed universe

Questions?