1

Setting up a Testbed with theNetbed/Emulab Software

2

Detailed documentation of all this is in the source tree:

doc/setup*.txt

3

Steps

• Get network infrastructure set up

• Install/configure FreeBSD on boss and ops– Linux should work, but never been done. Glitches will

occur, most likely in standard services, not Emulab itself. Rutgers Winlab about to tackle it.

• Run installation scripts on boss and ops

• Put information about the hardware into the database

• Bring up nodes

4

5

Control Network VLANs

• External – connection to campus network

• Node control – nodes’ control interfaces

• Private – boss and other ‘secure’ nodes

• Public – ops and other nodes with user accounts

• Control hardware – SNMP devices that must be protected from users

6

Switch (Cisco) Setup

• Enable SNMP read-write access• Control network

– ‘set port host’– Router: Set ‘UDP helper’ address

• Experimental Network– Disable noisy automatic protocols– CDP, spantree, bpdu-guard

• Connect control and experimental networks– Port in the ‘Control Hardware’ VLAN on control– Some VLAN with switch IP interface on exper.

7

Installing FreeBSD

• FreeBSD 4.9 recommended right now (11/03)

• Use standard FreeBSD installer

• Pick 'Developer' distribution set

• Install the ports collection

• Pick where you will put important filesystems

• Configure hostname, IP address, etc.

• Install your favorite editors, etc.

8

Important Filesystems on ops

• /usr/testbed/

• /users/

• /proj/

• /groups/

• /share/

9

Important Filesystems on boss

• /usr/

• /usr/testbed/

• /var/

10

Configuring Testbed Software

• Done with GNU autoconf

• Most configuration options stored in 'defs' file– '—with-TBDEFS='

• 'defs-example' provides a template

• Use an object tree outside your source tree

• Uses GNU make (gmake in FreeBSD)

• Install targets– boss-install, ops-install– post-install, to be run as root, on boss

11

Defs File Example# The name of this installation

THISHOMEBASE=Example.Emulab.Net

# Domain this testbed resides in

OURDOMAIN=example.emulab.net

# Host name of our web server (or host:port)

WWWHOST=www.example.emulab.net

# Fully-quailified hostname of the boss node

BOSSNODE=boss.example.emulab.net

# Fully-quailified hostname of the ops (also called users) node

USERNODE=ops.example.emulab.net

# Fully-quailified hostname of the fileserver (will probably # be the same as the ops node)

FSNODE=fs.example.emulab.net

12

Mailing Lists

• testbed-ops– User support– The most serious error messages

• testbed-approval– Requests to start new projects

• testbed-logs– Experiment creation/teardown

• testbed-audit– Audit user and project creation and changes

13

Mailing Lists (contd.)

• testbed-www– Web reports

• testbed-stated– Reports from state daemon

• testbed-users-archive– Archive for automatically-generated testbed-

users list – should be direct to a file

• testbed-active-users-archive– Archive for testbed-active-users lists

14

Setting Up ops

• Run install/ops-install from your object tree

• Install the ops part of the testbed software– 'gmake ops-install'

• Special kernel if using Cyclades Z-series on FreeBSD

• Mailing lists on ops• Put boss in ops' /etc/hosts file• Reboot ops

15

Setting Up boss

• Use same defs file you did with ops– Need a duplicate obj tree for boss

• Start install/boss-install• Install ports• Finish install/boss-install• Add {ops,users,fs} to /etc/hosts• Install the boss part of the software

– Two targets, boss-install and post-install

• ssh from boss to ops as root

16

Other Things To Do on boss

• setuid bit on /usr/bin/suidperl

• Get boss set up as the nameserver for your domain

• SSL certificates for the web

• 1000Hz kernel

• Bootloaders and MFSes from Utah

• Disk images for nodes come from Utah, for now

17

Bootstrapping Users and Projects

• Create first user with 'firstuser' script

• Create holding experiments via the web– In the 'emulab-ops' project– 'hwdown' – for nodes that have failed– 'reloadpending' – for nodes awaiting disk loads– 'reloading' – for nodes having their disks loaded

18

What Goes in the Database

• Switches (nodes)– Note: Switch node_id must be resolvable

– Types (node_types)– Stacks (switch_stacks, switch_stack_types)– Interconnects (interfaces, interface_types, wires)

• Power controllers (nodes, node_types)

• Images and OSids (images, os_info)

19

What Goes in The Database - Nodes

• Node itself (nodes)

• Virtual nodes (nodes)

• Node type (node_types)

• Interfaces (interfaces, interface_types)

• Connection to switches (wires)

• Serial line (tiplines)

• Power control outlet (outlets)

20

Bringing up Nodes - Overview

• Basic idea: Let nodes report in automatically• Nodes boot with a temporary IP address• Unknown nodes boot a 'newnode' FreeBSD

MFS• Nodes report their hardware to boss• boss guesses node names and addresses• Script finds interface ports on switches• Administrator adds nodes to the testbed via a

Web interface (at any time)

21

Bringing up Nodes – Details• Enter node and interface types

• BIOS setup– PXE in boot order– Power-loss behavior– Serial console redirection; password

• Dynamic IP range in dhcpd.conf– Start with dhcpd.conf.template– Use a range of IP addresses not to be used by

nodes

22

Bringing up Nodes (contd.)

• Start with one node, to tweak name and IP– Others will be inferred from the first

• Re-numbering interfaces– Convert from FreeBSD to Linux order

• Searching switches for nodes’ interfaces– Searches all switches in the database– Switch ports must be enabled and in a VLAN

• Create nodes• Free nodes from ‘hwdown’ experiment• Scripts don't do serial and power yet

23

Creating a Disk Imagefor Emulab Test PCs

24

Step by step

• Install a stock {Linux,BSD} of your choice– Make one image with both OS’s (if both popular)

• Get the serial console working

• Setup root access– root password (for console login)– boss root pubkey to ~root/.ssh/authorized_keys

• Ensure perl, rpm and sudo installed

• Create or copy over node host keys

• “make client-install” from build tree

25

Saving the Image

• If Emulab up and running:– Use image creation web form

• Otherwise:– Shutdown to single-user (FS’s still mounted)– Run /usr/local/etc/emulab/prepare– Umount FS’s and RO mount /– Reboot from CD or MFS– Use imagezip to create the image

26

Details: Programs added

• Daemons: idle detection, HW health monitor, watchdog

• Agents: traffic generation, link control, program execution

• Synchronization server

• TMCC

• Emulab startup/shutdown hooks

27

Details: Kernel

• Stock kernels mostly work• For FreeBSD

– Adding IPoD a simple patch

– Shaping nodes require additional features– BSD virtual nodes require massive changes

• For Linux– Adding IPoD a simple patch

28

All Linux, All the Time

• Some environments just do Linux

• Doing FreeBSD also asking too much?

• Converting servers: boss, ops– Get Emulab software to build– Get RPM equivalents to ports– Easy as that!

29

The BSD-free environment

• MFS: for creating and installing images– Should be PXE boot environments out there

• Delay nodes– Need Linux to act as a bridge– Is “tc” up to the task?

• Multiplexed node hosts– UML (probably not, too slow)– vservers (used in planetlab, which we support)– Xen (looks good)

30

Emulab Idle Detection and Policy

31

Idle Detection

• Why do it?– Ease contention for resources

– Encourage better resource use practices

• How activity is defined– CPU load (tunable threshold – currently 0.8)

– TTY activity (mod time on /dev entries)

– Network traffic• Control network vs. Experimental network: different levels

– External activity• E.g.: user-instantiated reboot.

32

Goals and Challenges

• Don’t affect experiment’s performance!

• No false positives!– Could ruin an experimenter’s day— or week—

since swapout saves no node state (currently)

• Few false negatives

• Non-goal: detect intentional circumvention

33

Idle Detection - slothd

• Small daemon runs on each test node• Tracks last observed metrics & compares vs.

current values: crossed activity thresholds?– Runs every 5 minutes when node is active– Runs every 5 seconds when inactive (aggressive mode).

• Instant detection of idle->active transition

• Reports via UDP to central collector– Every five minutes, or send immediately if transitions

from idle->active– Collector stores last activity timestamps in the DB

• One for each type of activity.

34

Idle Detection – policy enforcement

• Script runs every 5 minutes on server.– May send email to user warning of inactivity.

– May swap out experiment (idle, or max duration).

• Policy is dictated by tunable site-vars.– Frequency of email warnings.

– When to swap experiment out.

• Honors per-experiment override settings.– E.g.: expt not swappable; max duration; idle threshold.

– User has control over these both at experiment creation, and after swapin by editing expt. metadata.

35

Idle Detection cont.

• Last seen activity viewable on web page.– Per experiment.– Drill down at node level (CPU, TTY, Network).

• Issues– Easy to circumvent intentionally– Sometimes defeated unintentionally

• E.g.: User leaves traffic generator running

– Hard (impossible?) to determine what activity is meaningful (be context sensitive)

36

Emulab’s Administrative Structure

37

Executive Summary

• Emulab has a two level administrative structure: “Projects”, and “Groups”– Plus “Elab Admins” on top, and “Users” on the bottom

• Permits a variety of common organizational structures– A single user project with no groups

– A class project with multiple, isolated groups managed by TAs

• Administrative control is delegated to “leaders” at each level

38

A “Project”

• Central administrative entity• Started by a faculty member or senior student

– Submitted through web interface– User account gets created for experiment leader

• Approval of project users delegated to leader– Saves on administrative overhead– Project leader responsible for users' behaviour– Leader may grant leader rights to one or more users

• Essential for lazy faculty

• Project gets its own disk space/tree• Users may join multiple projects

39

A “Group”

• Projects may have multiple groups

• Project groups created by the project leader– Designates group leaders for each– Delegates group membership approval to them

• Groups are independent– Files/experiments are protected from each other

• Groups can share– Share the common project file hierarchy