Sharing Unix File SystemsNFS (Network File System ) allows you to share file systems Reasons for sharing file systemTransparent to userUser can keep use their familiar commandsAccess the same file from multiple nodes.To provide disk space to diskless clientsTo prevent duplicationTo provide centrally supported programs and dataTo share data among users Simplifies central support tasksSuch as backup.Space usage monitor

NFSCandidates could be:Home dirs,Web page,/usr/local and other common utilitiesDocumentationsTemporary huge space requestSource code repository

NFS protocol versionsNFS was introduced by Sun Microsystems in 1985The original public release of NFS was version 2In 1990s, version 3Increases performanceMakes writes safely asynchronousBetter support for large filesversion 4No ancillary protocols integrated locking and mountCompound operations bundle multiple RPC together in a single exchange Strong security uses RPCSSEC_GSS APIRequire use of transport protocols that offer congestion control hence NFS v4 will not support UDP transport.

More about NFSNFS run on top of Suns RPC (Remote Procedure Call) protocolRPC provides a system-independent way for processes to communicate in a client-server fashion over a network.Some NFS versions run on top of UDP and TCPWhy?WebNFSPromote the use of NFS over the InternetA extention of NFS protocol that allow easier access to servers and clients through Internet firewallsA WebNFS-enhanced web browser can use an NFS URL to access data directory from the server nfs:/www.YourCompany.com/

Security and NFSNFS protocol was originally designed with essentially no concerns for securityTight control over shared filesystemsDont export any filesystems if anyone that you dont trust has root access on a client hostIf you have a firewall, block port 2049, block SunRPC portmap deamon port 111Kerberos to authenticate to ensure that remote users really are who they say they are. Ipsec to encrypt the data

Network File SystemNFS consists of a number of componentsMounting protocolMount serverDaemons that coordinate basic file serviceSeveral diagnostic utilitiesBasically two parts:ClientMount a directoryServerExport a directory

Server-side NFSMount a filesystemDaemon: mountdAccess filesDaemon: nfsdBoth daemon started at boottimeMountd and nfsd use the same database ( (xtab on most systems, sharetab on Solaris)There are commands (exportfs, share) to add/remove the entries in the databaseExportfs reads /etc/exports fileRun exportfs aTo remove entries, run exportfs u

Server-side NFSOn Solaris, /etc/dfs/dfstab is a shell scriptRun shareall command, which equivalent to sh /etc/dfs/dfstabTo remove entries, run unshareSolaris also provides command exports, which translates export options to share/unshare commandsAny directory can be exportedClient can mount a subdirectories of an exported directoryExample: On HP-UX 11iYou can do above, but you may not to mount another sub dir from the same exported directory.Each device to be exported separatelyFor example/user is a separate partition, / can be exported without exporting /users

NFS Daemons on SolarisThe following daemons are running on server side nfsd [nservers] mountd nfslogdThe following daemons are running on both server side and client side rquotad lockd statd How to start the daemons:Daemons are under /usr/lib/nfs Daemons are started at boot time by /etc/init.d/nfs.client and nfs.server nfs.client starts statd and lockd nfs.server starts nfsd (16 copies default), mountd, nfslogd inetd start rquotad.

Configure NFS on Solaris The share command to export a dir share F nfs [-o options] pathnameOptions are: rw Export read-write to the entire world rw=accesslistExport read-only with access only by listed hosts ro ro=accesslist root=listLists hosts permitted to access this filesystem as rootOtherwise, root access from a client is equivalent to access by nobody (UID 2)

Configure NFS on Solaris anon=uid defines the uid used for users who do not provide a valid user ID. nosubForbids clients to mount subdirectories of the exported directory nosuidPrevents setuid and setgid files from being created via NFSAccess list format Separated by :Identify computers by individual hostnames/IPDomain, .domainNIS netgroup, groupnameNetwork, @network[/prefix]

Configure NFS on SolarisExamples: on Colossus /etc/dfs/dfstab

share -F nfs -o root=aa.aaa.aarw=keeper.csl.mtu.edu:csl:scifi:language:pizza:icu:cs:fishlab:cec /home/csdept

share -F nfs -o root=xx.xxx.xx:yy.yy.yyyrw=csl:language:scifi:pizza:icu:cs:fishlab:physpecial:cec:cs.mtu.edu /export/major

Configuring NFS on Linux/etc/exports The NFS server configuration fileControls which files/dirs are exportedAccess control The general format of entries: directory [host(option)]Host can be empty, means every hostindividual host, name or IP addressDomains, *wrotethebook.comNetworks, IP address/mask(prefix length)Netgroups, @group1Option can be ro rw

LINUX NFS Permission ControlNFS server trusts local authenticationUSE GID/UID to do regular Unix permission control. Mismatch uid/gid will be a unexpected problemWhat about root? Do you want root users on NFS client to act like root users on NFS servers? Probably Not.NFS prevents by default setting root_squashMapping root to nobody UID/GIDSet no_root_squash to allow it. Option all_squashMap all uids and gids to the anonymous user. Useful for NFS-exportd public FTP directories, news spool directories, etc.

LINUX NFSAssign anonymous a UID/GID using option anonuid=XXX, anongid=XXXThis option is primarily useful for PC/NFS clients, where you might want all requests appear to be from one user. UID/GID mapping file using option map_static=filenameCommand exportfs, build /var/lib/nfs/xtab export all in /etc/exports file using exportfs a Sync /etc/exports and /var/bin/nfs/xtab exportfs r Temporarily export a filesystem exportfs fox:/usr/local o rw remove from export list exportfs u fox:/usr/local remove all export list exportfs ua

Client-side NFSMounting remote filesystemsUse mount command to establish temporary network mountsList in /etc/fstab or /etc/vfstabbAutomatic mounting services such as automountwhat are exported or whereAsk administratorFind out yourself:what have been exported from a host: showmount e hostname

Mounting Remote FilesystemsUsing mount command Basic format: mount hostname:remote-dir local_dir hostname is the NFS server local_dir must exist alreadyUsing umount command umount local_dir[remote_dir]

Mounting NFS On Solaris, entries in /etc/vfstab are mounted by mountall during system startupOn Linux, entries in /etc/fstab are mounted via mount a in startup files.NFS mount flags ( page 502)Options:rwrobghard cause the operations that try to access it to block until the server comes backupSoftWhat about a job that has been running 18 hours and will be done in an hour and aborted?

Mounting NFSMore mounting options intrAllow user to interrupt blocked operations nointr retrans=nSpecifies the number of times to repeat a request before returning an error on a soft-mountd filesystem timeo=nSet the timeout period for requests rsize 8K for the same network wsize vers tcp

Common mount options on Linux

OptionFunctionasyncautoWhen a is useddevAllow character and block special files on this file systemexecnoautoDont mount even with -anoexecnosuidDont allow setuid, setgidnouserOnly root can mountremountrorwsuidAllow programs to run setuid or setgidsyncuserOrdinary users to mount the file systematimeUpdate inodenoatimedefaultsRw, suid, dev, exec, auto, nouser, and async

Dedicated NFS file ServersFast, reliable file service is one of the most important elementsDedicated NFS serversOptimized for file serviceStorage scale smoothlyReliable, simplified softwareRedundant hardwareSupport different file formats for different platforms, such as NFS, CIFSEasier to administer than UNIX file serversGood backup and checkpoint facilities

NFS AutomounterAutomatically mounts NFS when neededWhy automount?Maintaining /etc/fstab can be tediousMinimizes the number of active mount points to reduce/Avoid chaos when servers crashesHow?Mount a virtual filesystem driver on the directories for automatic mount to occurKernel-resident filesystem driver called autfs is usedWhen the user references a directory withing the virtual filesystem, the automounter intercepts the reference and mount the actual filesystem the user is trying to reach.

NFS AutomounterDaemonsautomountd and automount are started by /etc/init.d/autofsAutomount reads the configuration file, setup autofs mounts References to automounted filesystems are handled by a separate daemon automountdOn solaris 10, you can start automounter#svcadm enable system/filesystem/autofsThe actual script is under /lib/svc/method/svc-autofs

automountautomountBasic configuration filesMaster mapDirect mapsIndirect mapsDirect and indirect maps provide information about filesystems that are to be automountedA master map lists direct and indirect maps that automount should pay attention to. Only one master map that can be activeThe default one is /etc/auto_master

automountIndirect mapsAutomount several filesystems underneath a common directoryThe path is specified in the master maps, not in the indirect map itselfThe name indirect map will be used as subdirectory to install the mountExample, indirect mapInfo ro chimchimL/chimchim/infoUsing ls a directory full of indirect mounts can be confusing automount does not show the subdirectories until their contents have been accessedWhat does ls do?

automountDirect mapsList the maps that do not share a common prefixSuch as: /cs/tools ro anchor:/cs/tools/usr/srcchimchim:/usr/srcEach implemented with a separate autofs mountRequire slightly overheadBut mount point and directory structure are always accessible by commands such as ls

automountmaster mapformat: mount-point map-name optionList the direct and indirect mapFor indirect map, it specify root directory used by the mounts defined in the map./- is for direct mapsOption set the defaults for all mounts within the map[ruihong@cslserver nis]$ more /etc/auto_master# cs and csl don't use auto_home and auto yet.#/net -hosts -nosuid,nobrowse#/home auto_home -nobrowse#/local auto_local -nobrowse/fisher auto_fisher/- auto_direct

automatic automountHave automount to figure it out itselfQuery mountd running on a remote server to find out what filesystems the server exportsUse host as a map name, automount will map remote hostsexports into the specified automount directoryExampleThe serer chimchim exported /usr/share/manThe clients master map has a line/net -hosts nosuid,softThe mount point will be/net/chimchim/usr/share/manIt does enumerate all possible hostsIt waits for individual subdirectory names to be references,then runs off and mounts the exported filesystem

automatic automountKey substitutionsImpress a degree of regularity on the automounter mapsThe ampersand (&) expands to the matched key value in a mapThe asterisk(*) is a default case. Example: a indirect map:usr1-rwthud:/export/home/usr1usr2-rwthud:/export/home/usr2usr3-rwthud:/export/home/usr3usr4-rwthud:/export/home/usr4Can be rewritten as only one line*-rw thud:/export/home/&

NFS AutomounterAutomount can use maps from NIS server.Other type maps, like auto_home mount# Master map for automounter# #+auto_master #/xfn -xfn /net -hosts -nosuid /home auto_home /- auto_direct

# Home directory map for automounter # #+auto_home craig almond:/export/home/craig * pecan:/export/home/&

PracticeIn Lab4, we created a dir called /research. As a group, lets try the following tasksExport /research, so some hosts in your group as rwsome hosts in your group as roone extra host have root mappingAccess the exported from /net/hostname/homeMount /research from some hosts in your groupAutomount the exported /research to /research from some hosts in your group

Network Information ServiceNIS Was originally called the Sun Yellow Pages Is an administrative databaseWhat information does database store?Provides central control and automatic dissemination of important files.

Network Information ServiceThe unit of sharing in NIS is the record, not the file. A record is usually corresponds to one line Mast server maintains the authoritative copies of system filesAre kept in their original locations and formatsEdited with a text editor beforeA server process makes the contents of the file available over the network.A server and its clients constitute an NIS domainassociated with A set of mapsDifferent from DNS domain

Network Information ServiceOn the master serverData file are preprocessed into database files (called maps) by a hashing libraryOne key associated with each entryA system file may have to be translated into several NIS mapsExample:/etc/passwd is translated into two different NIS Mapspasswd.bynamepasswd.byuidThe record order is not preserved.

Network Information ServiceOn the slave serverRelieve load on the master and keep clients functioning when master is down. Replicate the network maps from master serverOn the clientDoes not distinguish between the master server and slavesTraditionalUse IP broadcasting to locate serversPlace at least one NIS server on every physical networkSome implementation allows to specify the server name instead of broadcasting

Network Information ServiceDatabases are called NIS maps/etc/passwd/etc/group/etc/netgroup/etc/hosts/etc/networks/etc/protocols/etc/services/etc/aliases/etc/auto_master

Network Information ServiceNetgroupsSets of users, machines, and nets for easy reference in other system filesThe format of netgroup entry isGroupname list-of-membersMembers are separated by whitespace. A member isEither a netgroup nameOr a tripnet of the form (hostname, username, nisdomainname) a empty value in a field matching any value in that fieldNetgroup can be used in several system files that define permissionsConfiguring NFS export ( /etc/exports file)Restrict remote logins and shell access ( /etc/hosts.equiv file)Local login access ( /etc/passwd file)

Network Information ServiceExmaples/etc/netgroupgrouplab \(goofy.csl.mtu.edu,,) (mickey.csl.mtu.edu,,) \(minnie.csl.mtu.edu,,) (donald.csl.mtu.edu,,) \(simba.csl.mtu.edu,,) (nala.csl.mtu.edu,,) \(tramp.csl.mtu.edu,,)

noncs1121d \(,msbenson,) (,behart,) (,adrogers,) (,agwalthe,) (,adpeters,) (,ammurrel,) \(,aeblechi,) (,jaeul,) (,srkelley,) (,daperry,) (,anlevend,) (,frtuntla,) \(,cwminnic,) (,rrmilkov,) (,iatrifon,) (,ksrivast,) (,talange,) (,mtnielse,) \(,csdummy,)

/etc/dfs/dfstab

share -F nfs -o root=aa.aaa.aarw=keeper.csl.mtu.edu:csl:scifi:language:pizza:icu:cs:fishlab:cec /home/csdept/etc/passwd+@nonenrolled::::::/usr/local/shells/gone+@cs1121nons::::::/usr/local/shells/limitaccess

Network Information ServiceHow NIS worksNISs data files are stored in one directoryUsually /var/ypEach NIS map is stored in a hashed format in a subdirectory named for the NIS domainExact Map files names depends on the hashing library being used.For example: On cslserver, under /var/yp/csl.mtu.edu, there are ndbm files: password.byname.dir password.byname.pag auto.direct.dir auto.direct.pag

Network Information ServiceThere is a makefile under /var/ypWhich calls makedbm to generate NIS maps from flat files. After you modify a system file, cd to NIS dir which is /var/yp, and run make. Make checks the modification date and rebuild some maps.Slave server run ypxfr command regularly as cron to pull the map copiesServer runs yppush to instruct each slave server to execute ypxfr. Yppush is used in makefile after rebuilding a updated map. a special map called ypservers no correspond flat fileconstructed automatically

Network Information ServiceAfter the initial configurationDeamon ypserv running only on the serversAccept queries from clients and answers them by looking up information in the hashed map filesDeamon ypbind running on every machine, include serversC library contacts the local ypbind to answer a queryprovided /etc/nsswitch.conf says to check with nisYpbind locates a ypserv and returns its identify to the C libraryC library contacts the server directly

NIS commands and daemons (page 527) ypservNIS server daemon ypbindNIS client daemon domainnameSets the NIS domain a machine is in ypxfrDownloads current version of a map from master ypxfrdServes request from ypxfr (runs on master server) yppushMakes slave servers update their versions of a map makedbmbuilds a hashed map from a flat file ypmake rebuild map ypinitConfigure the host as a master or slave server ypsetMakes ypbind connect to a particular server ypwhichfind out which server the current host is using ypcatPrints the values contained in an NIS map ypmatchprints map entries for a specified key yppasswdChanges a password on the NIS master server ypchfnChanges GECOS info on the NIS master server ypchshChanges a login shell on NIS master server yppassddServer for yppasswd, ypchsh, and ypchfn

Set UP NISProcedurePrepare the source fileInitialize master server Run ypinit -mStarting NIS service Run ypservInitialize slave serverSet domainnameRun ypinit command with c and master serve ip.Setup NIS clientsSetup default domainname /etc/nsswitch.confRun ypinit cStart ypbind daemonRun ypbind

NISCase study:Lets take a look at where the startup files for starting NIS server and NIS client./etc/init.d/rpc/usr/lib/netsvc/yp/ypstart (ypstop)Maps/var/yp/csl.mtu.eduLets take a look at the make file

NISWork with DNSHostname can be converted to IP address by DNS, NIS, and the host file. /etc/nsswitch.conf Define the order in which the sources are searched. Choices are:DnsNisFilesFor example:hosts: dns nis files networks: nis [NOTFOUND=return] files services: nis files protocols: nis files

NIS+NIS+ is a completely new software product and structured entirely differently from NISSuns new administrative database released in early 90sProvides all the functionality of NISImproved securityNIS does not authenticate servers or clientsNIS+ use secure DESNIS provide same access to everyone in the domainNIS+ provide access levels to different usersNIS+ is a hierarchical decentralized architecture.NIS+ has enhanced data structures.Multiple columns table and relation query between tablesNIS+ is complex and not as popular as NIS

NIS practiceRefer to the Sun document System Administration Guide: Naming and Directory Service section NIS Setup and Administration As a group Set up a NIS server and a slave serverOn the NIS servercreate a user account named visitor1 with home dir /research/visitor1Set up a NIS clientModify /etc/nsswitch.conf To have passwd line to include nis

Copying files aroundNot elegant solution, but it worksExample: A change to a startup script A new version of software File copying systems can use push model or pull model rdist: push files rsync: push files more securely expect: pull files

Copying files aroundRdistThe easiest way to distribute files from a central server. Sort of like makeA text specification of the files to be distributedRdist copies file when they are out of date.Preserves owner, group, mode and modification time of files. When updating an existing file, it first deletes the old version before installing the new Makes rdist suitable for transferring executables that might be in use during the update.

Copying files aroundIt runs on rsh and relies on rsh-style authentication to gain access to remote systemThe protocol can also be changed from command line rdist -P /usr/local/bin/ssh -f myDistfileControl file has the statements of the formLabel: pathnames -> destination commandsWhere:label: associate a name with the statementpathname: lists of files to be copieddestination: list of hosts to copy files tocommands: modify the behavior ( default is to copy files to the equivalent paths)

Copying files aroundEXAMPLE:SYS_FILES = (/etc/passwd /etc/group /etc/mail/aliases)GET_ALL = (chimchim lollopop barkadon) GET_SOME = (whammo spiff)all: ${SYS_FILES} -> ${GET_ALL}notify barb;special /etc/mail/aliases /usr/bin/newaliasessome: ${SYS_FILES} -> $ {GET_SOME}except /etc/mail/aliases;notify eddie@spiff;

Copying files aroundRsync: push filesFlexiblie and faster replacement of rcpOnly transmit the differences Support for copying links, devices,owners, groups and permissionsCan use any transparent remote shell rsh, sshCan use rsync daemon modeNo controlfileMust run repeatedly to transfer a set of files to multiple hosts.Example:#rsync gopt password-file=/etc/rsync.pwd /etc/passwd lollipop::sysfiles#rsync gopt e ssh /etc/passwd /etc/shadow lollipop:/etc

Copying files aroundExpect: pull filesExpect is a TCL extension scripting languageAllows to you write control scripts for interactive programsThe output produced by each operation is examined to determine what input should be send nextUse expect to retrieve and install themFundamental expect commands are: spawn : start up a subprocess to control send: feed input to a subprocess expect: take action depending on a subprocesss outputExample: Make system files available via FTP from central server

Copying files aroundExample: ftps the /etc/passwd file from the machine netserver

spawn /usr/bin/ftp netserverwhile 1 { expect {Name *: {send netclient\r}Password: {send netclientpassword\r}ftp> {break}failed {send_user Cant log in .\r; exit 1}timeout {send_user Timeout problem. \r; exit 2}}}send lcd /etc\rexpect ftp> {send cd pub/sysfiles\r}expect ftp> {send get passwd\r}expect ftp> {send quit\r, send_user \r}exit 0

SummaryNFSNISCopying file around