CIT 500: IT Fundamentals

Packages and Filesystems

1

Topics

1. Backups1. Policies and planning2. Backup software3. RAID4. LVM

2. Syslog3. /proc

2

Backup DecisionsWhy?

Why are you backing up data? What would happen if you lost data and didn’t back up? What types of data do you have?

What?What to back up—entire system, or specific filesystems? What OS to

backup? What other things to backup—MBR, LVM?When?

When is the best time to backup? How often?Where?

Where will backup occur? Where to store backup volumes?Who?

Who is going to provide backup system? Who will do backups?How?

How are you going to do backups? Tape, mirrors, off-site, etc.

Why Backups?

1. Accidental deletions.2. Hardware failures.3. Data corruption.4. Security incidents.5. Plan for the worst.

1. System catches fire.2. Fire spreads to replicated systems.3. Sprinklers destroy backup system in data ctr.

Backup Types

Full backupComplete copy of all files from a particular time.Backup: slow, requires high capacity.Restore: fast, simple.

Differential backupStorage of changed files since last backup.Backup: fast, may store many incrementals per tape.Restore: slow, complex (requires multiple tapes)

Backup Levels

• Levels define how much is backed up compared to another backup level.– Lower levels back up more data, but– Have higher cost in media and time.– Higher levels are differential backups that store

data that has changed since the last backup at one level below them.

– Higher level backups performed more frequently than low level backups, since are faster + cheaper.

6

Backup Level Examples

Level 0: A full backup of the selected filesystems.

Level 1: A differential backup that backs up only files that have been changed since the last level 0 backup.

Level 2: A differential backup that backs up only files that have been changed since the last level 1 backup.

7

Using a 3-Level Backup

Backup plan:– Perform a level 0 backup on first of month.– Perform a level 1 backup on first day of week.– Perform a level 2 backup each day.

Restore with the following procedure:– Restore most recent level 0 backup.– Restore most recent level 1 backup.– Restore most recent level 2 backup.

8

Capacity Planning

RequirementsHow long do you need to retain data?How much media do you need for each backup?

Example: 3 months of backups3 Level 0 sets of media5 Level 1 sets of media (up to 5 weeks per month)7 Level 2 sets of media (7 days per week)

9

Verifying Backups

• Select backup media to test.– Choose one level 2 per week, one level 1 per

month, one level 0 per year

• List files on backup media.• Restore a random file.

– Verify that a file of appropriate size was created.– Verify contents of file.

10

Backup Software

OS Provided (backup individual systems)cpio, dd, dump, tar, ntbackup

Open source (backup servers)AMANDABacula

Commercial (backup servers)Tivoli Storage Manager (IBM)Veritas Storage Manager

dd

dd – Copy data from input file to output fileif=inputfileof=outputfilebs=[1M]

Primarily used for disk-level backups.dd if=/dev/sda1 of=sda1.ddBacks up MBR, partition table, unused disk space

12

cpio

cpio – Copy input/output-i Extract files from backup-o Write backup to STDOUT

Used for file level backupsReceives list of files to backup on STDIN, sofind / -print | cpio -o > backup.cpio

13

tar

tar – Tape Archivec Create archivex Extract files from archivef Use a file instead of tapez Low compression (gzip format)j High compression (bzip2 format)

Tar is most commonly used file backupEasiest to use tool; uses BSD options so – optional.tar cf /tmp/home-backup.tar /home

14

Compression

Rely on hardware compression– Most tape drives perform compression.– Compression improves speed since there is less

data to write to tape.– Tape capacities often assume 50% compression.

Use software compression– gzip for fast, low compression– bzip2 for higher but slower compression– 7zip for highest but slowest compression

15

Redundant Disks

Disks are most likely component to fail– Moving parts– Constant heavy use

For high reliability, we need redundant disks– Backups will save our data, but if a disk fails, the

system will be down until a new disk is installed and the backup is restored.

– Redundant disks don’t remove need for backups; what happens if data center is destroyed?

16

RAID

Redundant Array of Independent DisksCombine physical disks into single logical unit.Can be implemented in hardware or software.Hardware RAID controllers may provide:

Caching for higher performanceHot swapping for higher reliability

Advantages of RAID over single disks:CapacityReliabilityThroughput

RAID Levels

Level Min Description

JBOD 2 Merge disks for capacity, no striping.

Book calls this RAID Linear.

RAID 0 2 Striped for performance + capacity.

RAID 1 2 Mirrored for fault tolerance.

RAID 3 3 Striped set with dedicated parity disk.

RAID 4 3 Block instead of byte level striping.

RAID 5 3 Striped set with distributed parity.

Striping

• Distribute data across multiple disks.• Improve speed by accessing disks in parallel.

– Independent requests can be serviced in parallel by separate disks.

– Single multi-block requests can be serviced by multiple disks.

• Performance vs. reliability– Performance increases with # disks.– Reliability decreases with # disks.

ParityStore extra bit with each chunk of data.

7-bit data even parity odd parity

0000000 00000000 10000000

1011011 11011011 01011011

1100110 01100110 11100110

1111111 11111111 01111111

Odd parity add 0 if # of 1s is

odd add 1 if # of 1s is

even

Even parity add 0 if # of 1s is

even add 1 if # of 1s is

odd

Error Detection with ParityEven: every byte must have even # of 1s.What if you read a byte with an odd # of 1s?

– It’s an error.– An odd # of bits were flipped.

What if you read a byte with an even # of 1s?– It may be correct.– It may be an error where an even # of bits are

bad.

RAID 0: Striping, no ParityPerformance

Throughput = n * disk speed

Reliability Lower reliability. If one disk lost, entire set is lost. MTBF = (avg MTBF)/# disks

Capacityn * disk size

RAID 1: Disk MirroringPerformance

– Reads are faster since read operations will return after first read is complete.

– Writes are slower because write operations return after second write is complete.

Reliability– System continues to work after one disk dies.– Doesn’t protect against disk or controller

failure that corrupts data instead of killing disk.

– Doesn’t protect against human or software error.

Capacity– n/2 * disk size

RAID 3: Striping + Dedicated ParityReliability

Survive failure of any 1 disk.

Performance Striping increases performance,

but Parity disk must be accessed on

every write. Parity calculation decreases

write performance. Good for sequential reads (large

graphics + video files.)

Capacity(n-1) * disk size

RAID 4: Stripe + Block Parity Disk• Identical to RAID 3

except uses block striping instead of byte striping.

RAID 5: Stripe + Distributed ParityReliability

Survive failure of any 1 disk.

Performance Fast reads (RAID 0), but

slow writes. Like RAID 4 but without

bottleneck of a single parity disk.

Still have to read blocks + write parity block if alter any data blocks.

Capacity(n-1) * disk size

You still need backupsHuman and software errors

– RAID won’t protect you from rm –rf / or copying over the wrong file.

System crash– Crashes can interrupt write operations, leading to situation

where data is updated but parity is not.Correlated disk failures

– Accidents (power failures, dropping the machine) can impact all disks at once.

– Disks bought at same time often fail at same time.Hardware data corruption

– If a disk controller writes bad data, all disks will have the bad data.

Logical Volumes

What are logical volumes?Appear to user as a physical volume.But can span multiple partitions and/or disks.

Why logical volumes?Aggregate disks for performance/reliability.Grow and shrink logical volumes on the fly.Move logical volumes btw physical devices.Replace volumes w/o interrupting service.

LVM

System Logs

• Logs record status and error conditions.• Where do log messages come from?

– Kernel– Accounting system– System services

• Logging methods:– Service records own logs (apache, cron).– Service uses syslog service to manage logs.

Rotation

• Keep backup log files for each day or weeklogfile

logfile.1

logfile.2

logfile.3

• Additional features:– Compress rotated logs to save disk space.– Remove/archive logs that are X days old.

logrotate

• Program to handle log rotation.– Run via /etc/cron.daily.– Configured via /etc/logrotate.conf.

• Options– How often to rotate– How long to keep logs– Compression or not– Log file permissions– Pre- and post-rotate scripts

logrotate.conf# rotate log files weeklyweekly# keep 4 weeks worth of backlogsrotate 4# create new (empty) log files after rotating oldcreate# uncomment if you want your log files compressed#compress# RPM packages drop log rotation information intoinclude /etc/logrotate.d# no packages own wtmp -- we'll rotate them here/var/log/wtmp { monthly create 0664 root utmp rotate 1}

Log file Program Contents

messages syslog Various program/kernel logs.

syslog syslog Various program/kernel logs.

auth.log su, ssh, login Authorization fail/success.

lastlog login, xdm Logins, commands.

wtmp login Login accounting data.

acct/pacct kernel UNIX process accounting.

Xorg.log X-Windows X-Windows failures/info.

Common Log Files

Syslog

Comprehensive logging system.Frees programmers from managing log files.Gives sysadmins control over log management.

Sorts messages bySources (services that generate log messages)Importance (as reported by the service)

Routes messages to different destinationsFilesNetworkTerminals

Syslog Components

SyslogDaemon that does actual logging.Additional daemon, klog, gets kernel messages.

loggerUser-level program to submit logs to syslog.Can use from shell scripts.

Syslog Message Format

• Timestamp: date and time of message• Hostname on which event occurred• Name of program generating log message• Text of log message

37

Example Syslog MessagesFeb 11 10:17:01 localhost /USR/SBIN/CRON[1971]: (root) CMD ( run-parts --

report /etc/cron.hourly)Feb 11 10:37:22 localhost -- MARK --Feb 11 10:51:11 localhost dhclient: DHCPREQUEST on eth1 to 192.168.1.1

port 67Feb 11 10:51:11 localhost dhclient: DHCPACK from 10.42.1.1Feb 11 10:51:11 localhost dhclient: bound to 10.42.1.55 -- renewal in 35330

seconds.Feb 11 14:37:22 localhost -- MARK --Feb 11 14:44:21 localhost mysqld[7340]: 060211 14:44:21 /usr/sbin/mysqld:

Normal shutdownFeb 12 04:46:42 localhost sshd[29093]: Address 218.38.30.101 maps to

ns.thundernet.co.kr, but this does not map back to the address - POSSIBLE BREAKIN ATTEMPT!

Feb 12 04:46:44 localhost sshd[29097]: Invalid user matt from ::ffff:218.38.30.101

Configuring Syslog

Configured in /etc/syslog.confFormat: selector <Tab> actionEx: mail.info /var/log/mail.log

Selector componentsSource (facility)

List of facilities separated by commas or *.Importance (level)

Can be none or *

/etc/syslog.conf# Log anything (except mail) of level info or higher.# Don't log private authentication messages!*.info;mail.none;authpriv.none;cron.none /var/log/messages

# The authpriv file has restricted access.authpriv.* /var/log/secure

# Log all the mail messages in one place.mail.* /var/log/maillog

# Log cron stuffcron.* /var/log/cron

# Everybody gets emergency messages*.emerg *

# Save news errors of level crit and higher in a special file.uucp,news.crit /var/log/spooler

# Save boot messages also to boot.loglocal7.* /var/log/boot.log

Syslog FacilitiesFacility Used By

kern The kernel

user User processes (default)

mail Mail servers and related software.

daemon System daemons (except mail, cron)

auth Security and authorization-related commands.

lpr Print server and related commands.

cron Cron daemon.

local0-7 Eight local levels for other programs.

Syslog LevelsLevel Meaning

emerg Panic situations (hardware failure, crash)

alert Urgent situations

crit Critical situations

err Non-critical errors.

warning Warnings.

notice Might merit investigation.

info Informational messages.

debug Debugging (typically enabled temporarily.)

Syslog ActionsAction Meaning

filename Write message to file on local machine.

@hostname Send message to syslogd on hostname.

@ip Send message to syslogd at IP address.

user1,user2 Write message to user screen if logged in.

* Write message to all logged-in users.

Logger

logger –p facility.level message– facility = facility (kern, user, … local0-7)– level = emerg .. debug– message = text message string, quote if spaces

44

/proc kernel informationPath Information

/proc/cmdline Options that were given to kernel at boot by GRUB

/proc/cpuinfo CPU manufacturer, features, and clock speed

/proc/dma Direct Memory Access channels

/proc/interrupts Interrupts configured for hardware devices

/proc/kcore Kernel memory image (can search using grep)

/proc/loadavg Load average for last 1, 5, 15 minutes

/proc/meminfo System memory usage information

/proc/mounts Mounted filesystems

/proc/modules List of currently loaded kernel modules

/proc/partitions Disk partitions (included unmounted partitions)

/proc/swaps List of swap resources with usage information

/proc/version Version information about the currently running kernel

45

/proc/sys

View running kernel configuration dataex: cat /proc/sys/fs/file-maxex: sysctl net.ipv4.ip_forward

Change running kernel configurationex: echo 48000>/proc/sys/fs/file-maxex: sysctl –w net.ipv4.ip_forward=1

Use /etc/sysctl.conf for permanent changes

46

References

1. Syed Mansoor Sarwar, Robert Koretsky, Syed Ageel Sarwar, UNIX: The Textbook, 2nd edition, Addison-Wesley, 2004.

2. Nicholas Wells, The Complete Guide to Linux System Administration, Thomson Course Technology, 2005.

47