Linux Monitoring and Troubleshooting

By

MySQL Consulting Team

About MyDBOPS

• MySQL Consulting• MySQL Support• Remote DBA support.• Expert MySQL solutions• MariaDB, Percona, Galera, TokuDB are supported too.

Agenda

• Introduction• Linux Architecture• Devices we monitor• Important tools• Examples

INTRODUCTION

Unix-like computer operating system. The monitoring and maintenance of the system is the most important task. There are many commands line utilities created for this purpose.

CPU l

Network Bandwidth

Memory usage

Disk usage

Syslogs & A

Running processes

etc.

Linux Architecture

Devices we monitor

users

Important tools

netstat

free

Df

Du

ps

history

lsof

top

systat

Sar

iostat

vmstat

Pidstat

mpstat

ping

telnet

ifconfig

w & uptime

who

whoami

Top

Small tool, pre-installed in many unix systems. Display all running and active real time process

in ordered list & updates it regularly. CPU usage, Memory usage, Swap Memory, Cache Size,

Buffer Size, Process PID, User, Commands and much

more shows high memory and cpu utilization of a running

processes.

sysstat

powerful logging and monitoring tool for Linux/Unix systems.

contains utilities to monitor system performance and usage activity.

used to monitor system performance and troubleshoot problems.

sysstat is a go-to for power users and can log and track pretty much everything going on

within

your Linux box.

To install :• redhat based - yum install sysstat• Debian based - apt-get install sysstat

sar

Sar(System Activity Report) is part of the sysstat package.

collects and displays ALL system activities statistics.

can monitor performance of various Linux subsystems (CPU, Memory, I/O..) in real time.

also collect all performance data on an on-going basis, store them, and do historical analysis to

identify bottlenecks.

collected information can be used with ksar to plot graphs.

Sar examples

CPU Usage of ALL CPUs (sar -u) This gives the cumulative real-time CPU usage of all CPUs.

sar -u 1 3 Displays real time CPU usage every 1 second

for 3 times.

“-P ALL” indicates that it should displays statistics for ALL

the individual Cores.

%user - % of cpu utilz. that occurred while executing at user level

%nice – at user level with nice priority.

%system – at sytem level

%iowait - % of time that cpu were idle during which s/m had an

outstanding disk I/O request

%idle – idle & s/m did not have an oustanding disk i/o

%steal - % of time spent in involuntary wait by cpu/cpu’s

Sar examples

• Memory Free and Used (sar -r) This reports the memory statistics.

“1 3 reports for every 1 seconds a total of 3 times. ″

focus on “kbmemfree” and “kbmemused” for free and used memory.

Sar examples• Overall I/O Activities (sar -b)

This reports I/O statistics.

“1 3 reports for every 1 seconds a total of 3 times.″ tps – Transactions per second (this includes both read and write)

rtps – Read transactions per second

wtps – Write transactions per second

bread/s – Bytes read per second

bwrtn/s – Bytes written per second

Sar examples

• Individual Block Device I/O Activities (sar -d)

To identify the activities by the individual block devices

(i.e a specific mount point, or LUN, or partition)

The device name (DEV column) can display the actual

device name, use “sar -d”

tps –no of transfer per second

rd_sec/s – no sector read from device

wr_sec/s – no of sectors write from

device

avgrq-sz – avg size in sectors of the

req that were issued to device.

avgqu-sz – avg queue length of

request

await – avg time for i/o req.

svctm – avg service time for i/o req

%util – % of cputime during which i/o

req where issued

Sar examples

• Reports run queue and load average (sar -q)

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes.

“1 3 reports for every 1 seconds a total of 3 times.″

runq-sz – run queue length (no of task wtng for run time)

plist-sz – no of tasks in the task list

Sar examples

• To view previous records(sar –f)

To view old day records

Followed by path of old files

Maintains one month records

sar -f /var/log/sa/sa15

sar -f /var/log/sa/sa15 -q 1 3

LC_ALL=C sar -f /var/log/sa/sa14 -A

iostat simple tool that will collect and show system

input and output storage device statistics.

often used to trace storage device performance issues including devices, local disks, remote disks such as NFS.

%user - % of cpu utilz. that occurred while executing at user level

%nice – at user level with nice priority.

%system – at sytem level

%iowait - % of time that cpu were idle during which s/m had an

outstanding disk I/O request

%idle – idle & s/m did not have an oustanding disk i/o

%steal - % of time spent in involuntary wait by cpu/cpu’s

blk_read/s – amt of data read from device

blk_wrtn/s – amt of data write to device

Blk_read - total no of block reads

Blk_wrtn - total no of block writes

vmstat

Summary information of Memory, Processes, Paging etc.

used to display statistics of virtual memory, kernerl threads, disks, system processes, I/O

blocks, interrupts, CPU activity and much more.

r: The no of processes waiting for run time. b - The no of processes in uninterruptable sle

bi: Blocks sent to a block device (blocks/s). bo-: Blocks received from a block device (blocks/s).

si – Amount of memory swapped in from disk (kB/s). so – Amount of memory swapped to disk (kB/s).

in: The no of interrupts/s including the clock. cs : The number of context switches per second.

us: user time sy: system time id: idle time

pidstat

pidstat is a part of sysstat utility.

To monitor an individual process which is managed by kernel and generate report.

Can also be used for monitoring the child processes of selected tasks.

reports various statistics including CPU used by a process, disk usage statistics of a process,

statistics for threads associated with selected tasks and child processes.

Display statistics for all running process - pidstat -p ALL

Display statistics for a particular process - pidstat -p pid (process id of application)

Display statistics based on process name - pidstat -C “mysql”Display Statistics of Dependent Process in Tree Format using –t - pidstat -t -C "mysql"

mpstat

mpstat is a part of sysstat utility.

To monitor CPU utilization in our system.

Will be more useful if our system has multiple processor.

Print CPU utilization per processors - mpstat -P 0 , mpstat -P 1

Print all CPU Utilization - mpstat -P ALL

Print CPU utilization using intervals - mpstat 3 4

Print mpstat version - mpstat -V

netstat

monitoring incoming and outgoing network packets statistics as well as interface statistics.

very useful tool for every system administrator to monitor network performance and

troubleshoot network related problems..

free

built-in command that displays the total amount of free and used physical memory on your

machine.

also displays the buffers used by the kernel at that given moment.

df

df is an abbreviation for disk free.

pre-installed program in all unix systems used to display the amount of available disk space for

filesystem which the user have access to.

du

Linux “du” (Disk Usage) is a standard Unix/Linux command.

used to check the information of disk usage of files and

directories on a machine.

has many parameter options that can be used to get the

results in many formats.

also displays the files and directory sizes in a recursively

manner.

ps

basic commands for viewing the processes running on the system

It provides a snapshot of the current processes along with detailed information like user id, cpu

usage, memory usage, command name etc.

It does not display data in real time like top or htop commands.

lsof

lsof meaning 'List Open Files' is used to find out which files are open by which process.

when a disk cannot be un-mounted as it says the files are being used.

lsof some examples

List User Specific Opened Files

lsof -u mydbops

Exclude User with ‘^’ Character lsof -i -u^root

Kill all Activity of Particular User kill -9 `lsof -t -u mydbops`

List all Network Connections

lsof -i

Search by PID

lsof -p 1

ping

used to find out whether the peer host/gateway is reachable.

how much time it takes for that data to be exchanged.

default ping waits for 1 second before sending the next packet. increase or decrease using

option -i.

ping 122.166.212.214

ping -i 5 122.166.212.214

ping -c 3 122.166.212.214

telnet

used for interactive communication with another host using the TELNET protocol.

command mode.

Syntax : telnet host port

w & uptime

w - displays information about the users currently on the machine, and their processes

uptime - Tell how long the system has been running.

who & whoami

who - prints information about all users who are currently logged in.

Displays the username, line, and time of all currently logged-in sessions.

whoami - This command prints the username associated with the current effective user ID.

history

The history command can be used to list Bash's log of the commands you have typed:

The history command performs one of several operations related to recently-executed

commands recorded in a history.

Syntax: history

Eg: history | less

Search the history using Control+R

How MyDBOPS Can Help You

• MySQL Consulting

• MySQL Support

• Remote DBA

• 24 * 7 Support

Contact : mysqlsupport@mydbops.com

Thank You