Post on 30-May-2018
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
1/698
Sun Microsystems500 Eldorado BoulevardMS: BRM01-209
Broomfield, Colorado 80021U.S.A.
SolarisSystemPerformance Management
Revision B, October 1999
SA-400
StudentGuide
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
2/698
PleaseRecycle
Cop yrigh t 1999 Sun Microsystem s, Inc., 901 San A nton io Road, Palo A lto, California 94303, U.S.A. All rights reserved .
This pro du ct or docu men t is pr otected by copy right and d istributed u nd er licenses restricting its use, copying ,
distribution, and decomp ilation. No p art of this produ ct or docum ent may be reprodu ced in any form by any m eans
with out p rior written au thor ization of Sun and its licensors, if any.
Third-p arty software, includ ing font techn ology, is copyr ighted an d licensed from Sun sup pliers.
Parts of the prod uct may be d erived from Berkeley BSD systems, licensed from the Univer sity of California. UNIX is a
registered trad emar k in the U.S. and o ther coun tries, exclusively licensed th rou gh X/ Open Com pa ny, Ltd.
Sun, Sun Microsystems, th e Sun Logo, Solaris, NFS, Sun OS, Solstice SyMON , HotJava, Solstice DiskSuite, Sun Tru nkin g,
SunVTS, OpenWind ow s, OpenBoot, Ultra Enterpr ise, Sun StorEdge, SunN et, CacheFS, SunVideo, Sun Swift and VideoPix
are trad emar ks or registered trad emar ks of Sun Microsystems, Inc. in the U.S. and other coun tries.
All SPARC tradem arks are used u nd er license and are trad emar ks or registered trad emar ks of SPARC Internation al, Inc.
in the U.S. and o ther coun tries. Prod ucts bearing SPARC tradem arks are based up on an architecture develop ed by Sun
Microsystems, Inc.
The OPEN LOOK and Sun Grap hical User Interface was d eveloped by Sun Microsystems, Inc. for its user s and licensees.
Sun acknow ledges the pion eering efforts of Xerox in researching and dev eloping th e concept of visual or grap hical user
interfaces for the compu ter ind ustr y. Sun hold s a non -exclusive license from Xerox to the Xerox Grap hical User Interface,
wh ich license also covers Suns licensees wh o implem ent OPEN LOOK GUIs and otherw ise comp ly with Suns wr itten
license agreemen ts.
U.S. Governm ent app roval required w hen exporting the prod uct.
RESTRICTED RIGHTS: Use, du p lication, or disclosu re by the U.S. Govern men t is subject to restr ictions of FAR 52.227-
14(g) (2)(6/ 87) and FAR 52.227-19(6/ 87), or DFAR 252.227-7015 (b)(6/ 95) and DFAR 227.7202-3(a).
DOC UMEN TATION ISPROVIDED "AS IS"A ND ALL EXPRESSO R IMPLIED CON DITION S, REPRESENTATION S,
AN D WARRAN TIES, INCLUDIN G ANY IMPLIED WARRAN TYOF MERCHANTABILITY, FITNESSFOR A
PARTICULAR PURPOSE OR NON -INFRING EMENT, ARE DISCLAIMED, EXCEPT TO TH E EXTENT THAT SUCH
DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
3/698
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
4/698
iv Solaris S ystem Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
sar....................................................................................................... 2-7vmstat ................................................................................................ 2-8iostat ................................................................................................ 2-9mpstat .............................................................................................. 2-10netstat ............................................................................................ 2-11nfsstat ............................................................................................ 2-12SyMON System Monitor................................................................ 2-13Other Utilities .................................................................................. 2-14memtool ............................................................................................ 2-15/usr/proc/bin............................................................................... 2-17The Process Manager ...................................................................... 2-18The SE Toolkit.................................................................................. 2-20System Perform ance Monitoring With the SE Toolkit .............. 2-21SE Toolkit Example Tools .............................................................. 2-22System Accou nting ......................................................................... 2-23Viewing Tunable Param eters ........................................................ 2-24
Setting Tunable Parameters........................................................... 2-25/etc/system ................................................................................... 2-26Check You r Progress ...................................................................... 2-27Think Beyond .................................................................................. 2-28
System Monitoring Tools ...................................................................... L2-1Tasks ................................................................................................ L2-2
Using the Tools....................................................................... L2-2Enabling Accounting .............................................................L2-3Installing the SE Toolkit ........................................................ L2-3Installing memtool................................................................ L2-4
Processes and Threads ..............................................................................3-1Relevance............................................................................................ 3-2Process Lifetime ................................................................................ 3-3forkWith exec Exam ple................................................................. 3-4Process Performance Issues............................................................. 3-5Process Lifetime Performance Data................................................ 3-6Process-Related Tun able Param eters ............................................. 3-8Mu ltith read ing ................................................................................ 3-10Ap plication Thread s ....................................................................... 3-12Process Execution ............................................................................ 3-13Thread Execution ............................................................................ 3-14
Process Thread Examples .............................................................. 3-15Perform ance Issues ......................................................................... 3-16Thread and Process Perform ance ................................................. 3-17Locking ............................................................................................. 3-18Locking Problems ........................................................................... 3-20The lockstatCommand .............................................................. 3-21Inter ru pt Levels ............................................................................... 3-22The clockRoutine .......................................................................... 3-23
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
5/698
Solaris System Performance Management vCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B
Clock Tick Processing ..................................................................... 3-24Process Monitoring Using ps........................................................ 3-25Process Monitoring Using System Accounting .......................... 3-28
msacct.se................................................................................3-32pea.se ......................................................................................3-32ps-ax.se..................................................................................3-34ps-p.se ....................................................................................3-34pwatch.se................................................................................3-34
Check You r Progress ...................................................................... 3-35Think Beyond .................................................................................. 3-36
Processes Lab ........................................................................................... L3-1Tasks ................................................................................................ L3-2
Using maxusers .................................................................... L3-2Using lockstat ..................................................................... L3-3Monitoring Processes ............................................................ L3-3
CPU Scheduling .........................................................................................4-1Relevance............................................................................................ 4-2Sched uling States .............................................................................. 4-3Sched uling Classes ............................................................................ 4-4
Class Characteristics .................................................................4-5Disp atch Priorities ............................................................................. 4-6Timesharing/ Interactive Dispatch Param eter Table ................... 4-7
CPU Starva tion ..........................................................................4-8Dispatch Param eter Table Issues .................................................... 4-9Timesharing Dispatch Parameter Table (Batch)......................... 4-10The dispadminComman d ............................................................ 4-12
dispadminExample ....................................................................... 4-13The Interactive Scheduling Class.................................................. 4-14Real-Time Sched uling .................................................................... 4-15Real-Time Issues.............................................................................. 4-16Real-Time Dispatch Parameter Table........................................... 4-18The priocntlComman d .............................................................. 4-19Kernel Thread Dispatching ........................................................... 4-21
Old Thread ...............................................................................4-21New Thread .............................................................................4-22
Processor Sets .................................................................................. 4-23Dynamic Reconfigu ration Interaction ..................................4-24
The Run Qu eu e................................................................................ 4-27CPU Activity .................................................................................... 4-28CPU Perform an ce Statist ics ........................................................... 4-30Available CPU Performance Data ................................................ 4-34CPU Control and Monitoring........................................................ 4-35The mpstatComman d ................................................................... 4-36mpstatData ..................................................................................... 4-38Solaris Resource Manager.............................................................. 4-39
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
6/698
vi Solaris S ystem Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
SRM Hierarchical Shares Exam ple ............................................... 4-40SRM Limit Nod e ............................................................................. 4-41CPU Monitoring Using SE Toolkit Programs............................. 4-42
cpu_meter.se.........................................................................4-42vmmonitor.se.........................................................................4-42mpvmstat.se ...........................................................................4-42
Check You r Progress ...................................................................... 4-43Think Beyond .................................................................................. 4-44
CPU Scheduling Lab .............................................................................. L4-1Tasks ................................................................................................ L4-2
CPU Dispatching.................................................................... L4-2Process Pr iorities .................................................................... L4-3The Dispatch Parameter Table ............................................. L4-6Real-Time Sched uling ........................................................... L4-7
System Caches ............................................................................................5-1
Relevance............................................................................................ 5-2What Is a Cache? ............................................................................... 5-3Hardware Caches.............................................................................. 5-4SRAM an d DRAM ............................................................................ 5-5CPU Caches........................................................................................ 5-6Cache Op eration ................................................................................ 5-7Cache Miss Processing ..................................................................... 5-8Cache Rep lacement ........................................................................... 5-9Cache Op eration Flow .................................................................... 5-10Cache Hit Rate ................................................................................. 5-11Effects of CPU Cache Misses ......................................................... 5-13
Cache Characteristics...................................................................... 5-14Virtual Address Cache ................................................................... 5-15Physical Address Cache ................................................................. 5-16Direct Mapped and Set Associative Caches................................ 5-17
Harvard Caches .......................................................................5-18Write-Through and Write-Back Caches....................................... 5-19
Write Cancellat ion ..................................................................5-19CPU Cache Snooping ..................................................................... 5-21Cache Thrashing.............................................................................. 5-23System Cache Hierarchies ............................................................. 5-25Relative Access Times .................................................................... 5-27
Cache Performance Issues ............................................................. 5-28Things You Can Tune..................................................................... 5-30Check You r Progress ...................................................................... 5-31Think Beyond .................................................................................. 5-32
Cache Lab.................................................................................................. L5-1Tasks ................................................................................................ L5-2
Cache Miss Costs .................................................................... L5-2
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
7/698
Solaris System Performance Management viiCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B
Memory Tuning .........................................................................................6-1Objectives ........................................................................................... 6-1Relevance............................................................................................ 6-2Virtu al Memory ................................................................................. 6-3Typ es of Segm ents ............................................................................ 6-6Virtu al Ad dress Translat ion ............................................................ 6-7Page Descrip tor Cache ..................................................................... 6-8Virtual Address Lookup ................................................................ 6-10The Memory Free Qu eu e ............................................................... 6-11The Paging Mechan ism .................................................................. 6-12Page Daemon Scanning.................................................................. 6-13Page Scanner Processin g ................................................................ 6-14Defau lt Paging Parameters ............................................................ 6-15An Alternate Approach The Clock Algorithm ........................ 6-16maxpgio ............................................................................................ 6-18File System Caching........................................................................ 6-20
The Buffer Cache ............................................................................. 6-22The Page Cache ............................................................................... 6-24Priority Paging................................................................................. 6-25Available Paging Statistics............................................................. 6-27Available Paging Data.................................................................... 6-31Additional Paging Statistics .......................................................... 6-32Swapping.......................................................................................... 6-34What Is Not Swappable ................................................................. 6-36Swapping Priorities ........................................................................ 6-37Swap In ............................................................................................. 6-38Swap Space ...................................................................................... 6-39
tmpfs ................................................................................................ 6-41Available Swap ping Statistics ....................................................... 6-43Available Swapping Data .............................................................. 6-46Shared Librar ies .............................................................................. 6-47Memory Utilization ........................................................................ 6-49Tota l Physical Memory .................................................................. 6-50File Buffering Memory ................................................................... 6-51Kern el Memory ............................................................................... 6-52Iden tifying an Applications Memory Requ irements ................ 6-54Free Memory.................................................................................... 6-56Identifying a Mem ory Shortage .................................................... 6-57
Using the memtoolGUI.................................................................... 6-59Things You Can Tune..................................................................... 6-63Check You r Progress ...................................................................... 6-65Think Beyond .................................................................................. 6-66
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
8/698
viii Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
Memory Tuning Lab .............................................................................. L6-1Tasks ................................................................................................ L6-2
Process Mem ory ..................................................................... L6-2Page Daemon Parameter s ..................................................... L6-3Scan Rates ................................................................................ L6-7
System Buses...............................................................................................7-1Relevance............................................................................................ 7-2What Is a Bus? ................................................................................... 7-3General Bus Characteristics............................................................. 7-4System Buses...................................................................................... 7-5
MBus an d XDbus ......................................................................7-6UPA and Gigaplane Bus ..........................................................7-7
Gigap lane XB Bus.............................................................................. 7-8The Gigap lan e XB Bus ...................................................................... 7-9Sun System Backp lanes Sum mary................................................ 7-10Perip heral Buses .............................................................................. 7-11
SBus Versus PCI Bus....................................................................... 7-12Enterprise Server I/ O Boards................................................7-13
PIO an d DVMA ............................................................................... 7-14PIO.............................................................................................7-15DVMA.......................................................................................7-15Which Is Being Used?.............................................................7-15
prtdiag ............................................................................................ 7-16prtdiagCPU Section ..................................................................... 7-17prtdiagMemory Section .............................................................. 7-18
Memory Interleaving..............................................................7-18prtdiag I/ O Section ...................................................................... 7-20
Bus Limits......................................................................................... 7-21Bus Ban dwidth ................................................................................ 7-22Diagnosing Bus Problems.............................................................. 7-23Avoid ing the Problem .................................................................... 7-24SBus Overload Example................................................................. 7-26Dynamic Reconfigu ration Con sidera tions .................................. 7-28Tuning Repor ts ................................................................................ 7-29Check You r Progress ...................................................................... 7-30Think Beyond .................................................................................. 7-31
System Buses Lab.................................................................................... L7-1
Tasks ................................................................................................ L7-2System Configu ration ............................................................ L7-2
I/O Tuning ...................................................................................................8-1Relevance............................................................................................ 8-2SCSI Bus Overview ........................................................................... 8-3SCSI Bus Characteristics .................................................................. 8-5SCSI Speeds........................................................................................ 8-6
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
9/698
Solaris System Performance Management ixCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B
SCSI Widths ....................................................................................... 8-8SCSI Lengths ...................................................................................... 8-9SCSI Properties Summary.............................................................. 8-10SCSI Inter face Ad dressing ............................................................. 8-11SCSI Bus Addressing ...................................................................... 8-13SCSI Target Pr iorities ..................................................................... 8-14Fibre Chan nel................................................................................... 8-15Disk I/ O Time Components.......................................................... 8-18
Access to the I/ O Bus .............................................................8-18Bus Transfer Time ...................................................................8-20Seek Time .................................................................................8-20Rota tion Time ..........................................................................8-20ITR Transfer Time ...................................................................8-21Reconnection Time..................................................................8-21Interru pt Time .........................................................................8-21
Disk I/ O Time Calcu lation ............................................................ 8-22
Bus Data Transfer Timing......................................................8-23Latency Timing........................................................................8-23Disk Drive Featu res ........................................................................ 8-24Multiple Zone Recording............................................................... 8-25Drive Caching and Fast Writes ..................................................... 8-27
Fast Writes ................................................................................8-27Tagged Qu eu eing ............................................................................ 8-29
Ordered Seeks ..........................................................................8-30Mode Pages ...................................................................................... 8-31Device Properties From prtconf ................................................. 8-32Device Properties From scsiinfo ............................................... 8-34
I/ O Performan ce Planning ............................................................ 8-36Decision Support Examp le (Sequen tial I/ O) ......................8-37Transaction Processing Examp le (Random I/ O) ...............8-37
Tap e Drive Operat ions ................................................................... 8-38Storage Arrays and JBODs............................................................. 8-39Storage Array Architecture............................................................ 8-40RAID ................................................................................................. 8-41RAID-0.............................................................................................. 8-42RAID-1.............................................................................................. 8-44RAID-0+1.......................................................................................... 8-46RAID-3.............................................................................................. 8-48
RAID-5.............................................................................................. 8-50Tuning the I/ O Subsystem ............................................................ 8-52Available Tuning Data ................................................................... 8-54Available I/ O Statistics .................................................................. 8-55Available I/ O Data ......................................................................... 8-58iostat Service Time....................................................................... 8-59
siostat.se .............................................................................8-60xio.se ......................................................................................8-60
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
10/698
x Solaris System Performance Management Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
xiostat.se .............................................................................8-61iomonitor.se.........................................................................8-61iost.se ....................................................................................8-61xit.se ......................................................................................8-62disks.se..................................................................................8-62
Check You r Progress ...................................................................... 8-63Think Beyond .................................................................................. 8-64
I/O Tuning Lab ........................................................................................ L8-1Tasks ................................................................................................ L8-2
SCSI Device Characteristics .................................................. L8-2
UFS Tuning .................................................................................................9-1Relevance............................................................................................ 9-2The vnode Interface .......................................................................... 9-3The Local File System ....................................................................... 9-5Fast File System Layou t ................................................................... 9-6
Disk Label...................................................................................9-6Boot Block...................................................................................9-6Sup erblock..................................................................................9-7Cylinder Grou ps........................................................................9-7
The Directory ..................................................................................... 9-9Directory Nam e Lookup Cache (DNLC)..................................... 9-10The inodeCache ............................................................................. 9-12DNLC and inodeCache Management ........................................ 9-14The inode......................................................................................... 9-16
inode Time Stamp Updates ..................................................9-17The inodeBlock Pointer s .............................................................. 9-18
UFS Buffer Cache ............................................................................ 9-19Hard Links ....................................................................................... 9-21Symbolic Links ................................................................................ 9-22Allocating Directories and inodes ............................................... 9-23Allocation Performance.................................................................. 9-24Allocating Data Blocks ................................................................... 9-25Qu ad ra tic Rehash ............................................................................ 9-26
The Algorithm .........................................................................9-27Fragm ents ......................................................................................... 9-28Logging File Systems...................................................................... 9-29Ap plication I/ O............................................................................... 9-30
Working With the segmapCache ................................................. 9-33File System Performance Statistics ............................................... 9-34fsflush ............................................................................................ 9-37Direct I/ O......................................................................................... 9-39
Restrictions...............................................................................9-40Accessing Data Directly With mmap.............................................. 9-41Using madvise................................................................................. 9-42
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
11/698
Solaris System Performance Management xiCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B
File System Per forman ce................................................................ 9-43Data-Intensive or Attribu te-Intensive App lications ..........9-43Sequential or Random Accesses ...........................................9-44
Cylinder Grou ps.............................................................................. 9-45Sequential Access Workloads........................................................ 9-47UFS File System Read Ahead ........................................................ 9-49Setting Cluster Sizes for RAID Volumes ..................................... 9-51Com mands to Set Cluster Size ...................................................... 9-53
Tuning for Sequ ential I/ O .....................................................9-54File System Write Behind ............................................................... 9-55UFS Write Throttle .......................................................................... 9-57Random Access Workloads........................................................... 9-59Tuning for Random I/ O ................................................................ 9-61Post -Creation Tuning Param eter s ................................................ 9-63
Fragment Optimization..........................................................9-63minfree ....................................................................................9-64
maxbpg ......................................................................................9-64Ap plication I/ O Perform ance Stat istics....................................... 9-65Check You r Progress ...................................................................... 9-67Think Beyond .................................................................................. 9-68
UFS Tuning Lab ...................................................................................... L9-1Tasks ................................................................................................ L9-2
File Systems............................................................................. L9-2Directory Nam e Looku p Cache ........................................... L9-3mmap an d read/write........................................................... L9-6Using madvise........................................................................ L9-8
Network Tuning .......................................................................................10-1Relevance.......................................................................................... 10-2Networks .......................................................................................... 10-3Network Ban dwidth ....................................................................... 10-4Full Duplex 100-BaseT Ethernet.................................................... 10-5
ndd Exam ples ...........................................................................10-6Forcing 100-Base T Full Duplex ............................................10-6
IP Tru nkin g ...................................................................................... 10-7Sun Trunking Software ..........................................................10-8
TCP Connections............................................................................. 10-9TCP Stack Issues............................................................................ 10-11
TCP Tuning Param eters ............................................................... 10-12Using ndd ....................................................................................... 10-14Network Hardware Performance............................................... 10-16NFS.................................................................................................. 10-18NFS Retran smission s .................................................................... 10-19NFS Server Daemon Thread s ...................................................... 10-21
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
12/698
xii Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
Mon itorin g N etw ork Performance............................................. 10-23ping........................................................................................10-24spray .....................................................................................10-25snoop .....................................................................................10-26
Isolating Problems ........................................................................ 10-27Tuning Repor ts .............................................................................. 10-28Network Monitoring Using SE Toolkit Programs ................... 10-29
net.se ....................................................................................10-29netstatx.se .........................................................................10-29nx.se ......................................................................................10-29netmonitor.se.....................................................................10-30nfsmonitor.se.....................................................................10-30tcp_monitor.se...................................................................10-30
Check You r Progress .................................................................... 10-31Think Beyond ................................................................................ 10-32
Performance Tuning Summary .............................................................11-1
Relevance.......................................................................................... 11-2Some General Guidelines............................................................... 11-3What You Can Do ........................................................................... 11-4Ap plication Source Cod e Tun ing ................................................. 11-5Compiler Optimization .................................................................. 11-6Ap plication Executable Tun ing Tips ............................................ 11-7Performance Bottlenecks................................................................ 11-8Memory Bottlen ecks ..................................................................... 11-10Memory Bottleneck Solutions ..................................................... 11-11I/ O Bottlen ecks ............................................................................. 11-13I/ O Bottlen eck Solutions ............................................................. 11-15
CPU Bottlen ecks ............................................................................ 11-17CPU Bottleneck Solu tion s ............................................................ 11-19Ten Tuning Tips ............................................................................ 11-21Case Study...................................................................................... 11-23Check You r Progress .................................................................... 11-46Think Beyond ................................................................................ 11-47
Performance Tuning Summary Lab .................................................. L11-1Tasks .............................................................................................. L11-2
vmstat Inform ation .............................................................L11-2sarAn alysis I ....................................................................... L11-3
sarAn alysis II...................................................................... L11-6sarAn alysis III ....................................................................L11-9
Interface Card Properties .........................................................................A-1Ad ditional Resou rces ...................................................................... A-2SBus Implementations and Capabilities....................................... A-3Typical and Peak Bandwidth of SBus Devices ............................ A-4
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
13/698
Solaris System Performance Management xiiiCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B
Installing and Configuring the SyMON System Monitor................ B-1Ad dition al Resou rces ...................................................................... B-2SyMON System Monitor ................................................................. B-3Sup ported Server Platforms ........................................................... B-4Up grad ing the SyMON System Monitor ...................................... B-5Installing the SyMON System Mon itor ........................................ B-6Installing the SyMON Packages .................................................... B-7
Env ironment Variables............................................................ B-7Configuring th e SyMON System Mon itor ................................... B-8
Configuring the Monitored Server System .......................... B-8Run ning the SyMON User In terface .....................................B-8Configuring the Event H andler System ............................... B-9
Setting Up SNMP With Solstice Domain (Sun Net)Manager ........................................................................................ B-10
Directin g the SNMP Trap s.................................................... B-10Writing or Mod ifying Event Rules to Send SNMP
Traps ..................................................................................... B-11How to Stop or Remove SyMON From the System .................. B-12Configu ration Issu es ...................................................................... B-13
Accounting .................................................................................................C-1Overview ........................................................................................... C-2
What Is Accounting? ...............................................................C-2What Is Accou nting Used For? ..............................................C-2
Types of Accounting........................................................................ C-3Connection Accounting...........................................................C-3Process Accounting..................................................................C-3Disk Accou nting .......................................................................C-4
Chargin g ....................................................................................C-4How Does Accounting Work? ....................................................... C-5
Location of Files .......................................................................C-5Typ es of Files ............................................................................C-5What Hap pens at Boot Time ..................................................C-5Programs That Are Run ..........................................................C-6Location of ASCII Report Files...............................................C-7
Starting and Stopping Account ing ................................................ C-8Genera ting the Data and Reports ................................................ C-10
Raw Data .................................................................................C-10rprtMMDD Daily Report File............................................C-12Generation of the rprtMMDD File....................................C-12The runacct Program ...........................................................C-17runacct States ........................................................................C-18Periodic Repor ts .....................................................................C-20Looking at the pacct File With acctcom ...........................C-21acctcomOp tions ....................................................................C-22
Genera ting Custom Analyses....................................................... C-24
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
14/698
xiv Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
Writing Your Ow n Program s ...............................................C-24Raw Data Formats..................................................................C-24/var/adm/wtmpFile ..............................................................C-25/var/adm/pacctFile ............................................................C-26/var/adm/acct/nite/disktacctFile .............................C-27
Summary of Accoun ting Program s an d Files ............................ C-28
The Cache File System .............................................................................D-1Introd uction ...................................................................................... D-2
Function .....................................................................................D-2Characteristics ..........................................................................D-2Terminology ..............................................................................D-2Benefits ......................................................................................D-3Limitations ................................................................................D-4Required Software ...................................................................D-4
Settin g Up CacheFS ......................................................................... D-5Creating a Cache ......................................................................D-5
Disp laying Inform ation ...........................................................D-5Deletin g a Cache ............................................................................... D-6Mounting With CacheFS................................................................. D-7
Using an Alread y Mounted NFS File Systems ....................D-7Using /etc/vfstab.................................................................D-7Using the Automounter ..........................................................D-7
cachefsOp eration .......................................................................... D-8fsck............................................................................................D-8Writes to Cached Data.............................................................D-8
Managing cachefs.......................................................................... D-9cachefsstat ............................................................................D-9
cachefslog ..............................................................................D-9cachefswssize......................................................................D-10cachefspack ..........................................................................D-10
CacheFS and CD-ROMs ................................................................ D-11Exercise: Using CacheFS ............................................................... D-12
Preparation ..............................................................................D-12Tasks ........................................................................................D-13
IPC Tuneables............................................................................................ E-1Ad ditional Resources ...................................................................... E-2Overview ........................................................................................... E-3
How to Determ ine Curren t IPC Values ................................ E-3How to Change IPC Parameters ............................................ E-4Tuneable Parameter Limits ..................................................... E-5
Shared Memory ................................................................................ E-6Semap hores ....................................................................................... E-8Message Qu eu es ............................................................................. E-11
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
15/698
Solaris System Performance Management xvCopyr ight 1999Sun Microsystems, Inc. All Rights Reserved . Enterp rise Services October 1999, Revision B
Performance Monitoring Tools .............................................................. F-1Ad dition al Resources ...................................................................... F-2sar...................................................................................................... F-3
Disp lay File Access Op era tion Statistics ............................... F-3Disp lay Buffer Activity Statistics ........................................... F-3Display System Call Statistics ................................................ F-4Display Disk Activity Statistics.............................................. F-6Display Pageou t and Memory Freeing Statistics
(in Averages).......................................................................... F-7Display Kernel Memory Allocation (KMA) Statistics ........ F-8Display Interp rocess Communication Statistics.................. F-9Disp lay Pagein Stat istics ......................................................... F-9Display CPU and Swap Queue Statistics ........................... F-10Display Available Memory Pages and Swap Space
Blocks .................................................................................... F-11Disp lay CPU Utilization (the Defau lt) ................................ F-11
Display Process, Inode, File, and Shared Memory TableSizes....................................................................................... F-12Display Swapping and Switching Statistics....................... F-12Disp lay Monitor Term inal Device Stat istics ...................... F-13Disp lay All Stat istics .............................................................. F-14Other Op tions ......................................................................... F-15
vmstat ............................................................................................. F-16Display Virtual Memory Activity Sum marized at Some
Op tional, User-Supplied Interval (in Second s)............... F-16Disp lay Cache Flush ing Statistics ........................................ F-19Disp lay Interrupts Per Device.............................................. F-19
Display System Event Information, Coun ted SinceBoot ....................................................................................... F-20Display Swapp ing Activity Sum marized at Some
Op tional, User-Supplied Interval (in Second s)............... F-21iostat ............................................................................................. F-22
Disp lay CPU Statistics ........................................................... F-22Display Disk Data Transferred and Service Times ........... F-22Disp lay Disk Op erations Statistics ...................................... F-23Disp lay Device Error Sum mary Stat istics .......................... F-23Disp lay All Device Errors ..................................................... F-24Display Counts Rather Than Rates, Where Possible ........ F-24
Report on On ly the First nDisks ......................................... F-24Display Data Throu ghp ut in Mbytes/ sec Rather Than
Kbytes/ sec ........................................................................... F-25Display Device Nam es as Mount Points or Device
Ad dresses ............................................................................. F-25Display Per-Par tition Statistics as Well as Per-Device
Statistics ................................................................................ F-25Disp lay Per-Partition Statistics On ly................................... F-26
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
16/698
xvi Solaris S yst em Performance M anagement Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999,Revision B
Display Characters Read and Written to Terminals ......... F-26Display Extend ed Disk Statistics in Tabu lar Form ........... F-27Break Down Service Time, svc_t, Into Wait and Active
Times an d Put Device Nam e at the End of the Line ...... F-28mpstat ............................................................................................. F-29netstat ........................................................................................... F-31
Show the State of All Sockets ............................................... F-31Show All Interfaces Under Dynamic Host Configuration
Protocol (DHCP) Control................................................... F-34Show Interface Mu lticast Group Membersh ips................. F-34Show State of All TCP/ IP Interfaces................................... F-35Show Detailed Kernel Memory Allocation Information .. F-36Show STREAMS Sta tistics .................................................... F-37Show Multicast Routing Tables ........................................... F-37Show Mu lticast Routing Statistics ....................................... F-38Show Network Addresses as Nu mbers, Not Nam es........ F-38
Show the Address Resolution Protocol (ARP) Tables...... F-39Show the Routing Tables ...................................................... F-39Show Per Protocol Statistics ................................................. F-41Show Extra Socket and Rout ing Table Information.......... F-42
nfsstat ........................................................................................... F-44Show Client Information On ly............................................. F-44Show NFS Mount Options.................................................... F-46Show NFS Inform ation; Both Client and Server................ F-47Show RPC Inform ation ......................................................... F-48Show Server Information only ............................................. F-49
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
17/698
xviiCopyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enterp rise Services October 1999,Revision B
About This Course
Course Goal
This course provides an introduction to performance tuning for larger
Sun environm ents. It covers most of the major areas of systemperformance, concentrating on ph ysical memory and inpu t/ outpu t
(I/ O) management.
The Sun environment is qu ite complex. There are thousand s of
possible hardw are configurations, and tens of thousand s of software
configurations. Because it is not p ossible to give cookbook-style
tuning information, this course examines how the components operate
and interoperate. This und erstanding can be used to determine what
kinds of problems can arise, how to identify them, and how to
alleviate them.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
18/698
x viii S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
CourseOverview
An overview of the operation of various system hard ware and
software components is given first, then specific areas are examined.
This provides an understanding of what effect a certain tuning changewill have and whether it is necessary.
This course is intended for system administrators and system
designers with a background in Sun SPARC system administration.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
19/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
CourseMap
The following course map enables you to see the general topics and
the modules for that topic area in reference to the course goal:
Overview
Introduction toPerformanceManagement
Tools
SystemMonitoring Tools
CPU
Processes andThreads
CPU Scheduling
Memory
System Caches Memory Tuning
I/O
SystemBuses
I/O Tuning UFS Tuning
Networks
Network Tuning
Summary
PerformanceTuning
Summary
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
20/698
xx Solaris S ystem Performance M anagement
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Module-by-ModuleOverview
This course contains the following modules:
q
Modu le 1 Introdu ction to Performan ce Management
This modu le introdu ces the concepts of tuning, tuning tasks, and
terminology, and the steps involved in tuning a system. A list of
tuning references is also provided.
q Module 2 System Monitoring Tools
The various performance monitoring tools available with the
Solaris Operating Environmen t (Solaris), the information th at
they provide, and how to inspect and set system tun ing
param eters are described in th is modu le.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
21/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
q Module 3 Processes and Threads
This module provides a description of the behavior of processes
and threads in the Solaris environment. It covers their creation,
execution, and related tuning issues, as well as system timing and
the clock thread.
q Module 4 CPU Scheduling
How to control work running on the systems central processing
un its (CPUs), d ispatch p riorities, do real-time p rocessing, and use
processor sets are all discussed in this module.
q Module 5 System Caches
This module describes the generic operation of caches and the
specific operation of the hardware caches. Since many system
componen ts use caches or op erate like one, un derstand ing cachesis fun dam ental to being able to manage system p erformance.
q Modu le 6 Memory Tun ing
The operation of the virtual and real memory subsystems is
covered in this module. Because it is perhaps the most important
componen t of system p erformance, memory operation is
discussed in some detail here.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
22/698
xxii Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
q Module 7 System Buses
This mod ule shows how buses operate, wh at kinds of buses there
are in the system, and what bus characteristics are important to
tuning.
q Module 8 I/ O Tuning
In this mod ule, the characteristics of the small compu ter system
interface (SCSI) bus and d isk drives and their interactions are
described. It covers redundant array of inexpensive disks (RAID)
levels and storage arrays, and looks at the cost of I/ O operations
to enable the system administrator to quickly identify and locate
the source of an I/ O problem.
q Module 9 UFS Tun ing
This module explains the structure and operation of the Solaris
UNIX file system (UFS). It discusses the various UFS disk
structures, caches, and allocation algorithm s. A description of the
various UFS configuration and tun ing param eters is also provided.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
23/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
q Modu le 10 Network Tun ing
The tuning required for a network, both the physical network an d
the network applications, is covered in this module. It discusses
tun ing the Transmission Con trol Protocol/ Internet Protocol
(TCP/ IP) stack and NFS.
q Module 11 Performance Tuning Summary
This module summarizes the material covered in this course. It
offers examp les of poor performance and their causes. It also
provides some general guidelines for tuning, a brief discussion of
application optimization, and suggestions for dealing with
common system performance bottlenecks.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
24/698
xxiv Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Appendices
q Appendix A Interface Card Properties
A sum mary of the characteristics of many of the I/ O interfacecards available for the SPARC systems is given in th is app end ix.
q Appendix B Installing and Configuring SyMON System
Monitor
This appendix discusses how to install, configure, and operate the
Solaris SyMONTM system monitor.
q App end ix C Accoun ting
This appen dix d iscusses the configuration and opera tion of Solaris
system accounting.
q Appen dix D The Cache File System
The configuration, operation and use of CacheFS on both the
client and the server is discussed in this appendix.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
25/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
q Appen dix E IPC Tuneables
While not a d irect part of the course, the setting of the Interprocess
Commun ication (IPC) shared memory and semaph ore parameters
is important. This appendix explains all of the IPC tunable
parameters.
q Appen dix F Performance Monitoring Tools
This appendix is to be used as a reference for the main
per formance monitoring tools that are par t of the Solaris operating
system. Sample output and descriptions of the fields are given for
the following comman ds: sar, vmstat, iostat, mpstat, netstat,
an d nfsstat.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
26/698
xxvi Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
CourseObjectives
Upon completion of this course, you should be able to:
q
Describe the pu rpose and goals of tuningq Describe the common characteristics of system hardware and
software comp onents
q Understand how the major system components operate
q Und erstand the meanings of tuning report elements and h ow they
relate to system performance
q Identify the usual tuning parameters and understand when and
how to set them
q Describe the different types of performance graphs
q Determine whether a problem exists in system hard ware
configurations
q Identify main memory problems and their solutions
q Identify I/ O hardware and file system performance problems and
their solutions
q Identify network p erformance problems and their solutions
q Und erstand CPU scheduling and the functions of threads in theSolaris operating system
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
27/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
Skills Gained by Module
The skills for Solaris System Performance Management are shown in
column 1 of the following m atrix. The black boxes ind icate the main
coverage for a top ic; the gray boxes ind icate the topic is brieflydiscussed.
Module
Skills Gained 1 2 3 4 5 6 7 8 9 10 11
Describe the pu rpose and goals of tuning
Describe the common characteristics of systemhardware and software comp onents
Und erstand how the major system comp onentsoperate
Und erstand the meanings of tun ing reportelements and how they relate to systemperformance
Identify the usu al tuning param eters andund erstand w hen and h ow to set them
Describe the d ifferent typ es of performancegraphs
Determine w hether a problem exists in systemhardware configurations
Identify main memory problems and theirsolutions
Identify netw ork performan ce problems and th eirsolutions
Understand CPU scheduling and the functions ofthread s in the Solaris operating system
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
28/698
x xv iii S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Guidelines for ModulePacing
The following table provides a rough estimate of pacing for this
course:
Module Day 1 Day 2 Day 3 Day 4 Day 5
"About This Course" A.M.
"Introdu ction to PerformanceManagement"
A.M.
"System Monitor ing Tools" P.M.
Processes an d Th read s P.M.
CPU Scheduling A.M.
System Caches P.M.
Memory Tuning A.M.
System Buses P.M.
I/ O Tuning A.M.
UFS Tuning P.M.
Network Tuning A.M.
Performance Tuning Summary P.M.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
29/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
Topics Not Covered
This course does not cover the topics shown on the above overhead .
Many of these topics are covered in other courses offered by Sun
Educational Services.
q Capacity p lanning Covered in ES-360: Planning and Managing
A7000 Intelligent Storage Servers
q General system administration Covered in SA-237: Solaris 7
System Administration I, SA-287: Solaris 7 System A dministration II,
and SA-350: Solaris 2.x Server Administration
q Netw ork ad ministration an d man agement Covered in SA-387:
Solaris TCP/IP Network Administration
q General Solaris problem resolution Covered in ST-350: SunSystems Fault Analysis Workshop
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
30/698
xxx Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
q Data center operat ions Covered in RS-350: Operating Client-Server
Systems
q Disk storage managem ent Covered in SA-347: Volume Manager
With StorEdge A5000
Refer to the Sun Edu cational Services catalog for specific information
and registration.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
31/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
How Prepared AreYou?
To be sure you are p repared to take this course, can you answer yes to
the questions shown on the above overhead?
q If you have had Solaris system administration training or
experience, you will be able to install software, edit cron tables,
start daemons, and perform the lab exercises.
q If you understand the function of the kernel, you will be able to
load kernel modu les and mod ify kernel tunable parameters.
q If you have basic knowledge of Sun server platforms, you will
un derstand the role of the CPU, memory, and buses and their
relationships to system performance.
q If you h ave an u nd erstand ing of system buses, peripheral buses,types of I/ O d evices, and the UN IX file system, you w ill be able to
recognize inappropriate configurations and tune the I/ O
subsystem properly for a given application.
q If you u nd erstand network p rotocols, you w ill be able to tune
them in order to maximize the network bandwidth.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
32/698
x xx ii S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Introductions
Now that you h ave been introdu ced to the course, introdu ce yourself
to each other an d the instructor, add ressing the items shown on the
above overhead.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
33/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
How toUsetheCourseMaterials
To enable you to su cceed in this course, these course materials emp loy
a learning mod el that is comp osed of the following components:
q Course map An overview of the course content ap pears in the
"About This Course" module so you can see how each module fits
into the overall course goal.
q Objectives - What you should be able to accomplish after
completing this module is listed here.
q Relevance This section, which appears in every module,
provides scenarios or questions that introdu ce you to the
information contained in the mod ule and p rovoke you to think
about h ow the m odu le content relates to optimizing system
performance.
q Additional resources Where you can look for more detailed
information, or information on configurations, options, or other
capabilities on the topics covered in the module or appendix.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
34/698
x xx iv S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
q Overhead image Reduced overhead images for the course are
included in the course materials to help you easily follow where
the instructor is at any point in time. Overheads do n ot app ear on
every page.
q Lecture The instru ctor will present information specific to the
topic of the module. This information will help you learn the
knowledge and skills necessary to succeed with the exercises.
q Exercise Lab exercises will give you the op por tun ity to practice
your skills and apply the concepts presented in the lecture.
q Check your progress Modu le objectives are restated , sometimes
in question format, so that before moving on to the next mod ule
you are sure that you can accomplish the objectives of the current
module.
q Think beyond Thought-provoking questions are posed to helpyou app ly the content of the module or pred ict the content in the
next module.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
35/698
About This Course
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
Course Icons and Typographical Conventions
The following icons and typographical conventions are used in this
course to represent various training elements and alternative learning
resources.
Icons
Additional resources Indicates additional reference materials are
available.
Discussion Indicates a small-group or class discussion on th e current
topic is recomm ended at this time.
Note Additional imp ortant, reinforcing, interesting, or sp ecial
information.
!Caution A potential hazard to data or machinery.
Warning Anything that p oses personal danger or irreversible
dam age to data or the operating system.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
36/698
x xx vi S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Typographical Conventions
Courier is used for the names of commands, files, and directories, as
well as on-screen compu ter outp ut. For examp le:
Use ls -al to list all files.system% You have mail.
Courier bold is used for characters and nu mbers that you type. For
example:
system% su
Password:
Courier italic is used for variables and command-line
placeholders tha t are rep laced w ith a real nam e or value. For example:
To d elete a file, type rm filename.
Palatino italics is used for book titles, new word s or terms, or word s
that are emp hasized. For example:
Read Chapter 6 in Users Guide.
These are called class options.
You mustbe root to do this.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
37/698
1-1Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enterp rise Services October 1999,Revision B
Introduction toPerformanceManagement 1
Objectives
Upon completion of this mod ule, you should be able to:
q Explain wh at performance tun ing really means
q Characterize the type of work performed on a system to m ake
meaningful measurements
q Explain p erformance measuremen t terminology
q Describe a typical performance tuning task
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
38/698
1
1-2 Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Relevance
Discussion The following questions are relevant to un derstand ing
the content of this mod ule:
q Why tun e a system?
q How do you know that a system requires tuning?
q How d o you know what to tune?
Additional Resources
Additional resources The following references can provide
add itional d etails on the top ics d iscussed in this mod ule:
q Cockcroft, Adrian, and Richard Pettit. 1998. Sun Performance and
Tuning. Palo Alto: Sun Microsystems Press/ Prentice Hall.
q Wong, Brian L. 1997. Configuration and Capacity Planning for Solaris
Servers. Palo Alto: Sun Microsystems Press/ Prentice Hall.
q Benner, Alan F. 1996. Fibre Channel. McGraw-Hill.
q Catanzaro, Ben. 1994. Multiprocessor System Architectures. SunSoft
Press/ Prentice Hall.
q Cervone, Frank. 1998. Solaris Performance A dministration. McGraw-
Hill.
q Goodheart, Berny, and James Cox. 1994. The Magic Garden
Explained. Prentice-Hall.
q Graham, John. 1995. Solaris 2.x Internals. McGraw -Hill.
q Gunther, Neil. 1998. The Practical Performance Analyst.
McGraw-Hill.
q Raj Jain , Raj. 1991. The Art of Computer System Performance Analysis.John Wiley & Sons, Inc.
q Ridge, Peter M., and David Deming. 1995. The Book of SCSI. N o
Starch Press.
q Solaris 2 Security, Performance and Accounting in the Solaris System
Administration AnswerBook.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
39/698
1
Introduction to Performance Management
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
q SMCC NFS Server Performance Tuning Guide in the Hardware
AnswerBook on the SMCC Supplements CD-ROM.
q The Sun web site http://www.sun.com/sun-on-
net/performance.html and related pages.
q Man p ages for the various comm and s (kstat, ps, lockstat,prtconf, prtdiag, sysdef, adb, ndd, fstyp, mkfs, newfs,
tunefs) and performance tools (sar, vmstat, iostat, mpstat,
nfsstat ,and netstat).
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
40/698
1
1-4 Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
What Is Tuning?
As shown in the overhead image above, tun ing means many things. In
general, however, it means making the most efficient use of the
resources that the system has for the workload that runs on it.
Tuning is aimed at two primary activities:
q Eliminating un necessary work performed by the system
q Taking advantage of system options to tailor the system to its
workload
Almost every tuning activity can be placed into one of these two
categories.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
41/698
1
Introduction to Performance Management
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
BasicTuning Procedure
Basic tun ing p rocedures are quite simple: locate a bottleneck and
remove it. This process is repeated un til performan ce is satisfactory,
however that is measured.
Note that you will never be finished tun ing: there will always be a
bottleneck. If there were none, infinite work could be completed in
zero time. Tuning provides a compromise between cost, the adding of
extra hardware, and performance, the ability to get the systems work
done efficiently.
Before beginning a tuning project, you should have a goal in mind:
what is satisfactory p erformance. A realistic goal is important:
expectations can be set that cannot be met, causing even the most
successful tun ing p roject to be perceived as a failure.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
42/698
1
1-6 Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Conceptual Model of Performance
There are many factors that make a system p erform the way it does.
These include:
q The workload itself. The workload may be I/ O bound, CPU
bound , or netw ork intensive, consisting of short jobs; long jobs;
simple queries; long, complex queries; simple text ed iting;
complex computer-aided design (CAD) simulations; or any
combination of these. It may change during the d ay or w eek.
q A rand om comp onent to the work; that is, how man y queries are
executed, how m any records are looked at, how many u sers are
supported , how much email is generated, how big files are, wh at
the file locations are, and so on. The exact same am oun t and type
of work is rarely d one repeatedly on a system.
q Changes to the system itself. These include new hardware
components, new network access, new software, new algorithms
from patches, or different media.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
43/698
1
Introduction to Performance Management
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
All of these items, and more, work together to make the system a very
dynamic entity, constantly changing and acting differently over time.
When tu ning, you need as m uch information as possible to make
correct decisions about the systems current and future behavior, and
where efficiencies can be obta ined.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
44/698
1
1-8 Solaris S ystem Perform ance M anagem ent
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
WheretoTuneFirst
Take a mom ent to prioritize the areas in the overhead image for tuning
purposes from 1 (most improvement possible) to 4 (least
improvement).
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
45/698
1
Introduction to Performance Management
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
WheretoTune?
As shown on p age -8, the closer to where the work is performed , the
more effect your tu ning w ill have. As system administrators, access to
the first tw o levels is difficult, so the tendency is to concentrate onwhat is accessible.
Tuning at the lower levels is usually an attempt to make the system
sup port the ap plications as efficiently as possible. This usu ally
involves making trad e-offs. Changes tha t benefit on e type of work
may have no effect on, or could hinder, another type of work. If you
tune the system specifically for a particular typ e of work, you must be
prepared to accept these trade-offs.
You a lso need to und erstand the characteristics of the app lications to
know how they use the system before you can tune the system
properly to support them.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
46/698
1
1-10 S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Performance Measurement Terminology
The terminology used in the tuning literature is:
q
Bandwidth
w The peak capacity that cannot be exceeded
w A measurement that ignores overhead
w The best (perfect) case nu mbers
w Someth ing that is never achieved in p ractice; there are always
some inefficiency and overhead
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
47/698
1
Introduction to Performance Management
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Services October 1999, Revision B
q Throughput
w How mu ch work gets done in a specified time interval
w What you really get; that is, how mu ch of the band wid th is
actually used
w A measuremen t that is dep enden t on many factors, including
hardw are, software, human, and random
w A number that is usually impossible to calculate, but can be
approximated
w Maximum throughput if you are measuring how much work
can be d one at 100 percent u tilization
q Response time
w How long the user or requester waits for a request to complete
w Elapsed time for an operation to complete
w Usually the same thing as latency
q Service t ime
w Actual request processing time
w How long a request takes to perform, ignoring any queu eing
delays
w Respon se time, wh ich is the same as service time wh en there is
no qu eueing
q Utilization
w How mu ch of a resource, either aggregate or individual, was
used to do the measured w ork
w The amoun t of throughpu t accomplished by a resource
w A percentage usu ally expressed as a bu sy percentage
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
48/698
1
1-12 S olaris S yst em Performan ce Man agemen t
Copyr ight 1999 Sun Microsystem s, Inc. All Rights Reserved . Enter prise Services October 1999,Revision B
Performance Graphs
The three graphs on this page show the most comm on app earance of
data obtained from performance measuremen ts.
The first two graph s indicate a problem, bu t not its source. You can see
that the work p erformed , measured either as throughp ut or response
time, improves to a certain level, and then begins to drop off. The
drop-off may occur quickly or slowly, but th e system is no longer able
to efficiently perform more work. A resource constraint has been
reached.
The third g raph , a utilization grap h, shows how mu ch capacity is
available for a given resource. When fully used, the resource is nolonger available and requesters have to wait for it.
When there is a d rop-off, as measured by the first two typ es of graph ,
there w ill be a resource that is fu lly u tilized, as measured by the third
grap h. Locating th is resource and increasing its availability is the goal
of tuning.
8/14/2019 Un [1999] Solaris System Performance Management Sa-400 Student Guide
49/698
1
Introduction to Performance Management
Copy right 1999 Sun Microsystem s, Inc. All Rights Reserved. Ent erpr ise Servi