Post on 29-Aug-2014
description
UNIX SHELL IN DBA EVERYDAY
:~> ps -ef | grep pmon | grep -v grep
WHAT IS UNIX?Dance bar in Ashdod, Israel
DO NOT DANCE• examples is not tested on production
• one size does not fits all
• before copy and paste, check manual
• afterwards check manual
• how to get shell • echo $SHELL • how to roll shell • examples • real life • resources
UNIX SHELL IN DBA EVERYDAY
SHELL IS SEXY!
# who | grep -i blonde | date; cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
• who - show who is logged on • grep - print lines matching a pattern • mount - mount a filesystem • touch - change file timestamps • strip - discard symbols from object files • finger - user information lookup program • umount - unmount a filesystem • gasp - a preprocessor for assembly programs • yes - output a string repeatedly until killed
# who | grep -i blonde | date; cd ~; unzip; touch; strip; finger; mount; gasp; yes; uptime; umount; sleep
SHELL IS SEXY!
SHELL IS SEXY!
SHELL IS SEXY!
SHELL IS SEXY!
SHELL IS SEXY!
WHY ME?• read founded and existing scripts • automate systems or databases • manual or scheduled tasks from shell side • monitoring • backups • multiple databases tasks
HOW TO GET SHELL?
CLIENTS • PuTTY
• KiTTY
• SuperPutty
• SecureCRT
http://www.chiark.greenend.org.uk/~sgtatham/putty/
http://www.9bis.net/kitty/
https://code.google.com/p/superputty/
http://www.vandyke.com/products/securecrt/
http://en.wikipedia.org/wiki/Comparison_of_SSH_clients
$ echo $SHELL
SHELL• sh - Bourne shell • ksh - Korn shell • bash - Bourne-Again shell • csh - C shell • tcsh, zsh, rc, es (10+)
http://en.wikipedia.org/wiki/Unix_shell
SHELL sh csh ksh bash tcsh zsh rc es Job control N Y Y Y Y Y N N Aliases N Y Y Y Y Y N N Shell functions Y(1) N Y Y N Y Y Y Command history N Y Y Y Y Y L L Command line editing N N Y Y Y Y L L Vi Command line editing N N Y Y Y(3) Y L L Emacs Command line editing N N Y Y Y Y L L User name look up N Y Y Y Y Y L L Login/Logout watching N N N N Y Y F F Filename completion N Y(1) Y Y Y Y L L Username completion N Y(2) Y Y Y Y L L Hostname completion N Y(2) Y Y Y Y L L History completion N N N Y Y Y L L Builtin artithmetic evaluation N Y Y Y Y Y N N Can follow symbolic links invisibly N N Y Y Y Y N N Periodic command execution N N N N Y Y N N Custom Prompt (easily) N N Y Y Y Y Y Y Underlying Syntax sh csh sh sh csh sh rc rc Freely Available N N N(4) Y Y Y Y Y Can cope with large argument lists Y N Y Y Y Y Y Y Has non-interactive startup file N Y Y(5) Y(5) Y Y N N Has non-login startup file N Y Y(5) Y Y Y N N Has anonymous functions N N N N N N Y Y List Variables N Y Y N Y Y Y Y Full signal trap handling Y N Y Y N Y Y Y Local variables N N Y Y N Y Y Y Exceptions N N N N N N N Y
KSH vs. BASH
$ typeset -l lowercase_only $ lowercase_only="WiBbLe" $ echo $lowercase_only wibble
lowercase_only=$(echo "WiBbLe" | tr "[:upper:]" "[:lower:]")
KSH vs. BASH
a=$(( b+c ))
let a=b+c
KSH vs. BASH
bash-3.1$ echo wibble | read variable bash-3.1$ echo $variable !
bash-3.1$
$ echo wibble | read variable $ echo $variable wibble $
KSH vs. BASH• bash is much easier to set a prompt that displays the current directory • ksh has associative arrays and bash doesn’t • ksh handles loop syntax a bit better • bash handles getting exit codes from pipes in a cleaner way • ksh has the print command which is way better than the echo command • bash has tab completions • ksh has the syntax cd old new which replaces old with new in your
directory and cd over there !
pwd - /foo/bar/barfoo/one/bar/bar/foo/bar cd to /foo/bar/barfoo/two/bar/bar/foo/bar ksh - cd one two bash - cd ../../../../../two/bar/bar/foo/bar
HOW TO ROLL SHELL?
SHEBANG
• #!/bin/ksh • special comment for interpreter • no bang, script run in current shell • shell may be on different hosts in different
locations
PROCESS RUN• cmd1 && cmd2 - run cmd1; if it returns 0 (success), run cmd2
• cmd1 || cmd2 - run cmd1; if it returns non-zero, run cmd2
• cmd1 & cmd2 - run cmd1 and also cmd2
• (ls -1) - run the command "ls -1" in a sub shell
• cmd1 | cmd2 - run cmd1 and output as input to cmd2
PIPEReal UNIX pipe
REDIRECTION
• command > outfile - redirect output to file
• command >> outfile - redirect output and append to file
• command 2> outfile - redirect STDERR
• command &> outfile - redirect STDOUT and STDERR
• tee - redirect STDOUT to file and STDOUT
REDIRECTION# ls # ls > file # ls | tee file # ls | tee –a file # ls | tee file1 file2 file3
mailx –s ‘alert.log errors’ dba@seb.lv < \ tail -10 alert_$ORACLE_SID.log | grep ORA-|\ tee error.log !
FLUSH
# > alert_ORCL.log
IF• [ -a FILE ] True if FILE exists. • [ -d FILE ] True if FILE exists and is a directory. • [ -h FILE ] True if FILE exists and is a symbolic link. • [ -s FILE ] True if FILE exists and has a size greater than zero. • [ -r FILE ] True if FILE exists and is readable. • [ -w FILE ] True if FILE exists and is writable. • [ -x FILE ] True if FILE exists and is executable. • [ -O FILE ] True if FILE exists and is owned by the effective user ID. • [ -G FILE ] True if FILE exists and is owned by the effective group ID. • [ -L FILE ] True if FILE exists and is a symbolic link.
IF• [ -z STRING ] True of the length if "STRING" is zero. • [ -n STRING ] True if the length of "STRING" is non-zero. • [ STRING1 == STRING2 ] True if the strings are equal. • [ STRING1 != STRING2 ] True if the strings are not equal. • [ STRING1 < STRING2 ] True if "STRING1" sorts before
"STRING2" lexicographically in the current locale. • [ STRING1 > STRING2 ] True if "STRING1" sorts after
"STRING2" lexicographically in the current locale. • [ ARG1 OP ARG2 ] "OP" is one of -eq, -ne, -lt, -le, -gt or -ge.
IF
• [ ! EXPR ] True if EXPR is false.
• [ EXPR1 -a EXPR2 ] True if both EXPR1 and EXPR2 are true.
• [ EXPR1 -o EXPR2 ] True if either EXPR1 or EXPR2 is true.
IFif [ -f alert_ORCL.log ]; then echo "alert_ORCL.log exists." fi
if [ "$STATUS" == "PRIMARY" ]; then echo "Database status - ${STATUS}" else echo "Database not in correct mode (${STATUS})" exit 0; fi
if [ "$num_of_beer" -gt "3" -a "$num_of_beer" -lt "5" ]; then echo "You've worked hard enough for today." fi
FOR WHILE• for loop
for FILE in udump/*.trc do tkprof $FILE $FILE.out sort=exeela,fchela,prsela done
• while loop
COUNTER=0 while [ $COUNTER -lt 10 ]; do echo The counter is $COUNTER let COUNTER=COUNTER+1 done
INPUT• get data from user
stty -echo echo -n “Please enter sys password: " read pw stty echo
cont=n echo -n "Do you really want to continue? (y/n) " read cont if [ "$cont" != "y" ]; then echo "Quitting..." exit fi
OPEN PORTSecho "Scanning TCP ports..." for p in {1..1023} do (echo >/dev/tcp/localhost/$p) >/dev/null 2>&1 && echo "$p open" done
TOOLSET• grep - searching for text pattern • awk - script language for text-processing • ps - information on running processes • find - searching for files • sed - stream editor for filtering and transforming text • watch - execute a program periodically • wc - counting lines/chars • cat - join files
http://en.wikipedia.org/wiki/List_of_Unix_programs
GREP
• grep -f badORA.txt alert_PROD.log
badORA.txt ORA-00600 ORA-1653 ORA-00257
• more alert_PROD.log | grep ORA-• grep -r ORA- admin/*
AWK
• more /etc/passwd | grep oracle | awk -F: '{print $3}'
http://www.catonmat.net/blog/awk-one-liners-explained-part-one/
• AWK one liners
awk '1; { print "" }'
awk 'BEGIN { ORS="\n\n" }; 1'
awk '{ print } { print "" }'
• awk '{pattern + action}' {filenames}
• cat /etc/oratab | awk -F: '{if ($1=="ORCL") print $2 }'
ONE LINERS• http://www.pement.org/awk/awk1line.txt
!
!
• http://sed.sourceforge.net/sed1line.txt
# delete trailing whitespace (spaces, tabs) from end of each line awk '{sub(/[ \t]+$/, "")};1' ! # change "scarlet" or "ruby" or "puce" to "red" awk '{gsub(/scarlet|ruby|puce/, "red")}; 1' ! # print any line where field #5 is equal to "abc123" awk '$5 == "abc123"'
http://www.cheat-sheets.org/saved-copy/awk_quickref.pdf
PROCESS KILLING
!
• ps -ef | grep LOCAL=NO | awk '{print "kill -9 " $2}' | sh !
• ps -ef | grep LOCAL=NO | awk '{print $2}' | xargs kill -9 !
• killall oracle
WATCH• watch -tn 0.2 'ps -o cmd -C zabbix_server -C zabbix_agentd'
ALIAS
# alias name=value # alias name='command' # alias name='command arg1 arg2' # alias name='/path/to/script'
ALIAS• alias ..='cd ..' • alias ...='cd ../..' • alias rm='rm -i’ • alias mv="mv -iv" • alias grep="grep -i” • alias mkdir='mkdir -pv’ • alias df='df -H' • alias du='du -ch'
ALIAS• alias mount='mount |column -t’
!
!
• alias path='echo -e ${PATH//:/\\n}'
/dev/sda1 on / type ext4 (rw,errors=remount-ro)proc on /proc type proc (rw,noexec,nosuid,nodev)sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)none on /sys/fs/fuse/connections type fusectl (rw)none on /sys/kernel/debug type debugfs (rw)none on /sys/kernel/security type securityfs (rw)udev on /dev type devtmpfs (rw,mode=0755)devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
/usr/local/sbin/usr/local/bin/usr/sbin/usr/bin/sbin/bin/usr/games
ALIAS• alias ports='netstat -tulanp’
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:55255 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN - tcp 0 0 136.169.15.197:80 210.97.192.220:55544 TIME_WAIT - tcp 0 0 136.169.15.197:80 180.76.6.213:35700 TIME_WAIT - tcp 0 0 136.169.15.197:80 180.76.5.189:29922 TIME_WAIT - tcp 0 416 136.169.15.197:55255 87.110.183.173:24426 ESTABLISHED - tcp6 0 0 :::4949 :::* LISTEN - tcp6 0 0 :::55255 :::* LISTEN - tcp6 0 0 :::25565 :::* LISTEN 15500/java udp 0 0 127.0.0.1:11211 0.0.0.0:* - udp 0 0 0.0.0.0:68 0.0.0.0:* -
ALIAS• alias last="find . -type f -print0 | xargs -0 stat -c'%Y
:%y %12s %n' | sort -nr | cut -d: -f2- | head"
2013-12-10 16:53:07.000000000 +0200 31750 ./.bash_history2013-12-08 13:02:26.000000000 +0200 35 ./.lesshst2013-12-05 18:23:23.000000000 +0200 17962 ./.viminfo2013-10-31 16:51:20.000000000 +0200 6630 ./.ssh/known_hosts2013-08-21 12:58:51.000000000 +0300 273 ./pwd2013-08-20 16:01:42.000000000 +0300 432 ./.vim/.netrwhist2013-08-04 12:47:09.000000000 +0300 111891750 ./vegetarisms.sql.zip2013-01-14 08:18:13.000000000 +0200 163 ./.mysql_history2013-01-13 12:46:18.000000000 +0200 69632 ./skriesim_2013.xls2012-11-14 10:13:49.000000000 +0200 105 ./Scripts/ezermala.sh
ALIAS• alias cpp=“rsync --progress -ravz”
• alias most='du -hsx * | sort -rh | head -10’
• alias usage='du -ch 2> /dev/null |tail -1’
• alias untar='tar -zxvf ’
• alias nocomment='grep -Ev '\''^(#|$)'\'''
ALIASbu() { cp $@ $@.backup-`date +%y%m%d`; echo "`date +%Y-%m-%d` backed up $PWD/$@" >> ~/.backups.log; } alias backup=bu
ALIAS
• alias instances='ps -ef | grep -v grep | grep dbw | cut -d _ -f 3‘
• alias sid=‘env | grep ORACLE_SID’
• alias alert='tail -100 $ORACLE_BASE/admin/$ORACLE_SID/bdump/alert_$ORACLE_SID.log | more’
ALIAS
# alias
# unalias name # unalias -a
# \name
BONUS ALIAS
doskey /macrofile=\doskey.mac
doskey.mac
ls=dir $* /o/w cat=type $* rm=del $* lsl=dir $* /o/p quit=exit
BRACE• cp admin/ORCL/pfile/initORCL.ora{,.bak}
!
• cp admin/ORCL/pfile/initORCL.ora admin/ORCL/pfile/initORCL.ora.bak
• ls *.{trc,log}
HISTORY
SUDO !!Run last command from history
SPYhistory | awk '{print $2}' | sort | uniq -c | sort -n | tail
7 ps 9 exit 9 rm 12 more 19 sqlplus 20 vi 22 pwd 85 cd 109 df 185 ls
SPACE USAGE
SPACE USAGE• du
• find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
• find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
• find . -type d -exec du -sk {} \; | sort -n -k 1
SPACE USAGE# sh get_largest_files.sh /root 10 ![SIZE (BYTES)] [% OF DISK] [OWNER] [LAST MODIFIED ON] [FILE] !913608 0% root 2012-08-25 17:33:33 /root/test.h.gch 73164 0% root 2012-09-18 11:06:02 /root/mem.log 38772 0% root 2012-08-02 15:17:39 /root/install.log 22455 0% root 2012-09-17 21:09:15 /root/.bash_history 4890 0% root 2012-08-25 17:33:33 /root/a.out 170 0% root 2012-09-03 20:10:44 /root/userinfo.sh 385 0% root 2012-09-07 01:02:02 /root/test.sh 23 0% root 2012-08-25 16:58:19 /root/test.h 175 0% root 2012-08-25 17:04:46 /root/test.c.good 134 0% root 2012-08-25 17:32:57 /root/test.c !Total disk size: 23792652288 Bytes Total size occupied by these files: 1053776 Bytes [ 0% of Total Disc Space ] !*** Note: 0% represents less than 1% ***
http://www.thegeekscope.com/linux-script-to-find-largest-files/
LOAD GENERATION
LOAD GENERATION• cat /dev/urandom > md5sum &
• dd if=/dev/zero of=/dev/null
• while true; do /bin/true; done
• yes > /dev/null &
!
• dd if=/dev/random of=/tmp/largefile bs=1024 count=4000000
fulload() { dd if=/dev/zero of=/dev/null | dd if=/dev/zero of=/dev/null & }; fulload; read; killall dd
LOAD GENERATION• stress --cpu 2 --timeout 60
#!/bin/bash duration=120 # seconds instances=4 # cpus endtime=$(($(date +%s) + $duration)) for ((i=0; i<instances; i++)) do while (($(date +%s) < $endtime)); do :; done & done
http://linuxdrops.com/how-to-produce-high-cpu-load-memory-io-or-stress-test-a-linux-server/
FORK BOMB
:(){ :|:& };:
http://en.wikipedia.org/wiki/Fork_bomb
bomb() { bomb | bomb & }; bomb
REAL LIFE
say() { echo $1 | tee -a oraup_stat.txt } !up_db() { say ">>>>>>>> ORACLE_SID=$ORACLE_SID, celju augshaa..." sqlplus "/ as sysdba" <<EOF | tee -a oraup_stat.txt startup exit EOF } !if [ ! -f $ORACLE_HOME/bin/oracle ]; then say "Nav uzstadita ORACLE_HOME." exit else say "ORACLE_HOME=$ORACLE_HOME" fi !if [ $# -eq 0 ]; then say ">>> Nav noraditi parametri" die() fi !while [ $# -ne 0 ]; do export ORACLE_SID=$1 up_db done
TEST ENVIr paceltas sekojoshas datubaazes: ====================================================================================================================== DB SID Memory ORACLE_HOME and Product Version Actual Database Version ---------------------------------------------------------------------------------------------------------------------- APVDIG 1029 M /export/home/oracle/product/10.2.0 10.2.0.4.0 64-bit 10.2.0.4.0 64-bit ARHTEST 545 M /export/home/oracle/product/8.1.7 8.1.7.4.0 32-bit 8.1.7.4.0 32-bit EBRTEST 1895 M /export/home/oracle/product/11.2.0. 11.2.0.2.0 64-bit 11.2.0.2.0 64-bit ECRMUAT 1496 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit IDM 856 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit MBPTEST 1736 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit PRO11 515 M /export/home/oracle/product/11.2.0. 11.2.0.2.0 64-bit 11.2.0.2.0 64-bit RMSTEST 429 M /export/home/oracle/product/10.2.0 10.2.0.4.0 64-bit 10.2.0.4.0 64-bit ROSMET11 856 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit SMTEST 183 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit SYMDW 1511 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit SYMIBM 4903 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit SYMUAT 6503 M /export/home/oracle/product/9.2.0.1 9.2.0.8.0 64-bit 9.2.0.8.0 64-bit TPENS11 828 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit TPENSNEW 588 M /export/home/oracle/product/11.1.0 11.1.0.7.0 64-bit 11.1.0.7.0 64-bit TRISD 960 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit UFTEST 381 M /export/home/oracle/product/10.2.0 10.2.0.4.0 64-bit 10.2.0.4.0 64-bit VOIS11T 884 M /export/home/oracle/product/11.2.0. 11.2.0.3.0 64-bit 11.2.0.3.0 64-bit ====================================================================================================================== Kopaa DB: 19 26472 M
_run_sql() { selekts=`sqlplus "/ as sysdba"<<endsql set head off; set serveroutput on; declare x number; i number; n number; comp varchar2(255); vers varchar2(255); c varchar2(1000); begin select count(*) into x from dba_views where view_name = 'DBA_REGISTRY' and rownum = 1; if x = 0 then -- versijas zem 9.0.0.0.0 c := 'select product, version from product_component_version where product like ''Oracle%Enterprise Edition%'' or product like ''Oracle%Server%'''; else -- versijas virs 9.0.0.0.0 c := 'select comp_name, version from dba_registry where comp_name like ''Oracle%Catalog Views%'''; end if; i := dbms_sql.open_cursor; dbms_sql.parse(i, c, dbms_sql.native); dbms_sql.define_column(i, 1, comp, 255); dbms_sql.define_column(i, 2, vers, 255); n := dbms_sql.execute(i); if dbms_sql.fetch_rows(i) > 0 then dbms_sql.column_value(i, 1, comp); dbms_sql.column_value(i, 2, vers); dbms_output.put_line('>>>'||comp||' '||vers||'<<<'); end if; dbms_sql.close_cursor(i); end; / select '>>>'||length(addr)*4||'<<<' from v\\$process where ROWNUM = 1; exit endsql` echo $selekts }
for pmonproc in `ps -ef -o pid,args | grep pmon | grep -v grep | grep -v sed | \ sed 's/\(.*\) ora_pmon_\(.*\)/\1 \2/' | sort -k 2`; do ! orastr=`_run_sql` orver=`echo $orastr | sed 's/.*Release \([^ ]*\) .*/\1/1'` orabinv=`file $env_orahome/bin/oracle | sed 's/.*ELF \(.*\)-bit .*/\1/'` typeset -L35 env_orahome typeset -R11 orver typeset -R11 prodshortver echo "$dbsid $mem2 M $env_orahome $orver $orabinv-bit\c" ! prodver=`echo $orastr | perl -ne 's/.*?>>>(.*?)<<<.*/\1/ && print'` prodshortver=`echo $prodver | sed 's/.* \([^ ]*\)/\1/'` prodbytes=`echo $orastr | perl -ne 's/.*>>>(.*?)<<<.*/\1/ && print'` vermat=`echo $prodshortver | awk '{if ($0 == $orver) {print 0} else {print 1}}'` echo " $prodshortver $prodbytes-bit" if [ $vermat -ne 0 -o $prodbytes -ne $orabinv ]; the echo " ^ ^ ^ ^ ^ ^ !!!!!!! NESAKRIIT VERSIJAS !!!!!!! ^ ^ ^ ^ ^ ^" fi !!done
env_orahome=`pargs -e "$pid" | grep "ORACLE_HOME=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_ldlib=`pargs -e "$pid" | grep "LD_LIBRARY_PATH" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_path=`pargs -e "$pid" | grep ^PATH= | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_orabase=`pargs -e "$pid" | grep "ORACLE_BASE=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_orasid=`pargs -e "$pid" | grep "ORACLE_SID=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` env_nlsl=`pargs -e "$pid" | grep "NLS_LANG=" | awk '{print $2}' | sed 's/.*=\(.*\)/\1/'` ! mem=`pmap "$pid" 2>/dev/null | tail -1 | awk '{ print $2 }'` mem2=`echo $mem | awk '{split($0,a,"K"); print a[1]}'` mem2=`expr $mem2 / 1024` memtot=`expr $memtot + $mem2` ! if [ "$env_path" -eq "" ]; then env_path="/usr/bin:/etc:/usr/ccs/bin/:/usr/local/bin:/usr/usb:.:"$env_orahome/bin fi PATH="$env_path" LD_LIBRARY_PATH="$env_ldlib" ORACLE_BASE="$env_orabase" ORACLE_HOME="$env_orahome" ORACLE_SID="$env_orasid" NLS_LANG="$env_nlsl" export PATH LD_LIBRARY_PATH ORACLE_BASE ORACLE_HOME ORACLE_SID NLS_LANG
oracle@test-dbs>pargs -e 9298 9298: ora_pmon_IDM envp[0]: SKGP_SPAWN_DIAG_PRE_EXEC_TS= envp[1]: SKGP_HIDDEN_ARGS= envp[2]: SKGP_SPAWN_DIAG_POST_FORK_TS= envp[3]: SKGP_SPAWN_DIAG_PRE_FORK_TS= envp[4]: ORACLE_SPAWNED_PROCESS=1 envp[5]: MANPATH=/usr/share/man:/usr/local/share/man/:/opt/local/share/man/ envp[6]: TERM=vt100 envp[7]: SHELL=/bin/bash envp[8]: NLS_LANG=AMERICAN_AMERICA.AL32UTF8 envp[9]: ODBC_HOME=/export/home/oracle/odbc64 envp[10]: USER=oracle envp[11]: LD_LIBRARY_PATH=/export/home/oracle/odbc64/lib:/export/home/oracle/product/11.2.0.3/lib:/usr/lib:/usr/openwin/lib:/usr/local/lib:/usr/dt/lib envp[12]: ORACLE_SID=IDM envp[13]: ORACLE_BASE=/export/home/oracle envp[14]: TNS_ADMIN=/export/home/oracle/product/11.2.0.3/network/admin envp[15]: PATH= envp[16]: PWD=/export/home/oracle/admin/IDM/pfile envp[17]: EDITOR=vi envp[18]: PS1=\\u@\\h> envp[19]: SHLVL=1 envp[20]: HOME=/export/home/oracle envp[21]: ODBCINI=/export/home/oracle/odbc64/odbc.ini envp[22]: DISPLAY=test-dbs:10.0 envp[23]: ORACLE_HOME=/export/home/oracle/product/11.2.0.3 envp[24]: _=/export/home/oracle/product/11.2.0.3/bin/sqlplus envp[25]: OLDPWD=/export/home/oracle/admin/IDM envp[26]: ORA_NET2_DESC=8,1
TOPP PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 3587 oracle 6522M 6518M cpu6 1 0 0:00:14 4.9% oracle/1 3587 :15 ora_j002_SYMUAT 2011 oracle 495M 479M sleep 59 0 0:01:42 1.7% oracle/71 2011 :42 oracleRMSTEST (LOCAL=NO) 15441 oracle 1501M 1483M sleep 49 0 0:16:26 1.3% oracle/1 15441 :26 oracleECRMUAT (LOCAL=NO) 18128 oracle 6504M 6500M sleep 5 0 1:51:00 1.1% oracle/1 18128 :00 oracleSYMUAT (LOCAL=NO) 27174 oracle 1500M 1483M sleep 48 0 0:01:21 0.5% oracle/1 27174 :22 oracleECRMUAT (LOCAL=NO) 3255 oracle 6510M 6507M sleep 32 0 0:00:02 0.2% oracle/1 23255 :42 ora_dia0_PRO11 20861 oracle 399M 392M sleep 57 0 34:05:31 0.1% tnslsnr/1 20861 :31 /export/home/oracle/product/9.2.0.1/bin/tnslsnr NET9 -inherit 2953 oracle 1500M 1482M sleep 26 0 0:00:13 0.1% oracle/1 2953 :13 oracleECRMUAT (LOCAL=NO) 3591 oracle 6504M 6500M sleep 52 0 0:00:00 0.1% oracle/1 3591 :00 ora_j004_SYMUAT 2014 oracle 446M 417M sleep 59 0 0:00:44 0.1% oracle/15 2014 :44 ora_lgwr_RMSTEST 2010 oracle 444M 426M sleep 1 0 0:01:44 0.1% oracle/258 2010 :44 ora_dbw0_RMSTEST 29354 oracle 1502M 1480M sleep 1 0 8:17:49 0.1% oracle/1 29354 :49 ora_dia0_ECRMUAT 2012 oracle 441M 425M sleep 1 0 0:01:11 0.0% oracle/258 2012 :11 ora_dbw1_RMSTEST 3522 oracle 4904M 4900M sleep 55 0 0:00:00 0.0% oracle/1 3522 :01 ora_j002_SYMIBM 3821 oracle 855M 625M sleep 59 0 0:00:00 0.0% oracle/1 3821 :00 ora_j001_IDM 8178 oracle 1898M 1879M sleep 56 0 0:23:22 0.0% oracle/1 8178 :22 ora_psp0_EBRTEST 8222 oracle 1900M 1881M sleep 59 0 3:50:20 0.0% oracle/1 8222 :21 ora_dia0_EBRTEST 3819 oracle 856M 626M sleep 59 0 0:00:00 0.0% oracle/1 3819 :00 ora_j000_IDM 3777 oracle 1899M 1883M sleep 53 0 0:00:00 0.0% oracle/1 3777 :00 ora_p019_EBRTEST 10501 oracle 6508M 6503M sleep 1 0 0:06:01 0.0% oracle/1 10501 :02 oracleSYMUAT (LOCAL=NO) 3769 oracle 1899M 1883M sleep 52 0 0:00:00 0.0% oracle/1 3769 :00 ora_p015_EBRTEST 3767 oracle 1899M 1883M sleep 52 0 0:00:00 0.0% oracle/1 3767 :00 ora_p014_EBRTEST 3761 oracle 1899M 1883M sleep 53 0 0:00:00 0.0% oracle/1 3761 :00 ora_p011_EBRTEST 3743 oracle 1899M 1883M sleep 54 0 0:00:00 0.0% oracle/1 3743 :00 ora_p002_EBRTEST 3771 oracle 1899M 1883M sleep 53 0 0:00:00 0.0% oracle/1 3771 :00 ora_p016_EBRTEST 3787 oracle 1899M 1883M sleep 54 0 0:00:00 0.0% oracle/1 3787 :00 ora_p024_EBRTEST 3589 oracle 6503M 6500M sleep 59 0 0:00:00 0.0% oracle/1 3589 :00 ora_j003_SYMUAT 3789 oracle 1899M 1883M sleep 54 0 0:00:00 0.0% oracle/1 3789 :00 ora_p025_EBRTEST 3813 oracle 1899M 1883M sleep 55 0 0:00:00 0.0% oracle/1 3813 :00 ora_p037_EBRTEST 3824 oracle 432M 416M sleep 58 0 0:00:00 0.0% oracle/1 3824 :00 ora_j000_RMSTEST Total: 689 processes, 9160 lwps, load averages: 1.63, 1.37, 1.29
#!/usr/bin/perl !$bg = $ARGV[0]; !$pid_list = ''; if ($bg ne '') { foreach (split(/\n/,`ps -ef | grep $bg | grep -v grep`)) { @top=split(/\s+/," ".$_); $pid_list = $pid_list.$top[2].","; print $_."\n"; } } if ($pid_list ne '') {$pid_list = " -p $pid_list";} !while (1 eq 1) { system("clear"); $pid=""; foreach (split(/\n/,`prstat -n 30 $pid_list 1 1`)) { @top=split(/\s+/," ".$_); $ps_line ="";$ps=""; if ($pid eq "PID") { @ps = split(/\n/,`ps -ef | grep $top[1] | grep -v grep`); $ps_line = substr($ps[0],9,6)." ".substr($ps[0],47,length($ps[0])); } else { $pid=$top[1]; } print $_." ".$ps_line."\n"; } `sleep 10`; }
ALERT.LOG• alert.log smart error checkMon Aug 19 17:36:13 2013 Errors in file /export/home/oracle/admin/ORCL/bdump/symuat_j003_20936.trc: ORA-12012: error on auto execute of job 4469813 ORA-20777: Kïûda -20777 ORA-20777: Kïûda 100 ORA-01403: no data foundprocedûrâ 0 KK_FG_RMSACCT_UNprocedûrâ 0 RP_AB_RMSACCT_UN ORA-06512: at "MASTER.ERROR_REG", line 70 ORA-06512: at "MASTER.RP_AB_RMSACCT_UN", line 292 ORA-06512: at line 1
• adrciADR Home = /u01/app/oracle/product/11.1.0/db_1/log/diag/rdbms/orclbi/orclbi: ***************************************************************************** INCIDENT_ID PROBLEM_KEY CREATE_TIME ----------------- ------------------------- --------------------------------- 3808 ORA 603 2007-06-18 21:35:49.322161 -07:00 3807 ORA 600 [4137] 2007-06-18 21:35:47.862114 -07:00 3805 ORA 600 [4136] 2007-06-18 21:35:25.012579 -07:00 3804 ORA 1578 2007-06-18 21:35:08.483156 -07:00 4 rows fetched
function get_errors ($cmd) { foreach (split("\n",$cmd) as $row) { preg_match( "/^(\S+\s+\S+\s+\d+)\s+\d+:\d+:\d+\s+(\d+)/x", $row, $match ); if (isset($match[0])) { $starting = 1; $grouping[] = $group; $group=$row."\n"; } else { if ($starting == 1) $group.=$row."\n"; } } ! $grouping_ora = array(); foreach ($grouping as $row) { $ja=0; if (check_group($row, "ORA-")) { ! if (check_group($row,"ORA-01554")) $ja=1; if (check_group($row,"ORA-3136")) $ja=1; if (check_group($row,"SYSTOR")) $ja=1; if ((check_group($row,"12333"))&&(check_group($row, "ORA-00600"))) $ja=1; if ($ja == 0) {$grouping_ora[]=trim($row);} } $ja=0; if (check_group($row, "Failure to extend rollback segment")) { $grouping_ora[]=trim($row); } $ja=0; if (check_group($row, "WARNING")) { if (check_group($row,"Starting ORACLE instance")) $ja=1; if ($ja == 0) {$grouping_ora[]=trim($row);} } } return $grouping_ora; }
DB BACKUP
• RMAN backup • one script • flexible configuration
# datu bazes statusa noskaidrosana STATUS_SQL="select DATABASE_ROLE FROM v\$database;\nexit\n" STATUS=`echo $STATUS_SQL | sqlplus -S / as sysdba | tail -2| head -1` !echo $STATUS !if [ "$STATUS" == "PRIMARY" ]; then echo "Database status - ${STATUS}" else echo "Database not in correct mode (${STATUS})" exit 0; fi
# # backup konfiguracija # !# --------------------------------------------------------------------------- # Ikdienas backup # FULL - pilnais backup (incremental 0) # INCR - inkrementals backup (incremental 1) !IKDIENAS_BACKUP.1=INCR IKDIENAS_BACKUP.2=INCR IKDIENAS_BACKUP.3=INCR IKDIENAS_BACKUP.4=INCR IKDIENAS_BACKUP.5=INCR IKDIENAS_BACKUP.6=INCR IKDIENAS_BACKUP.0=FULL !!# --------------------------------------------------------------------------- # Papildus backup. Tiek noteikts pec datuma (yyyy.mm.dd) # MONTH - izpildit backup menesea lentaas # QUART - izpildit backup kvartala lentaas # YEAR - izpildit backup gada lentaas !PAPILDUS_BACKUP.2012.05.31=MONTH PAPILDUS_BACKUP.2012.06.30=QUART PAPILDUS_BACKUP.2012.07.31=MONTH PAPILDUS_BACKUP.2012.08.31=MONTH PAPILDUS_BACKUP.2012.09.29=QUART PAPILDUS_BACKUP.2012.10.18=FULL PAPILDUS_BACKUP.2012.10.19=FULL
# check for business backup required PAPILDUS_BACKUP=`cat ${BACKUP_CFG} | grep -v "#" | grep "PAPILDUS_BACKUP.${D_DATE}" | awk 'BEGIN {FS="="} {print $2} '` IKDIENAS_BACKUP=`cat ${BACKUP_CFG} | grep -v "#" | grep "IKDIENAS_BACKUP.${D_WEEK_DAY}" | awk 'BEGIN {FS="="} {print $2} '` !BACKUP_TIME=28; !if [ "${PAPILDUS_BACKUP}" = "MONTH" ] ; then # start business backup BACKUP_TIME=120; BACKUP_TAPE=tdpo_monthly.opt BACKUP_LEVEL=0; BACKUP_TAG=${PAPILDUS_BACKUP}` date '+%Y%m' ` # elif [ "${IKDIENAS_BACKUP}" = "INCR" ] ; then BACKUP_TIME=28; BACKUP_TAPE=tdpo.opt PAPILDUS_BACKUP=INCR BACKUP_LEVEL=1; BACKUP_TAG=${PAPILDUS_BACKUP}` date '+%Y%m%d' ` # elif [ "${IKDIENAS_BACKUP}" = "FULL" ] ; then BACKUP_TIME=28; BACKUP_TAPE=tdpo.opt PAPILDUS_BACKUP=FULL BACKUP_LEVEL=0; BACKUP_TAG=${PAPILDUS_BACKUP}` date '+%Y%m%d' ` # fi
cat > $RUNCMD <<! connect target /; !# Parameters set section CONFIGURE ENCRYPTION FOR DATABASE OFF; CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 28 DAYS; CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE 'sbt_tape' TO 'online_ctl_%F.ctl'; CONFIGURE CHANNEL DEVICE TYPE 'sbt_tape' PARMS 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/${BACKUP_TAPE})' FORMAT 'backup_df_%t_%s_%p_%c' MAXPIECESIZE 100G; CONFIGURE DEVICE TYPE 'sbt_tape' PARALLELISM 2 BACKUP TYPE TO BACKUPSET; CONFIGURE DEFAULT DEVICE TYPE TO SBT; CONFIGURE ARCHIVELOG DELETION POLICY TO SHIPPED TO ALL STANDBY; !run { set duplex 1; backup incremental level ${BACKUP_LEVEL} database tag '${BACKUP_TAG}' KEEP UNTIL TIME 'sysdate + ${BACKUP_TIME}' LOGS; backup current controlfile; backup spfile format 'spfile_%d_%s_%T_dbid%I.ora' tag 'spfile ${BACKUP_TAG}'; backup as compressed backupset archivelog all delete input filesperset=20; } ! !# ${ORACLE_HOME}/bin/rman msgno cmdfile=${RUNCMD} log=${RMANLOG} 1>>${RUNLOG} 2>&1
# log apvienosans un parsesana cat ${RMANLOG} | sed 's/^RMAN-[0-9][0-9][0-9][0-9][0-9]\:.//g' >> ${RUNLOG} cat ${RMANLOG_DEL} | sed 's/^RMAN-[0-9][0-9][0-9][0-9][0-9]\:.//g' >> ${RUNLOG} cat ${RMANLOG_CTRL} | sed 's/^RMAN-[0-9][0-9][0-9][0-9][0-9]\:.//g' >> ${RUNLOG} !!# ORA- kludu pievienosana log faila sakumaa grep ORA- ${RUNLOG} > /tmp/ora_errors.log cat /tmp/ora_errors.log ${RUNLOG} > /tmp/tempfile ; mv /tmp/tempfile ${RUNLOG} !!cat ${RUNLOG} | ${MAILER} -s "BACKUP at `date` on ${ORACLE_SID} DONE `head -1 /tmp/ora_errors.log`" $MAILTO
HA WORKAROUND• AIX hardware • Oracle Database on ASM • Oracle DataGuard + Observer • Full HA tests • Service Request + AIX support
HA WORKAROUND!
• database is PHYSICAL STANDBY (RULE_1) • if dgmgrl show configuration hang (RULE_2) • has errors "DMON: Database ora is still
working on the task." (RULE_3) • manual failover
#setup home . /home/oracle/oracleDB.sh !RUN_NODE=secondary RUN_CHECK_NODE=primary RUN_SLEEP=10 LOG_SLEEP=60 LOG_TAIL=10 RUN_HOME=/home/oracle/local/dg
!INSTANCE_COUNT=`ps -ef | grep master.sh | grep -v grep | grep -v $$ | wc -l` if [ $INSTANCE_COUNT != 0 ]; then date echo "Instance $$" ps -ef | grep master.sh exit 0 fi
echo "RULE 1 check" !STATUS_SQL="select DATABASE_ROLE FROM v\$database;\nexit\n" STATUS=`echo $STATUS_SQL | sqlplus -S / as sysdba | tail -2| head -1` !if [ "$STATUS" == "PHYSICAL STANDBY" ]; then echo "\tOK" echo "\tDatabase status - ${STATUS}" else echo "\tDatabase not correct mode (${STATUS})" exit 0; fi
#!/bin/sh !#setup dgmgrl home . /home/oracle/oracleDB.sh !# start check configuration dgmgrl <<EOL connect sys/master; show configuration verbose; EOL
DG_STATUS.SH
echo "RULE 2 check" !NOW=`date` echo "\tCheck time - ${NOW}" echo "\tSleep time - ${RUN_SLEEP} sec" echo "\tSleeping..." sh $RUN_HOME/DG_status.sh > /dev/null 2>&1 & sleep $RUN_SLEEP !NOW=`date` echo "\tCheck time - ${NOW}" !PROCESS=`ps -ef | grep DG_status.sh | grep -v grep` PROCESS_COUNT=`ps -ef | grep DG_status.sh | grep -v grep | wc -l` !echo "\tProcess count - ${PROCESS_COUNT}" !if [ $PROCESS_COUNT != 0 ]; then echo "\tOK" echo "\t${PROCESS}" else echo "\tDG configuration check run normaly"; exit 0 fi
echo "RULE 3 check" !sleep $LOG_SLEEP !NOW=`date` echo "\tCheck time - ${NOW}" !# DG log faila atrasanas vietas noskaidrosana DIAG_DEST_SQL="select value from v\$parameter where name='diagnostic_dest';\nexit\n" DIAG_DEST=`echo $DIAG_DEST_SQL | sqlplus -S / as sysdba | tail -2| head -1` DGLOG_FILE="${DIAG_DEST}/diag/rdbms/${RUN_NODE}/${RUN_NODE}/trace/drc${RUN_NODE}.log" DGLOG_ERROR="DMON: Database ${RUN_CHECK_NODE} is still working on the task.” !if [ -f $DGLOG_FILE ]; then echo "\tDGLOG_FILE=${DGLOG_FILE}" echo "\tDGLOG_ERROR='${DGLOG_ERROR}'" echo "\tLOG_TAIL=${LOG_TAIL}" else echo "DG log file don't exists - ${DGLOG_FILE} (diagnostic_dest=${DIAG_DEST})" exit 0 fi !LINE=`tail -${LOG_TAIL} ${DGLOG_FILE} | grep "${DGLOG_ERROR}"` LINE_COUNT=`tail -${LOG_TAIL} ${DGLOG_FILE} | grep "${DGLOG_ERROR}" | wc -l` !if [ $LINE_COUNT != 0 ]; then echo "\tOK" echo "\tLINE_COUNT=${LINE_COUNT}" echo "\t${LINE}" else echo "\tNO errors in log file"; exit 0 fi
# te jataisa ir failover dgmgrl <<EOL connect sys/master; failover to ${RUN_NODE} EOL
FAILOVER
RESOURCES
MYSELF
• Signis Vāvere
• AS SEB banka
• database analyst
• http://lv.linkedin.com/in/signis
SQL> set pagesize 0 SQL> set head off SQL> with Mx as (select 60 as MaxWidth from dual) select decode ( sign(floor(MaxWidth /2)-rownum) , 1 , lpad( ' ', floor(MaxWidth /2)-(rownum-1)) || rpad( '*', 2*(rownum-1)+1, ' *') , lpad( '* * *', floor(MaxWidth/2)+3) ) from all_tables ,Mx where rownum < floor(MaxWidth /2) + 6;
HAVE YOURSELF A MERRY LITTLE CHRISTMAS
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * !35 rows selected.