au11qap25gtels2_newevrzfs

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.10.06 19:28:34 =~=~=~=~=~=~=~=~=~=~=~=iduid=0(root) gid=0(root)au11qap25gtels2 $ lustatusBoot Environment Is Active Active Can Copy Name Complete Now On Reboot Delete Status -------------------------- -------- ------ --------- ------ ----------Sol10_u10_pb1211 yes no no yes - zfsroot yes yes yes no - au11qap25gtels2 $ au11qap25gtels2 $ df -hFilesystem size used avail capacity Mounted onrpool/ROOT/zfsroot 66G 8.0G 34G 19% //devices 0K 0K 0K 0% /devicesctfs 0K 0K 0K 0% /system/contractproc 0K 0K 0K 0% /procmnttab 0K 0K 0K 0% /etc/mnttabswap 54G 1.8M 54G 1% /etc/svc/volatileobjfs 0K 0K 0K 0% /system/objectsharefs 0K 0K 0K 0% /etc/dfs/sharetab/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/libc_psr.so.1/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1rpool/ROOT/zfsroot/var 66G 4.0G 34G 11% /varswap 6.0G 256K 6.0G 1% /tmpswap 54G 48K 54G 1% /var/runswap 54G 0K 54G 0% /dev/vx/dmpswap 54G 0K 54G 0% /dev/vx/rdmp/dev/vx/dsk/au11qap25gtels2dg/optappvl 27G 13G 13G 51% /opt/app/dev/vx/dsk/au11qap25gtels2dg/appdatavl 10G 20M 9.4G 1% /var/opt/apprpool 66G 106K 34G 1% /rpoolrpool/appstats 66G 478M 34G 2% /apps/statsrpool/home 66G 569M 34G 2% /homerpool/netbackup 66G 120M 34G 1% /usr/openv/netbackup/logsau11qap25gtels2 $ au11qap25gtels2 $ df -h /var/crashFilesystem size used avail capacity Mounted onrpool/ROOT/zfsroot/var 66G 4.0G 34G 11% /varau11qap25gtels2 $ cd /va ^? ̂?^?bash^?^?bash: not foundau11qap25gtels2 $ c^? bashau11qap25gtels2 $ cd /var/crash/au11qap25gtels2 $ lsau11qap25gtels2au11qap25gtels2 $ rm -rf au11qap25gtels2/ au11qap25gtels2 $ cdau11qap25gtels2 $ zfs mount -aau11qap25gtels2 $ df -hFilesystem size used avail capacity Mounted onrpool/ROOT/zfsroot 66G 8.0G 34G 19% //devices 0K 0K 0K 0% /devicesctfs 0K 0K 0K 0% /system/contractproc 0K 0K 0K 0% /proc

mnttab 0K 0K 0K 0% /etc/mnttabswap 54G 1.8M 54G 1% /etc/svc/volatileobjfs 0K 0K 0K 0% /system/objectsharefs 0K 0K 0K 0% /etc/dfs/sharetab/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/libc_psr.so.1/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1rpool/ROOT/zfsroot/var 66G 4.0G 34G 11% /varswap 6.0G 256K 6.0G 1% /tmpswap 54G 48K 54G 1% /var/runswap 54G 0K 54G 0% /dev/vx/dmpswap 54G 0K 54G 0% /dev/vx/rdmp/dev/vx/dsk/au11qap25gtels2dg/optappvl 27G 13G 13G 51% /opt/app/dev/vx/dsk/au11qap25gtels2dg/appdatavl 10G 20M 9.4G 1% /var/opt/apprpool 66G 106K 34G 1% /rpoolrpool/appstats 66G 478M 34G 2% /apps/statsrpool/home 66G 569M 34G 2% /homerpool/netbackup 66G 120M 34G 1% /usr/openv/netbackup/logs/dev/odm 0K 0K 0K 0% /dev/odmrpool/varcrash 66G 373M 34G 2% /var/crashau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ hastartau11qap25gtels2 $ au11qap25gtels2 $ hastatus -summVCS ERROR V-16-1-10600 Cannot connect to VCS engineVCS WARNING V-16-1-11046 Local system not availableau11qap25gtels2 $ au11qap25gtels2 $ gabconfi g-agabconfig -aGAB Port Memberships===============================================================au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ gabconfig -aGAB Port Memberships===============================================================au11qap25gtels2 $ au11qap25gtels2 $ hastartau11qap25gtels2 $ hastatus -summvxdg listNAME STATE IDau11qap25gtels2dg enabled,cds 1250850288.40.au11qap25gtels2au11qap25gtels2 $ au11qap25gtels2 $ hastatus -summVCS ERROR V-16-1-10600 Cannot connect to VCS engineVCS WARNING V-16-1-11046 Local system not availableau11qap25gtels2 $ gabconfig -aGAB Port Memberships===============================================================au11qap25gtels2 $ au11qap25gtels2 $ pwd/au11qap25gtels2 $ au11qap25gtels2 $ gabconfig -ahastatus -summVCS ERROR V-16-1-10600 Cannot connect to VCS engine

VCS WARNING V-16-1-11046 Local system not availableau11qap25gtels2 $ au11qap25gtels2 $ tail -100 /varcd /etc var/VRTSvcs/log/au11qap25gtels2 $ lsApplication_A.log hacf-err_A.log IPMultiNICB_A.log Netlsnr_B.log notifier-out_A.log Proxy_A.logCmdServer-log_A.log hashadow_A.log Mount_A.log Netlsnr_C.log NotifierMngr_A.log tmpDiskGroup_A.log hashadow-err_A.log MultiNICB_A.log NIC_A.log Oracle_A.log vxfen.logengine_A.log IP_A.log Netlsnr_A.log notifier-err_A.log Phantom_A.logau11qap25gtels2 $ au11qap25gtels2 $ tail -f engine_A.log2012/10/06 15:23:09 VCS ERROR V-16-2-13027 (au11qap26htels2) Resource(vistadb003-lsr) - monitor procedure did not complete within the expected time.2012/10/07 00:00:57 VCS INFO V-16-1-10196 Cluster logger started2012/10/07 00:00:57 VCS NOTICE V-16-1-11022 VCS engine (had) started2012/10/07 00:00:57 VCS NOTICE V-16-1-11050 VCS engine version=5.02012/10/07 00:00:57 VCS NOTICE V-16-1-11051 VCS engine join version=5.0.10.02012/10/07 00:00:57 VCS NOTICE V-16-1-11052 VCS engine pstamp=Veritas-5.0MP1-11/29/06-17:15:002012/10/07 00:00:57 VCS NOTICE V-16-1-10114 Opening GAB library2012/10/07 00:00:57 VCS NOTICE V-16-1-10619 'HAD' starting on: au11qap25gtels22012/10/07 00:00:57 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms2012/10/07 00:01:11 VCS CRITICAL V-16-1-11306 Did not receive cluster membership, manual intervention may be needed for seeding2012/10/07 00:03:44 VCS INFO V-16-1-10077 Received new cluster membership2012/10/07 00:03:44 VCS NOTICE V-16-1-10112 System (au11qap25gtels2) - Membership: 0x3, DDNA: 0x02012/10/07 00:03:44 VCS NOTICE V-16-1-10322 System (Node '1') changed state from UNKNOWN to INITING2012/10/07 00:03:44 VCS NOTICE V-16-1-10086 System au11qap25gtels2 (Node '0') is in Regular Membership - Membership: 0x32012/10/07 00:03:44 VCS NOTICE V-16-1-10086 System (Node '1') is in Regular Membership - Membership: 0x32012/10/07 00:03:44 VCS NOTICE V-16-1-10453 Node: 1 changed name from: '' to: 'au11qap26htels2'2012/10/07 00:03:44 VCS NOTICE V-16-1-10322 System au11qap26htels2 (Node '1') changed state from INITING to CURRENT_DISCOVER_WAIT2012/10/07 00:03:44 VCS NOTICE V-16-1-10322 System au11qap25gtels2 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD2012/10/07 00:03:44 VCS NOTICE V-16-1-10322 System au11qap26htels2 (Node '1') changed state from CURRENT_DISCOVER_WAIT to CURRENT_PEER_WAIT2012/10/07 00:03:46 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying...2012/10/07 00:03:48 VCS NOTICE V-16-1-10032 VxFEN driver available. Local node id=02012/10/07 00:03:48 VCS NOTICE V-16-1-10322 System au11qap25gtels2 (Node '0') changed state from LOCAL_BUILD to RUNNING2012/10/07 00:03:48 VCS NOTICE V-16-1-10322 System au11qap26htels2 (Node '1') changed state from CURRENT_PEER_WAIT to REMOTE_BUILD2012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Application/ApplicationAgent for resource type Application successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/DiskGroup/DiskGroupAgent for resource type DiskGroup successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IP/IPAgent for resource type IP successfully started at Sun Oct 7 00:03:48 2012

2012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/IPMultiNICB/IPMultiNICBAgent for resource type IPMultiNICB successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Mount/MountAgent for resource type Mount successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/MultiNICB/MultiNICBAgent for resource type MultiNICB successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NIC/NICAgent for resource type NIC successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:48 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Netlsnr/NetlsnrAgent for resource type Netlsnr successfully started at Sun Oct 7 00:03:48 20122012/10/07 00:03:49 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/NotifierMngr/NotifierMngrAgent for resource type NotifierMngr successfully started at Sun Oct 7 00:03:49 20122012/10/07 00:03:49 VCS NOTICE V-16-1-10016 Agent /opt/VRTSagents/ha/bin/Oracle/OracleAgent for resource type Oracle successfully started at Sun Oct 7 00:03:49 20122012/10/07 00:03:49 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Phantom/PhantomAgent for resource type Phantom successfully started at Sun Oct 7 00:03:49 20122012/10/07 00:03:49 VCS NOTICE V-16-1-10016 Agent /opt/VRTSvcs/bin/Proxy/ProxyAgent for resource type Proxy successfully started at Sun Oct 7 00:03:49 20122012/10/07 00:03:49 VCS INFO V-16-1-10463 Sending snapshot to node: 12012/10/07 00:03:49 VCS WARNING V-16-1-11155 LLT heartbeat link status changed. Previous status = 0xffffffff; Current status = 0x3.2012/10/07 00:03:50 VCS INFO V-16-1-10304 Resource vistamdb3ebr-ip (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:50 VCS INFO V-16-1-10304 Resource vistadbebr-ip (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:50 VCS INFO V-16-1-10304 Resource vistapdb-ipm (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:50 VCS INFO V-16-1-50135 User root fired command: hares -modify vistamdb001-dgp tempUseFence SCSI3 from localhost2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource clusterservices-ntf (Owner: unknown, Group: ClusterService) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb3-ipm (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadb003-lsr (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002-ocl (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb-ipm (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002-lsr (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadb003-ocl (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource clustermgmt-ipm (Owner: unknown, Group: ClusterService) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001-lsr (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001-ocl (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001_redo001-mnt (Owne

r: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadb003_dbfra001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadb003_redo002-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadb003_data001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadb003_redo001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002_emagent-mnt (Owner: unknown, Group: vistamartemagent-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistadbemagent3-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdb002_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdb001_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-6-15004 (au11qap25gtels2) hatrigger:Failed to send trigger for multinicb_postchange; script doesn't exist2012/10/07 00:03:51 VCS INFO V-16-6-15004 (au11qap25gtels2) hatrigger:Failed to send trigger for multinicb_postchange; script doesn't exist2012/10/07 00:03:51 VCS NOTICE V-16-1-10322 System au11qap26htels2 (Node '1') changed state from REMOTE_BUILD to RUNNING2012/10/07 00:03:51 VCS INFO V-16-1-10297 Resource db-ptm (Owner: unknown, Group: mnics-psg) is online on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group mnics-psg on node au11qap25gtels22012/10/07 00:03:51 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group mnics-psg on node au11qap26htels22012/10/07 00:03:51 VCS NOTICE V-16-1-10447 Group mnics-psg is online on system au11qap25gtels22012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistamdbemagent1-app (Owner: unknown, Group: vistamartemagent-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS INFO V-16-1-10304 Resource vistapdbemagent2-app (Owner: unknown, Group: vistamartemagent-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:51 VCS NOTICE V-16-1-10438 Group vistamartemagent-fsg has been probed on system au11qap25gtels22012/10/07 00:03:51 VCS NOTICE V-16-1-10439 Group (vistamartemagent-fsg) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system au11qap25gtels2 after probe2012/10/07 00:03:51 VCS INFO V-16-1-50135 User root fired command: hares -modify vistapdb002-dgp tempUseFence SCSI3 from localhost2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistamdb001-dgp (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistamdb3ebr-ip (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistadbebr-ip (Owner: unknown

, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistamdb3-ipm (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistamdb-ipm (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource clustermgmt-ipm (Owner: unknown, Group: ClusterService) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistapdb-ipm (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource clusterservices-ntf (Owner: unknown, Group: ClusterService) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistapdb002-lsr (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistamdb001-lsr (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistadb003-ocl (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistadb003-lsr (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistamdb001-ocl (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistapdb002-ocl (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-50135 User root fired command: haclus -modify ClusterUUID gMwC5JRLXZth1TXY from localhost2012/10/07 00:03:52 VCS INFO V-16-1-10304 Resource vistapdb002-dgp (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:52 VCS INFO V-16-1-50135 User root fired command: hares -modify vistadb003-dgp tempUseFence SCSI3 from localhost2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdb001_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdb002_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdb001_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdb002_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-6-15004 (au11qap26htels2) hatrigger:Failed to send trigger for multinicb_postchange; script doesn't exist2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdb001_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdb001_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdb002_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdb002_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdb002_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistadb003_redo001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistadbemagent3-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdb002_emagent-mnt (Owner: unknown, Group: vistamartemagent-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-6-15004 (au11qap26htels2) hatrigger:Failed to send trigger for multinicb_postchange; script doesn't exist2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistadb003_dbfra001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)

2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistadb003_redo002-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistadb003_data001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdb001_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10297 Resource db-ptm (Owner: unknown, Group: mnics-psg) is online on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group mnics-psg on node au11qap25gtels22012/10/07 00:03:53 VCS NOTICE V-16-1-10232 Clearing Restart attribute for group mnics-psg on node au11qap26htels22012/10/07 00:03:53 VCS NOTICE V-16-1-10447 Group mnics-psg is online on system au11qap26htels22012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdbemagent1-app (Owner: unknown, Group: vistamartemagent-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistapdbemagent2-app (Owner: unknown, Group: vistamartemagent-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS NOTICE V-16-1-10438 Group vistamartemagent-fsg has been probed on system au11qap26htels22012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistamdb001-dgp (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:53 VCS INFO V-16-1-10304 Resource vistadb003-dgp (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap25gtels2 (First probe)2012/10/07 00:03:54 VCS INFO V-16-1-10304 Resource vistapdb002-dgp (Owner: unknown, Group: vistamartdb-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:55 VCS NOTICE V-16-1-10438 Group mnics-psg has been probed on system au11qap25gtels22012/10/07 00:03:55 VCS INFO V-16-1-10304 Resource vistadb003-dgp (Owner: unknown, Group: vistamartdb3-fsg) is offline on au11qap26htels2 (First probe)2012/10/07 00:03:57 VCS NOTICE V-16-1-10438 Group mnics-psg has been probed on system au11qap26htels22012/10/07 00:04:50 VCS NOTICE V-16-1-10438 Group vistamartdb-fsg has been probed on system au11qap25gtels22012/10/07 00:04:50 VCS NOTICE V-16-1-10439 Group (vistamartdb-fsg) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system au11qap25gtels2 after probe2012/10/07 00:04:50 VCS NOTICE V-16-1-10438 Group vistamartdb3-fsg has been probed on system au11qap25gtels22012/10/07 00:04:50 VCS NOTICE V-16-1-10439 Group (vistamartdb3-fsg) waiting to probe on all alive nodes in SystemList. Not initiating auto-start online of group on system au11qap25gtels2 after probe2012/10/07 00:04:50 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system au11qap25gtels22012/10/07 00:04:50 VCS INFO V-16-1-50007 Initiating auto-start online of group ClusterService2012/10/07 00:04:50 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group ClusterService on all nodes2012/10/07 00:04:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource clustermgmt-ipm (Owner: unknown, Group: ClusterService) on System au11qap25gtels22012/10/07 00:04:50 VCS NOTICE V-16-1-10301 Initiating Online of Resource clusterservices-ntf (Owner: unknown, Group: ClusterService) on System au11qap25gtels22012/10/07 00:04:50 VCS INFO V-16-1-10298 Resource clustermgmt-ipm (Owner: unknown, Group: ClusterService) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:04:50 VCS INFO V-16-1-10298 Resource clusterservices-ntf (Owner: unknown, Group: ClusterService) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:04:50 VCS NOTICE V-16-1-10447 Group ClusterService is online on system au11qap25gtels2

2012/10/07 00:04:51 VCS INFO V-16-6-15051 (au11qap25gtels2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group2012/10/07 00:04:51 VCS INFO V-16-6-15002 (au11qap25gtels2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart ClusterService successfully2012/10/07 00:04:51 VCS INFO V-16-6-15004 (au11qap25gtels2) hatrigger:Failed to send trigger for postonline; script doesn't exist2012/10/07 00:04:52 VCS NOTICE V-16-1-10438 Group ClusterService has been probed on system au11qap26htels22012/10/07 00:04:52 VCS NOTICE V-16-1-10438 Group vistamartdb3-fsg has been probed on system au11qap26htels22012/10/07 00:04:52 VCS INFO V-16-1-50007 Initiating auto-start online of group vistamartdb3-fsg2012/10/07 00:04:52 VCS INFO V-16-1-10493 Evaluating au11qap26htels2 as potential target node for group vistamartdb3-fsg2012/10/07 00:04:52 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group vistamartdb3-fsg on all nodes2012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003-dgp (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb3ebr-ip (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb3-ipm (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:04:52 VCS NOTICE V-16-1-10438 Group vistamartdb-fsg has been probed on system au11qap26htels22012/10/07 00:04:52 VCS INFO V-16-1-50007 Initiating auto-start online of group vistamartdb-fsg2012/10/07 00:04:52 VCS INFO V-16-1-10493 Evaluating au11qap25gtels2 as potential target node for group vistamartdb-fsg2012/10/07 00:04:52 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group vistamartdb-fsg on all nodes2012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001-dgp (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002-dgp (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadbebr-ip (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb-ipm (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:04:52 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb-ipm (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:04:52 VCS INFO V-16-1-10298 Resource vistamdb3-ipm (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:04:53 VCS INFO V-16-1-10298 Resource vistamdb-ipm (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:04:53 VCS INFO V-16-1-10298 Resource vistapdb-ipm (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:04:53 VCS WARNING V-16-10001-1013 (au11qap25gtels2) DiskGroup:vistamdb001-dgp:online:Diskgroups will be imported with reservations2012/10/07 00:04:53 VCS WARNING V-16-10001-1013 (au11qap26htels2) DiskGroup:vistadb003-dgp:online:Diskgroups will be imported with reservations2012/10/07 00:04:56 VCS INFO V-16-1-10298 Resource vistamdb3ebr-ip (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:04:56 VCS INFO V-16-1-10298 Resource vistadbebr-ip (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:02 VCS NOTICE V-16-10001-1009 (au11qap26htels2) DiskGroup:vistadb003-dgp:online:vxdg import succeeded on Disk Group oradb003dg2012/10/07 00:05:10 VCS NOTICE V-16-10001-1009 (au11qap25gtels2) DiskGroup:vistamdb001-dgp:online:vxdg import succeeded on Disk Group oradb001dg2012/10/07 00:05:21 VCS NOTICE V-16-10001-1010 (au11qap26htels2) DiskGroup:vista

db003-dgp:online:Volumes in Disk Group oradb003dg are started. Any mirrors are updated in background2012/10/07 00:05:23 VCS INFO V-16-1-10298 Resource vistadb003-dgp (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadbemagent3-mnt (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:05:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003_redo002-mnt (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:05:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003_redo001-mnt (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:05:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003_dbfra001-mnt (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:05:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003_data001-mnt (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:05:25 VCS INFO V-16-1-10298 Resource vistadb003_redo001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:25 VCS INFO V-16-1-10298 Resource vistadbemagent3-mnt (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:25 VCS INFO V-16-1-10298 Resource vistadb003_data001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:26 VCS INFO V-16-1-10298 Resource vistadb003_dbfra001-mnt (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:26 VCS INFO V-16-1-10298 Resource vistadb003_redo002-mnt (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003-ocl (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:05:27 VCS INFO V-16-20002-104 (au11qap26htels2) Oracle:vistadb003-ocl:online:The IPC library is not from Veritas. 2012/10/07 00:05:38 VCS NOTICE V-16-10001-1010 (au11qap25gtels2) DiskGroup:vistamdb001-dgp:online:Volumes in Disk Group oradb001dg are started. Any mirrors are updated in background2012/10/07 00:05:40 VCS WARNING V-16-10001-1013 (au11qap25gtels2) DiskGroup:vistapdb002-dgp:online:Diskgroups will be imported with reservations2012/10/07 00:05:41 VCS INFO V-16-1-50133 User admin has logged in from 10.117.218.522012/10/07 00:05:43 VCS NOTICE V-16-10001-1009 (au11qap25gtels2) DiskGroup:vistapdb002-dgp:online:vxdg import succeeded on Disk Group oradb002dg2012/10/07 00:05:43 VCS NOTICE V-16-10001-1010 (au11qap25gtels2) DiskGroup:vistapdb002-dgp:online:Volumes in Disk Group oradb002dg are started. Any mirrors are updated in background2012/10/07 00:05:44 VCS INFO V-16-1-10298 Resource vistamdb001-dgp (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:05:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:05:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:05:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels2

2012/10/07 00:05:44 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:05:47 VCS INFO V-16-1-10298 Resource vistamdb001_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:47 VCS INFO V-16-1-10298 Resource vistamdb001_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:47 VCS INFO V-16-1-10298 Resource vistamdb001_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:48 VCS INFO V-16-1-10298 Resource vistamdb001_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:48 VCS INFO V-16-1-10298 Resource vistamdb001_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:05:48 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001-ocl (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:05:57 VCS NOTICE V-16-20002-21 (au11qap26htels2) Oracle:vistadb003-ocl:online:Oracle startup returned the output +--------------------------------------------------------------------+LD_LIBRARY_PATH - /usr/lib:/opt/app/oracle/product/10.2.0/db_1/lib:ORACLE instance started.

Total System Global Area 1.3640E+10 bytesFixed Size 2075512 bytesVariable Size 570430600 bytesDatabase Buffers 1.3053E+10 bytesRedo Buffers 14696448 bytesDatabase mounted.Database opened.+====================================================================+2012/10/07 00:05:59 VCS INFO V-16-1-10298 Resource vistadb003-ocl (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:05:59 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistadb003-lsr (Owner: unknown, Group: vistamartdb3-fsg) on System au11qap26htels22012/10/07 00:06:00 VCS NOTICE V-16-10001-40 (au11qap26htels2) Netlsnr:vistadb003-lsr:online:lsnrctl returned the following output+--------------------------------------------------------------------+LD_LIBRARY_PATH - /usr/lib:

LSNRCTL for Solaris: Version 10.2.0.5.0 - Production on 07-OCT-2012 00:05:59

Copyright (c) 1991, 2010, Oracle. All rights reserved.

Starting /opt/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Solaris: Version 10.2.0.5.0 - ProductionSystem parameter file is /var/opt/oracle/listener.oraLog messages written to /opt/app/oracle/product/10.2.0/db_1/network/log/listener_ivdb1cp.logListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.117.221.151)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=nsavmprd1dbs03.in.telstra.com.au)(PORT=1521)))STATUS of the LISTENER------------------------Alias LISTENER_IVDB1CPVersion TNSLSNR for Solaris: Version 10.2.0.5.0 - ProductionStart Date 07-OCT-2012 00:05:59Uptime 0 days 0 hr. 0 min. 0 sec

Trace Level offSecurity ON: Local OS AuthenticationSNMP OFFListener Parameter File /var/opt/oracle/listener.oraListener Log File /opt/app/oracle/product/10.2.0/db_1/network/log/listener_ivdb1cp.logListening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.117.221.151)(PORT=1521)))Services Summary...Service IVDB1CP has 1 instance(s). Instance IVDB1CP, status UNKNOWN, has 1 handler(s) for this service...The command completed successfully+====================================================================+2012/10/07 00:06:00 VCS INFO V-16-20002-104 (au11qap25gtels2) Oracle:vistamdb001-ocl:online:The IPC library is not from Veritas. 2012/10/07 00:06:01 VCS INFO V-16-1-10298 Resource vistadb003-lsr (Owner: unknown, Group: vistamartdb3-fsg) is online on au11qap26htels2 (VCS initiated)2012/10/07 00:06:01 VCS NOTICE V-16-1-10447 Group vistamartdb3-fsg is online on system au11qap26htels22012/10/07 00:06:02 VCS INFO V-16-6-15051 (au11qap26htels2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group2012/10/07 00:06:02 VCS INFO V-16-6-15002 (au11qap26htels2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart vistamartdb3-fsg successfully2012/10/07 00:06:02 VCS INFO V-16-6-15004 (au11qap26htels2) hatrigger:Failed to send trigger for postonline; script doesn't exist2012/10/07 00:06:08 VCS INFO V-16-1-10298 Resource vistapdb002-dgp (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:08 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:10 VCS INFO V-16-1-10298 Resource vistapdb002_redo001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:11 VCS INFO V-16-1-10298 Resource vistapdb002_redo002-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:11 VCS INFO V-16-1-10298 Resource vistapdb002_inx001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:11 VCS INFO V-16-1-10298 Resource vistapdb002_dbfra001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:11 VCS INFO V-16-1-10298 Resource vistapdb002_data001-mnt (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:11 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002-ocl (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:11 VCS INFO V-16-20002-104 (au11qap25gtels2) Oracle:vistapdb002-ocl:online:The IPC library is not from Veritas. 2012/10/07 00:06:22 VCS NOTICE V-16-20002-21 (au11qap25gtels2) Oracle:vistapdb002-ocl:online:Oracle startup returned the output

+--------------------------------------------------------------------+LD_LIBRARY_PATH - /usr/lib:/opt/app/oracle/product/10.2.0/db_1/lib:ORACLE instance started.

Total System Global Area 1073741824 bytesFixed Size 2057624 bytesVariable Size 645929576 bytesDatabase Buffers 419430400 bytesRedo Buffers 6324224 bytesDatabase mounted.Database opened.+====================================================================+2012/10/07 00:06:23 VCS NOTICE V-16-20002-21 (au11qap25gtels2) Oracle:vistamdb001-ocl:online:Oracle startup returned the output +--------------------------------------------------------------------+LD_LIBRARY_PATH - /usr/lib:/opt/app/oracle/product/10.2.0/db_1/lib:ORACLE instance started.

Total System Global Area 1.3959E+10 bytesFixed Size 2076128 bytesVariable Size 536875552 bytesDatabase Buffers 1.3405E+10 bytesRedo Buffers 14696448 bytesDatabase mounted.Database opened.+====================================================================+2012/10/07 00:06:23 VCS INFO V-16-1-10298 Resource vistamdb001-ocl (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdb001-lsr (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:23 VCS INFO V-16-1-10298 Resource vistapdb002-ocl (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:23 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002-lsr (Owner: unknown, Group: vistamartdb-fsg) on System au11qap25gtels22012/10/07 00:06:25 VCS NOTICE V-16-10001-40 (au11qap25gtels2) Netlsnr:vistamdb001-lsr:online:lsnrctl returned the following output+--------------------------------------------------------------------+LD_LIBRARY_PATH - /usr/lib:




TNSLSNR for Solaris: Version 10.2.0.5.0 - ProductionSystem parameter file is /var/opt/oracle/listener.oraLog messages written to /opt/app/oracle/product/10.2.0/db_1/network/log/listener_ivdb1ap.logListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.117.221.136)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=nsavmprd1dbs01.in.telstra.com.au)(PORT=1521)))STATUS of the LISTENER------------------------Alias LISTENER_IVDB1APVersion TNSLSNR for Solaris: Version 10.2.0.5.0 - ProductionStart Date 07-OCT-2012 00:06:25Uptime 0 days 0 hr. 0 min. 0 sec

Trace Level offSecurity ON: Local OS AuthenticationSNMP OFFListener Parameter File /var/opt/oracle/listener.oraListener Log File /opt/app/oracle/product/10.2.0/db_1/network/log/listener_ivdb1ap.logListening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.117.221.136)(PORT=1521)))Services Summary...Service IVDB1AP has 1 instance(s). Instance IVDB1AP, status UNKNOWN, has 1 handler(s) for this service...The command completed successfully+====================================================================+2012/10/07 00:06:25 VCS NOTICE V-16-10001-40 (au11qap25gtels2) Netlsnr:vistapdb002-lsr:online:lsnrctl returned the following output+--------------------------------------------------------------------+LD_LIBRARY_PATH - /usr/lib:




TNSLSNR for Solaris: Version 10.2.0.5.0 - ProductionSystem parameter file is /var/opt/oracle/listener.oraLog messages written to /opt/app/oracle/product/10.2.0/db_1/network/log/listener_vpdb1ap.logListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.117.221.137)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=nsavmprd1dbs02.in.telstra.com.au)(PORT=1521)))STATUS of the LISTENER------------------------Alias LISTENER_VPDB1APVersion TNSLSNR for Solaris: Version 10.2.0.5.0 - ProductionStart Date 07-OCT-2012 00:06:25Uptime 0 days 0 hr. 0 min. 0 secTrace Level offSecurity ON: Local OS AuthenticationSNMP OFFListener Parameter File /var/opt/oracle/listener.oraListener Log File /opt/app/oracle/product/10.2.0/db_1/network/log/listener_vpdb1ap.logListening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.117.221.137)(PORT=1521)))Services Summary...Service VPDB1AP has 1 instance(s). Instance VPDB1AP, status UNKNOWN, has 1 handler(s) for this service...The command completed successfully+====================================================================+2012/10/07 00:06:26 VCS INFO V-16-1-10298 Resource vistamdb001-lsr (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:26 VCS INFO V-16-1-10298 Resource vistapdb002-lsr (Owner: unknown, Group: vistamartdb-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:26 VCS NOTICE V-16-1-10447 Group vistamartdb-fsg is online on system au11qap25gtels22012/10/07 00:06:26 VCS WARNING V-16-1-50045 Initiating online of parent group vistamartemagent-fsg, PM will select the best node

2012/10/07 00:06:26 VCS INFO V-16-1-10493 Evaluating au11qap25gtels2 as potential target node for group vistamartemagent-fsg2012/10/07 00:06:26 VCS NOTICE V-16-1-10233 Clearing Restart attribute for group vistamartemagent-fsg on all nodes2012/10/07 00:06:26 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdb002_emagent-mnt (Owner: unknown, Group: vistamartemagent-fsg) on System au11qap25gtels22012/10/07 00:06:27 VCS INFO V-16-6-15051 (au11qap25gtels2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group2012/10/07 00:06:28 VCS INFO V-16-6-15002 (au11qap25gtels2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart vistamartdb-fsg successfully2012/10/07 00:06:28 VCS INFO V-16-6-15004 (au11qap25gtels2) hatrigger:Failed to send trigger for postonline; script doesn't exist2012/10/07 00:06:30 VCS INFO V-16-1-10298 Resource vistapdb002_emagent-mnt (Owner: unknown, Group: vistamartemagent-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistapdbemagent2-app (Owner: unknown, Group: vistamartemagent-fsg) on System au11qap25gtels22012/10/07 00:06:30 VCS NOTICE V-16-1-10301 Initiating Online of Resource vistamdbemagent1-app (Owner: unknown, Group: vistamartemagent-fsg) on System au11qap25gtels22012/10/07 00:06:45 VCS INFO V-16-10001-4 (au11qap25gtels2) Application:vistamdbemagent1-app:online:Executed /opt/VRTSvcs/bin/scripts/emctl1 as user oracle2012/10/07 00:06:45 VCS INFO V-16-10001-4 (au11qap25gtels2) Application:vistapdbemagent2-app:online:Executed /opt/VRTSvcs/bin/scripts/emctl2 as user oracle2012/10/07 00:06:46 VCS INFO V-16-2-13001 (au11qap25gtels2) Resource(vistamdbemagent1-app): Output of the completed operation (online) stty: : No such device or addressstty: : No such device or address#################################################################### This system is for the use of authorized users only. ## Individuals using this computer system without authority, or in ## excess of their authority, are subject to having all of their ## activities on this system monitored and recorded by system ## personnel. ## ## In the course of monitoring individuals improperly using this ## system, or in the course of system maintenance, the activities ## of authorized users may also be monitored. ## ## Anyone using this system expressly consents to such monitoring ## and is advised that if such monitoring reveals possible ## evidence of criminal activity, system personnel may provide the ## evidence of such monitoring to law enforcement officials. ####################################################################

WARNING: Access to this computer system is limited to authorised users only.Unauthorised users may be subject to prosecution under the CrimesAct or State legislation.

Please note, ALL CUSTOMER DETAILS are confidential and must not bedisclosed.

stty: : No such device or addressstty: : No such device or address �H �H �H �H �H �H �H �H �HSummary:-------------------------Server Name: au11qap25gtels2

Aliases: au11qap26htels2 Past PWPs: Previous PWP's that are now completed.Current PWP: PWP for any current activityCurrent BAU DBs: databases already in BAU if anyCurrent Build Status: Build/Apps Load/Awaiting final OQA/etc

Support Level: Base ExtendedApplication Support Hours: 24x7

Installed RDBMS Versions and Homes:/opt/app/oracle/product/10.2.0/db_1 -- 10.2.0.4/opt/app/oracle/product/9.2.0/db_2 2012/10/07 00:06:46 VCS INFO V-16-2-13001 (au11qap25gtels2) Resource(vistapdbemagent2-app): Output of the completed operation (online) stty: : No such device or addressstty: : No such device or address#################################################################### This system is for the use of authorized users only. ## Individuals using this computer system without authority, or in ## excess of their authority, are subject to having all of their ## activities on this system monitored and recorded by system ## personnel. ## ## In the course of monitoring individuals improperly using this ## system, or in the course of system maintenance, the activities ## of authorized users may also be monitored. ## ## Anyone using this system expressly consents to such monitoring ## and is advised that if such monitoring reveals possible ## evidence of criminal activity, system personnel may provide the ## evidence of such monitoring to law enforcement officials. ####################################################################

WARNING: Access to this computer system is limited to authorised users only.Unauthorised users may be subject to prosecution under the CrimesAct or State legislation.

Please note, ALL CUSTOMER DETAILS are confidential and must not bedisclosed.

stty: : No such device or addressstty: : No such device or address �H �H �H �H �H �H �H �H �HSummary:-------------------------Server Name: au11qap25gtels2 Aliases: au11qap26htels2 Past PWPs: Previous PWP's that are now completed.Current PWP: PWP for any current activityCurrent BAU DBs: databases already in BAU if anyCurrent Build Status: Build/Apps Load/Awaiting final OQA/etc

Support Level: Base ExtendedApplication Support Hours: 24x7

Installed RDBMS Versions and Homes:/opt/app/oracle/product/10.2.0/db_1 -- 10.2.0.4/opt/app/oracle/product/9.2.0/db_2 2012/10/07 00:06:48 VCS INFO V-16-1-10298 Resource vistapdbemagent2-app (Owner: unknown, Group: vistamartemagent-fsg) is online on au11qap25gtels2 (VCS initiated)

2012/10/07 00:06:48 VCS INFO V-16-1-10298 Resource vistamdbemagent1-app (Owner: unknown, Group: vistamartemagent-fsg) is online on au11qap25gtels2 (VCS initiated)2012/10/07 00:06:48 VCS NOTICE V-16-1-10447 Group vistamartemagent-fsg is online on system au11qap25gtels22012/10/07 00:06:49 VCS INFO V-16-6-15051 (au11qap25gtels2) nfs_restart:nfs_restart trigger did not do anything as there is no NFS/NFSLock/Share resource in the group2012/10/07 00:06:49 VCS INFO V-16-6-15002 (au11qap25gtels2) hatrigger:hatrigger executed /opt/VRTSvcs/bin/triggers/nfs_restart vistamartemagent-fsg successfully2012/10/07 00:06:49 VCS INFO V-16-6-15004 (au11qap25gtels2) hatrigger:Failed to send trigger for postonline; script doesn't exist2012/10/07 00:07:01 VCS INFO V-16-1-50133 User admin has logged in from 10.117.218.44

^Cau11qap25gtels2 $ pwd/var/VRTSvcs/logau11qap25gtels2 $ iduid=0(root) gid=0(root)au11qap25gtels2 $ pwd/var/VRTSvcs/logau11qap25gtels2 $ exitexitau11qap25gtels2 $ exit iduid=0(root) gid=0(root)au11qap25gtels2 $ exitau11qap25gtels2 $ iduid=36844(d629807) gid=5004(admin)au11qap25gtels2 $ exitau11qap26htels2 Y N OFFLINE B mnics-psg au11qap25gtels2 Y N ONLINE B mnics-psg au11qap26htels2 Y N ONLINE B vistamartdb-fsg au11qap25gtels2 Y N STARTING|PARTIALB vistamartdb-fsg au11qap26htels2 Y N OFFLINE B vistamartdb3-fsg au11qap25gtels2 Y N OFFLINE B vistamartdb3-fsg au11qap26htels2 Y N ONLINE B vistamartemagent-fsg au11qap25gtels2 Y N OFFLINE B vistamartemagent-fsg au11qap26htels2 Y N OFFLINE

-- RESOURCES ONLINING-- Group Type Resource System IState

E vistamartdb-fsg Oracle vistamdb001-ocl au11qap25gtels2 W_ONLINEE vistamartdb-fsg Oracle vistapdb002-ocl au11qap25gtels2 W_ONLINEau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ hastatus -summ

-- SYSTEM STATE-- System State Frozen

A au11qap25gtels2 RUNNING 0 A au11qap26htels2 RUNNING 0

-- GROUP STATE-- Group System Probed AutoDisabled State

B ClusterService au11qap25gtels2 Y N ONLINE B ClusterService au11qap26htels2 Y N OFFLINE B mnics-psg au11qap25gtels2 Y N ONLINE B mnics-psg au11qap26htels2 Y N ONLINE B vistamartdb-fsg au11qap25gtels2 Y N ONLINE B vistamartdb-fsg au11qap26htels2 Y N OFFLINE B vistamartdb3-fsg au11qap25gtels2 Y N OFFLINE B vistamartdb3-fsg au11qap26htels2 Y N ONLINE B vistamartemagent-fsg au11qap25gtels2 Y N OFFLINE|STARTINGB vistamartemagent-fsg au11qap26htels2 Y N OFFLINE


E vistamartemagent-fsg Mount vistapdb002_emagent-mnt au11qap25gtels2 W_ONLINEau11qap25gtels2 $ hastatus -summ




B ClusterService au11qap25gtels2 Y N ONLINE B ClusterService au11qap26htels2 Y N OFFLINE B mnics-psg au11qap25gtels2 Y N ONLINE B mnics-psg au11qap26htels2 Y N ONLINE B vistamartdb-fsg au11qap25gtels2 Y N ONLINE

B vistamartdb-fsg au11qap26htels2 Y N OFFLINE B vistamartdb3-fsg au11qap25gtels2 Y N OFFLINE B vistamartdb3-fsg au11qap26htels2 Y N ONLINE B vistamartemagent-fsg au11qap25gtels2 Y N STARTING|PARTIALB vistamartemagent-fsg au11qap26htels2 Y N OFFLINE


E vistamartemagent-fsg Application vistamdbemagent1-app au11qap25gtels2 W_ONLINEE vistamartemagent-fsg Application vistapdbemagent2-app au11qap25gtels2 W_ONLINEau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ hastatus -summ




B ClusterService au11qap25gtels2 Y N ONLINE B ClusterService au11qap26htels2 Y N OFFLINE B mnics-psg au11qap25gtels2 Y N ONLINE B mnics-psg au11qap26htels2 Y N ONLINE B vistamartdb-fsg au11qap25gtels2 Y N ONLINE B vistamartdb-fsg au11qap26htels2 Y N OFFLINE B vistamartdb3-fsg au11qap25gtels2 Y N OFFLINE B vistamartdb3-fsg au11qap26htels2 Y N ONLINE B vistamartemagent-fsg au11qap25gtels2 Y N STARTING|PARTIALB vistamartemagent-fsg au11qap26htels2 Y N OFFLINE


E vistamartemagent-fsg Application vistamdbemagent1-app au11qap25gtels2 W_ONLINEE vistamartemagent-fsg Application vistapdbemagent2-app au11qap25gtels2 W_ONLINEau11qap25gtels2 $ au11qap25gtels2 $ hastatus -summ






E vistamartemagent-fsg Application vistamdbemagent1-app au11qap25gtels2 W_ONLINEE vistamartemagent-fsg Application vistapdbemagent2-app au11qap25gtels2 W_ONLINEau11qap25gtels2 $ hastatus -summ




B ClusterService au11qap25gtels2 Y N ONLINE

B ClusterService au11qap26htels2 Y N OFFLINE B mnics-psg au11qap25gtels2 Y N ONLINE B mnics-psg au11qap26htels2 Y N ONLINE B vistamartdb-fsg au11qap25gtels2 Y N ONLINE B vistamartdb-fsg au11qap26htels2 Y N OFFLINE B vistamartdb3-fsg au11qap25gtels2 Y N OFFLINE B vistamartdb3-fsg au11qap26htels2 Y N ONLINE B vistamartemagent-fsg au11qap25gtels2 Y N STARTING|PARTIALB vistamartemagent-fsg au11qap26htels2 Y N OFFLINE


E vistamartemagent-fsg Application vistamdbemagent1-app au11qap25gtels2 W_ONLINEE vistamartemagent-fsg Application vistapdbemagent2-app au11qap25gtels2 W_ONLINEau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ hastatus -summ






E vistamartemagent-fsg Application vistamdbemagent1-app au11qap25gtels2 W_ONLINEE vistamartemagent-fsg Application vistapdbemagent2-app au11qap25gtels2 W_ONLINEau11qap25gtels2 $ au11qap25gtels2 $ hastatus -summ






E vistamartemagent-fsg Application vistamdbemagent1-app au11qap25gtels2 W_ONLINEE vistamartemagent-fsg Application vistapdbemagent2-app au11qap25gtels2 W_ONLINEau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ hastatus -summ




B ClusterService au11qap25gtels2 Y N ONLINE B ClusterService au11qap26htels2 Y N OFFLINE B mnics-psg au11qap25gtels2 Y N ONLINE B mnics-psg au11qap26htels2 Y N ONLINE B vistamartdb-fsg au11qap25gtels2 Y N ONLINE B vistamartdb-fsg au11qap26htels2 Y N OFFLINE B vistamartdb3-fsg au11qap25gtels2 Y N OFFLINE B vistamartdb3-fsg au11qap26htels2 Y N ONLINE B vistamartemagent-fsg au11qap25gtels2 Y N ONLINE B vistamartemagent-fsg au11qap26htels2 Y N OFFLINE au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ df -hFilesystem size used avail capacity Mounted onrpool/ROOT/zfsroot 66G 8.0G 34G 19% //devices 0K 0K 0K 0% /devicesctfs 0K 0K 0K 0% /system/contractproc 0K 0K 0K 0% /procmnttab 0K 0K 0K 0% /etc/mnttabswap 38G 1.8M 38G 1% /etc/svc/volatileobjfs 0K 0K 0K 0% /system/objectsharefs 0K 0K 0K 0% /etc/dfs/sharetab/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/libc_psr.so.1/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1rpool/ROOT/zfsroot/var 66G 4.0G 34G 11% /varswap 6.0G 312K 6.0G 1% /tmpswap 38G 48K 38G 1% /var/runswap 38G 0K 38G 0% /dev/vx/dmpswap 38G 0K 38G 0% /dev/vx/rdmp/dev/vx/dsk/au11qap25gtels2dg/optappvl 27G 13G 13G 51% /opt/app/dev/vx/dsk/au11qap25gtels2dg/appdatavl 10G 20M 9.4G 1% /var/opt/apprpool 66G 106K 34G 1% /rpoolrpool/appstats 66G 479M 34G 2% /apps/statsrpool/home 66G 569M 34G 2% /home

rpool/netbackup 66G 120M 34G 1% /usr/openv/netbackup/logs/dev/odm 0K 0K 0K 0% /dev/odmrpool/varcrash 66G 373M 34G 2% /var/crash/dev/vx/dsk/oradb001dg/redo001vl 21G 1.1G 20G 6% /ora/db001/redo001/dev/vx/dsk/oradb001dg/data001vl 2.5T 692G 1.8T 28% /ora/db001/data001/dev/vx/dsk/oradb001dg/inx001vl 1.5T 609G 971G 39% /ora/db001/inx001/dev/vx/dsk/oradb001dg/dbfra001vl 1.2T 75G 1.1T 7% /ora/db001/dbfra001/dev/vx/dsk/oradb001dg/redo002vl 21G 1.1G 20G 6% /ora/db001/redo002/dev/vx/dsk/oradb002dg/redo001vl 2.0G 231M 1.8G 12% /ora/db002/redo001/dev/vx/dsk/oradb002dg/data001vl 16G 7.0G 8.9G 45% /ora/db002/data001/dev/vx/dsk/oradb002dg/inx001vl 6.0G 17M 5.9G 1% /ora/db002/inx001/dev/vx/dsk/oradb002dg/redo002vl 2.0G 231M 1.8G 12% /ora/db002/redo002/dev/vx/dsk/oradb002dg/dbfra001vl 10G 19M 9.9G 1% /ora/db002/dbfra001/dev/vx/dsk/oradb002dg/optapphavl 5.0G 2.5G 2.4G 52% /opt/app/ha/oracle/emagentau11qap25gtels2 $ au11qap25gtels2 $ cd /var/crash/PATCHES/au11qap25gtels2 $ ls121430-80 apply-NIC-patch-to-LU-BE.sh check-patch-revision.sh list-devalias-for-disk.sh PREBOOT-snapshot.sh121430-80.zip at-lucreate.conf copy-root-vtoc.sh lucreate.out print-disk-usage.sh121430-83 at-luupgrade.conf create-LU-BE-via-at.sh lustarter121430-83.zip check-disk-slices.sh excludelist patch-LU-BE-via-at.shau11qap25gtels2 $ cat excludelist/opt/app/var/opt/app/ora/db001/redo002/ora/db001/dbfra001/ora/db001/data001/ora/db001/inx001/ora/db002/inx001/ora/db001/redo001/ora/db002/redo002/ora/db002/data001/ora/db002/dbfra001/ora/db002/redo001/opt/app/ha/oracle/emagentau11qap25gtels2 $ au11qap25gtels2 $ cdau11qap25gtels2 $ au11qap25gtels2 $ pwd/au11qap25gtels2 $ df ifconfig -alo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 ce0: flags=1009040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,

FIXEDMTU> mtu 1500 index 2 inet 10.117.221.134 netmask ffffff80 broadcast 10.117.221.255 groupname db-mnb ether 0:14:4f:67:20:da ce0:1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 2 inet 10.117.221.133 netmask ffffff80 broadcast 10.117.221.255ce0:2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.117.221.138 netmask ffffff80 broadcast 10.117.221.255ce0:3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.117.221.136 netmask ffffff80 broadcast 10.117.221.255ce0:4: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 10.117.221.137 netmask ffffff80 broadcast 10.117.221.255ce1: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 1500 index 3 inet 130.103.249.147 netmask ffffff80 broadcast 130.103.249.255 ether 0:14:4f:67:20:db ce4: flags=1001000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,FIXEDMTU> mtu 9000 index 4 inet 10.106.213.107 netmask fffffe00 broadcast 10.106.213.255 ether 0:21:28:1a:d4:c4 ce4:1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 9194 index 4 inet 10.106.213.104 netmask fffffe00 broadcast 10.106.213.255ce6: flags=1009040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILOVER,FIXEDMTU> mtu 1500 index 5 inet 10.117.221.135 netmask ffffff80 broadcast 10.117.221.255 groupname db-mnb ether 0:14:4f:67:1e:58 au11qap25gtels2 $ au11qap25gtels2 $ zfs listNAME USED AVAIL REFER MOUNTPOINTrpool 32.0G 34.5G 106K /rpoolrpool/ROOT 12.0G 34.5G 31K legacyrpool/ROOT/zfsroot 12.0G 34.5G 7.96G /rpool/ROOT/zfsroot/var 4.01G 34.5G 4.01G /varrpool/appstats 479M 34.5G 479M /apps/statsrpool/dump 2.00G 34.5G 2.00G -rpool/home 569M 34.5G 569M /homerpool/netbackup 120M 34.5G 120M /usr/openv/netbackup/logsrpool/swap 16.5G 51.0G 16K -rpool/varcrash 373M 34.5G 373M /var/crashau11qap25gtels2 $ au11qap25gtels2 $ df -h zfsdf: (zfs ) not a block device, directory or mounted resourceau11qap25gtels2 $ df -hFilesystem size used avail capacity Mounted onrpool/ROOT/zfsroot 66G 8.0G 34G 19% //devices 0K 0K 0K 0% /devicesctfs 0K 0K 0K 0% /system/contractproc 0K 0K 0K 0% /procmnttab 0K 0K 0K 0% /etc/mnttabswap 36G 1.8M 36G 1% /etc/svc/volatileobjfs 0K 0K 0K 0% /system/objectsharefs 0K 0K 0K 0% /etc/dfs/sharetab/platform/sun4u-us3/lib/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/libc_psr.so.1/platform/sun4u-us3/lib/sparcv9/libc_psr/libc_psr_hwcap2.so.1 42G 8.0G 34G 19% /platform/sun4u-us3/lib/sparcv9/libc_psr.so.1

rpool/ROOT/zfsroot/var 66G 4.0G 34G 11% /varswap 6.0G 304K 6.0G 1% /tmpswap 36G 48K 36G 1% /var/runswap 36G 0K 36G 0% /dev/vx/dmpswap 36G 0K 36G 0% /dev/vx/rdmp/dev/vx/dsk/au11qap25gtels2dg/optappvl 27G 13G 13G 51% /opt/app/dev/vx/dsk/au11qap25gtels2dg/appdatavl 10G 20M 9.4G 1% /var/opt/apprpool 66G 106K 34G 1% /rpoolrpool/appstats 66G 479M 34G 2% /apps/statsrpool/home 66G 569M 34G 2% /homerpool/netbackup 66G 120M 34G 1% /usr/openv/netbackup/logs/dev/odm 0K 0K 0K 0% /dev/odmrpool/varcrash 66G 373M 34G 2% /var/crash/dev/vx/dsk/oradb001dg/redo001vl 21G 1.1G 20G 6% /ora/db001/redo001/dev/vx/dsk/oradb001dg/data001vl 2.5T 692G 1.8T 28% /ora/db001/data001/dev/vx/dsk/oradb001dg/inx001vl 1.5T 609G 971G 39% /ora/db001/inx001/dev/vx/dsk/oradb001dg/dbfra001vl 1.2T 75G 1.1T 7% /ora/db001/dbfra001/dev/vx/dsk/oradb001dg/redo002vl 21G 1.1G 20G 6% /ora/db001/redo002/dev/vx/dsk/oradb002dg/redo001vl 2.0G 231M 1.8G 12% /ora/db002/redo001/dev/vx/dsk/oradb002dg/data001vl 16G 7.0G 8.9G 45% /ora/db002/data001/dev/vx/dsk/oradb002dg/inx001vl 6.0G 17M 5.9G 1% /ora/db002/inx001/dev/vx/dsk/oradb002dg/redo002vl 2.0G 231M 1.8G 12% /ora/db002/redo002/dev/vx/dsk/oradb002dg/dbfra001vl 10G 41M 9.9G 1% /ora/db002/dbfra001/dev/vx/dsk/oradb002dg/optapphavl 5.0G 2.6G 2.4G 52% /opt/app/ha/oracle/emagentau11qap25gtels2 $ au11qap25gtels2 $ df -hF zfsFilesystem size used avail capacity Mounted onrpool/ROOT/zfsroot 66G 8.0G 34G 19% /rpool/ROOT/zfsroot/var 66G 4.0G 34G 11% /varrpool 66G 106K 34G 1% /rpoolrpool/appstats 66G 479M 34G 2% /apps/statsrpool/home 66G 569M 34G 2% /homerpool/netbackup 66G 120M 34G 1% /usr/openv/netbackup/logsrpool/varcrash 66G 373M 34G 2% /var/crashau11qap25gtels2 $ au11qap25gtels2 $ zfs set refquota=10g rpool/ROOT/zfsrootau11qap25gtels2 $ au11qap25gtels2 $ zfs set refquota=8g rpool/ROOT/zfsroot/varau11qap25gtels2 $ au11qap25gtels2 $ zfs set refquota=2g rpool/appstatsau11qap25gtels2 $ au11qap25gtels2 $ zfs set refquota=2g rpool/homeau11qap25gtels2 $ au11qap25gtels2 $ zfs set refquota=16g rpool/varcrashau11qap25gtels2 $

au11qap25gtels2 $ zfs refquota=10g rpool/netbackupau11qap25gtels2 $ au11qap25gtels2 $ zfs get -r refquota rpoolNAME PROPERTY VALUE SOURCErpool refquota none defaultrpool/ROOT refquota none defaultrpool/ROOT/zfsroot refquota 10G localrpool/ROOT/zfsroot/var refquota 8G localrpool/appstats refquota 2G localrpool/dump refquota - -rpool/home refquota 2G localrpool/netbackup refquota 10G localrpool/swap refquota - -rpool/varcrash refquota 16G localau11qap25gtels2 $ pwd/au11qap25gtels2 $ au11qap25gtels2 $ metadb -i flags first blk block count a m pc luo 16 8192 /dev/dsk/c1t10d0s7 a pc luo 8208 8192 /dev/dsk/c1t10d0s7 a pc luo 16400 8192 /dev/dsk/c1t10d0s7 a pc luo 16 8192 /dev/dsk/c1t11d0s7 a pc luo 8208 8192 /dev/dsk/c1t11d0s7 a pc luo 16400 8192 /dev/dsk/c1t11d0s7 a pc luo 16 8192 /dev/dsk/c3t10d0s7 a pc luo 8208 8192 /dev/dsk/c3t10d0s7 a pc luo 16400 8192 /dev/dsk/c3t10d0s7 a pc luo 16 8192 /dev/dsk/c3t11d0s7 a pc luo 8208 8192 /dev/dsk/c3t11d0s7 a pc luo 16400 8192 /dev/dsk/c3t11d0s7 a pc luo 16 8192 /dev/dsk/c0t9d0s7 a pc luo 8208 8192 /dev/dsk/c0t9d0s7 a pc luo 16400 8192 /dev/dsk/c0t9d0s7 r - replica does not have device relocation information o - replica active prior to last mddb configuration change u - replica is up to date l - locator for this replica was read successfully c - replica's location was in /etc/lvm/mddb.cf p - replica's location was patched in kernel m - replica is master, this is replica selected as input W - replica has device write errors a - replica is active, commits are occurring to this replica M - replica had problem with master blocks D - replica had problem with data blocks F - replica had format problems S - replica is too small to hold current data base R - replica had device read errorsau11qap25gtels2 $ au11qap25gtels2 $ metadb -d c3t10d0s7au11qap25gtels2 $ au11qap25gtels2 $ metastat -p|grep -i c3t10d0d25 1 1 c3t10d0s5d24 1 1 c3t10d0s4d23 1 1 c3t10d0s3d21 1 1 c3t10d0s1d20 1 1 c3t10d0s0d26 1 1 c3t10d0s6au11qap25gtels2 $ metaclear -fr d0 d1 d3 d4 d5 d6d0: Mirror is cleared

d10: Concat/Stripe is clearedd20: Concat/Stripe is clearedd1: Mirror is clearedd11: Concat/Stripe is clearedd21: Concat/Stripe is clearedd3: Mirror is clearedd13: Concat/Stripe is clearedd23: Concat/Stripe is clearedd4: Mirror is clearedd14: Concat/Stripe is clearedd24: Concat/Stripe is clearedd5: Mirror is clearedd15: Concat/Stripe is clearedd25: Concat/Stripe is clearedmetaclear: au11qap25gtels2: d6: metadevice in use

au11qap25gtels2 $ metastatr metastat -p d6d6 -m d16 d26 1d16 1 1 c1t10d0s6d26 1 1 c3t10d0s6au11qap25gtels2 $ metastat -pd105 -m d115 1d115 1 1 c0t9d0s5d104 -m d114 1d114 1 1 c0t9d0s4d103 -m d113 1d113 1 1 c0t9d0s3d101 -m d111 1d111 1 1 c0t9d0s1d100 -m d110 1d110 1 1 c0t9d0s0d2 -m d12 d22 1d12 1 1 c1t11d0s1d22 1 1 c3t11d0s1d72 -p d106 -o 4194368 -b 20971520 d106 -m d116 1d116 1 1 c0t9d0s6d70 -p d106 -o 32 -b 4194304 d62 -p d6 -o 4194368 -b 20971520 d6 -m d16 d26 1d16 1 1 c1t10d0s6d26 1 1 c3t10d0s6d60 -p d6 -o 32 -b 4194304 au11qap25gtels2 $ metacler� ar d60 d62d60: Soft Partition is clearedd62: Soft Partition is clearedau11qap25gtels2 $ metaclear -rf d6d6: Mirror is clearedd16: Concat/Stripe is clearedd26: Concat/Stripe is clearedau11qap25gtels2 $ au11qap25gtels2 $ metaclear d20 d21 d23 d24 d25 d26metaclear: au11qap25gtels2: d20: No such file or directory

metaclear: au11qap25gtels2: d21: No such file or directory





au11qap25gtels2 $ metastat -p d20metastat: au11qap25gtels2: d20: No such file or directory

au11qap25gtels2 $ metastat -pd105 -m d115 1d115 1 1 c0t9d0s5d104 -m d114 1d114 1 1 c0t9d0s4d103 -m d113 1d113 1 1 c0t9d0s3d101 -m d111 1d111 1 1 c0t9d0s1d100 -m d110 1d110 1 1 c0t9d0s0d2 -m d12 d22 1d12 1 1 c1t11d0s1d22 1 1 c3t11d0s1d72 -p d106 -o 4194368 -b 20971520 d106 -m d116 1d116 1 1 c0t9d0s6d70 -p d106 -o 32 -b 4194304 au11qap25gtels2 $ au11qap25gtels2 $ grep -i d2 /etc/vfstabau11qap25gtels2 $ au11qap25gtels2 $ pwd/au11qap25gtels2 $ iduid=0(root) gid=0(root)au11qap25gtels2 $ au11qap25gtels2 $ prtvtoc /dev/rdsk/c2t9d0s2 | fmthard -s - /dev/rdsk/c3t10d0s2fmthard: New volume table of contents now in place.au11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ zpool listNAME SIZE ALLOC FREE CAP HEALTH ALTROOTrpool 67.5G 15.5G 52.0G 22% ONLINE -au11qap25gtels2 $ au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINE scan: none requestedconfig:

NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 c2t9d0s0 ONLINE 0 0 0

errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ zpool attach -f rpool c2t9d0s0 c3t10d0s0Make sure to wait until resilver is done before rebooting.au11qap25gtels2 $ au11qap25gtels2 $ zpool status pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will

continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 147M scanned out of 15.5G at 13.4M/s, 0h19m to go 147M scanned out of 15.5G at 13.4M/s, 0h19m to go 143M resilvered, 0.93% doneconfig:

NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c2t9d0s0 ONLINE 0 0 0 c3t10d0s0 ONLINE 0 0 0 (resilvering)

errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ zpool status -x pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 389M scanned out of 15.5G at 17.7M/s, 0h14m to go 389M scanned out of 15.5G at 17.7M/s, 0h14m to go 389M resilvered, 2.45% doneconfig:


errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 841M scanned out of 15.5G at 13.6M/s, 0h18m to go 841M scanned out of 15.5G at 13.6M/s, 0h18m to go 837M resilvered, 5.30% doneconfig:


errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ ls -l /dev/rdsk/c2t9d0s0lrwxrwxrwx 1 root root 60 Oct 6 21:43 /dev/rdsk/c2t9d0s0 -> ../../devices/pci@21c,700000/pci@1/pci@1/scsi@2/sd@9,0:a,raw

au11qap25gtels2 $ au11qap25gtels2 $ ls -l /dev/rdsk/c3t10d0s0lrwxrwxrwx 1 root root 62 Oct 6 21:43 /dev/rdsk/c3t10d0s0 -> ../../devices/pci@21c,700000/pci@1/pci@1/scsi@2,1/sd@a,0:a,rawau11qap25gtels2 $ au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 1.79G scanned out of 15.5G at 12.1M/s, 0h19m to go 1.79G scanned out of 15.5G at 12.1M/s, 0h19m to go 1.78G resilvered, 11.53% doneconfig:


errors: No known data errorsau11qap25gtels2 $ zpool status -v pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 4.56G scanned out of 15.5G at 14.4M/s, 0h12m to go 4.56G scanned out of 15.5G at 14.4M/s, 0h12m to go 4.56G resilvered, 29.45% doneconfig:


errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ hasys -listau11qap25gtels2au11qap26htels2au11qap25gtels2 $ pwd/au11qap25gtels2 $ iduid=0(root) gid=0(root)au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 5.44G scanned out of 15.5G at 14.1M/s, 0h12m to go 5.44G scanned out of 15.5G at 14.1M/s, 0h12m to go

5.44G resilvered, 35.15% doneconfig:






errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINEstatus: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.action: Wait for the resilver to complete. scan: resilver in progress since Sun Oct 7 00:59:22 2012 10.9G scanned out of 15.5G at 14.2M/s, 0h5m to go 10.9G scanned out of 15.5G at 14.2M/s, 0h5m to go 10.9G resilvered, 70.37% doneconfig:




errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINE scan: resilvered 15.4G in 0h17m with 0 errors on Sun Oct 7 01:17:10 2012config:

NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c2t9d0s0 ONLINE 0 0 0 c3t10d0s0 ONLINE 0 0 0

errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c3t10d0s0au11qap25gtels2 $ au11qap25gtels2 $ ls -l /dev/rdsk/c2t9d0s0lrwxrwxrwx 1 root root 60 Oct 6 21:43 /dev/rdsk/c2t9d0s0 -> ../../devices/pci@21c,700000/pci@1/pci@1/scsi@2/sd@9,0:a,rawau11qap25gtels2 $ au11qap25gtels2 $ ls -l /dev/rdsk/c3t10d0s0lrwxrwxrwx 1 root root 62 Oct 6 21:43 /dev/rdsk/c3t10d0s0 -> ../../devices/pci@21c,700000/pci@1/pci@1/scsi@2,1/sd@a,0:a,rawau11qap25gtels2 $ au11qap25gtels2 $ au11qap25gtels2 $ prtcoprtconf -pvv|grep -i bp ootpath bootpath: '/pci@21c,700000/pci@1/pci@1/scsi@2/disk@9,0:a'au11qap25gtels2 $ au11qap25gtels2 $

au11qap25gtels2 $ eeprom "nvramrc=devalias rootdisk /pci@21c,700000/pci@1/pci@1/scsi@2/disk@9,0:a> devalias rootmirror /pci@21c,700000/pci@1/pci@1/scsi@2,1/disk@a,0:a"au11qap25gtels2 $ au11qap25gtels2 $ eeprom nvramrcnvramrc=devalias rootdisk /pci@21c,700000/pci@1/pci@1/scsi@2/disk@9,0:adevalias rootmirror /pci@21c,700000/pci@1/pci@1/scsi@2,1/disk@a,0:aau11qap25gtels2 $ au11qap25gtels2 $ zpool status -v pool: rpool state: ONLINE scan: resilvered 15.4G in 0h17m with 0 errors on Sun Oct 7 01:17:10 2012config:

NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c2t9d0s0 ONLINE 0 0 0 c3t10d0s0 ONLINE 0 0 0

errors: No known data errorsau11qap25gtels2 $ au11qap25gtels2 $ exitexitau11qap25gtels2 $ iduid=0(root) gid=0(root)au11qap25gtels2 $ exitau11qap25gtels2 $ iduid=36844(d629807) gid=5004(admin)au11qap25gtels2 $ au11qap25gtels2 $ pwd/home/admin/d629807au11qap25gtels2 $ iduid=36844(d629807) gid=5004(admin)au11qap25gtels2 $ exit au11qap25gtels2 $ exit

au11qap25gtels2_newevrzfs

Documents

Transcript of au11qap25gtels2_newevrzfs