Daily Operational 7 11

Procedure to login to 7-111) Use below link to connect to SSL VPN using EDS net ID and password.https://sslvpnpl.eds.com/ics.html2) Once connected to SSL VPN, login to the jump server using 7-11 username/password. Use one of the jump servers as below205.191.168.69 and 205.191.168.2423) List of NBU servers in 7-11slc014buSolaris 10Master

slc090buSolaris 10Media

slc045buHP-UXMedia

slc046buHP-UXMedia

slc047buHP-UXMedia

slc070buHP-UXMedia

m0318pl2buHP-UXMedia

pls7ermd01buWindows 2003Media

slc049buHP-UXMedia

slc091buSolaris 10Media

4) Password change linkhttps://lighthouse.7-eleven.com/lighthouse/user/processLaunch.jsp?id=7-Eleven+Change+PasswordProcedure to reset (IPL) the tape drive from media management1) Login to ACSLS Tape library and dismount stuck tape if any from the drive for which reset is required2) Tape drive should be DOWN on all shared media server before reset of drive.3) Raise an SC incident and assign it to media management teams queue (Z_MEDIA_MGMT_PL)4) Send an email to media management team ([email protected]) with incident number to do the needful.5) Once drive has been reset then make this UP on all shared media server and monitor the same.

Do not start oracle BLI backups from NBU end, in case of failure, need to send an e-mail to database team. For Oracle BLI backup & restore, we dont have control, DBAs are having full control and they do backups and restores. Vault policies are scheduled on 1st & 2nd Wednesday of each month. Important points: ACSLS Library for 7-11 EnvironmentsHow to login ACSLS server1. Login to master server : SLC014BU, using your username and password2. telnet IP of the ACSLS server: PLEBACSLS2 from master server as below #telnet 192.168.100.353. Enter username/password (acsss/Schwing1) for the ACSLS server.4. To come in command prompt for the ACSLS server, run the below command #cmd_proc lImportant ACSLS commands for daily operation1. To check the drive status #q dr drive_id | allExamples: To query transport 0,3,1,0 : enter following command #q dr 0,3,1,0To query all transport: enter following command #q dr all 2. To dismount the tape from drive #dismount vol_id drive_id [force]Examples:To dismount volume AAA001 from the transport 0,3,1,0 : enter following command #dismount AAA001 0,3,1,0 force 3. To mount the tape into drive # mount vol_id drive_id [bypass] [readonly] Bypass : overrides some of the checks that ACSLS does before trying to mount a volume.Readonly : specifies that the volume is mounted write-protected Examples:To mount volume AAA001 on transport 0,3,1,0 #mount AAA001 0,3,1,0To mount volume AAA001 on transport 0,3,1,0 using the readonly option #mount AAA001 0,3,1,0 readonly 4. To exit from command prompt for the ACSLS library #logoffExample: To exit a cmd_proc : enter below command #log to get drive detailsPath /usr/openv/volmgr/bin#./tpconfig dlOut putDrive Name STKT9940B8 Index 28 NonRewindDrivePath /dev/rmt/82cbn Type hcart3 Status UP SCSI Protection SR (Global) Shared Access Yes ACS(1) Definition ACS=0, LSM=2, PANEL=7, DRIVE=8 Serial Number 479002015682

5. How to get the transport (ACS, LSM, PANEL, DRIVE) details of any drive.We can get the transport details by executing the following command in Master Server: SLC014BU#/usr/openv/volmgr/bin/vmoprcmd -devconfig -l 6. ACSLS Event LogThe base event log, which is automatically created when ACSLS is installed, is contained in the following file: - $ACS_HOME/log/acsss_event.log which is usually: /export/home/ACSSS/log/acsss_event.log Event log can be viewed using tail command as below#tail f /export/home/ACSSS/log/acsss_event.log7. To display the serial number of all tape drives# display drive *,*,*,* -f type serial_num8. To make the drive Online from Offline# vary drive drive_id online9. To check available cleaning tapes# query clean allKEDB for 7: ELP1) For error code 2 in case of online SQL backup failures:If we have any failure with error code 2 for SQL online backups then we need to check with SQL DBA team and WINTEL team for cluster failover, accordingly we need to change the client name in respective policy. Please refer the below attached mail for reference.

2) For System State backup failure (backups completing with status code 1) in NBU 6.5.4We need to install the EEB: NB_6.5.4_ET1765302_1_329235.zip. Please refer below attached mails for reference.

3) For error code 13 in case of / filesystem for HP-UX client (V11.23 & V11.31)We need to do the following entry in bp.conf file of the clientIGNORE_XATTR=1 OR we need to exclude /dev file from backup.Please refer below links & mail for the reference.http://www.symantec.com/business/support/index?page=content&id=TECH65101&actp=search&viewlocale=en_US&searchid=1311079270080http://www.symantec.com/business/support/index?page=content&id=TECH73719&actp=search&viewlocale=en_US&searchid=1311695996023

Case 414-930-954 (System state backup partially successful...)[email protected], ArunRecipientsRECIPIENTS/CN=DZ1MTL

Hi Arun,

Please refer the following links and installthe following EEB: NB_6.5.4_ET1765302_1_329235.zip is a NetBackup 6.5.4 EEB

http://support.veritas.com/docs/327105

A potential for System State data loss has been discovered in NetBackup Server / Enterprise Server 6.5.4. This occur when the System State is selected for backup and snapshot error control is changed from the default setting to "Disable snapshot and Continue". This issue only affects Windows clients.

http://support.veritas.com/docs/329235

NB_6.5.4_ET1765302_1_329235.zip is a NetBackup 6.5.4 EEB (Emergency Engineering Binary) for Windows clients.

Warm Regards______________Mohammed KhursheedSenior Technical Support EngineerNetBackup TeamSymantec Software Corporationwww.symantec.com------------------------------------------------------------------------------------------------

Please note that my hours of operation are between: 03:00 AM to 12:00 PM GMT10:00 PM to 7:00 AM EST7:00 PM to 4:00 AM PST07:00 AM to 4:00 PM IST

Working Days: Monday to FridayDirect Number:1-407-357-6809Email: [email protected]

Submit, Review, Manage your case online at MySupport.https://mysupport.symantec.com/

Support Phone: Please choose country from below link. Enter the CASE ID when prompted.http://www.symantec.com/business/support/contact_techsupp_static.jsp

***NOTICE: Determine if your systems are ready to install or upgrade to NBU 7.0 Visit: https://sort.symantec.com/checklist/install

For NetBackup Status code information, visit the Status Code Table at: http://www.symantec.com/business/support/index?page=content&id=TECH58686

For Important NetBackup 6.5 and 7.0 Common Topics visit: http://go.symantec.com/nbu

***NOTICE: Determine if your systems are ready to install or upgrade to NBU 7.0 Visit: https://sort.symantec.com/checklist/install

For NetBackup Status code information, visit the Status Code Table at: http://www.symantec.com/business/support/index?page=content&id=TECH58686

For Important NetBackup 6.5 and 7.0 Common Topics visit: http://go.symantec.com/nbu

***

***NOTICE: Symantec limits all e-mail, including attachments, to 10MB. Your message will not be delivered if it exceeds this limit. Please create a shorter message, remove attachments, or consult your technician if your message exceeds the 10MB limit. ***

CaseAuthKey=18AFE6A7

***NOTICE: This e-mail message is for the sole use of the intended recipient(s) and may contain certain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by e-mail and destroy all copies of the original message. Symantec and the Symantec logo are U.S. registered trademarks of Symantec Corporation. VERITAS is a trademark of Symantec Corporation. Other brands and products are trademarks of their respective holders. Copyright 2008 Symantec Corporation. All rights reserved.***

RE: 7-11 | System state backup Partial success | 100-02-8841017FromD s, Senthil KumarToD s, Senthil Kumar; S, AnandhanCcHP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Bestshore NBU BUR TeamRecipientsRECIPIENTS/CN=HZD09W; RECIPIENTS/CN=GZJY9Y; RECIPIENTS/CN=USW_SW_WINTEL_BEST_SHORE-CNT; RECIPIENTS/CN=RZN0XS; [email protected]

Hi Anandhan,

The change made by you is successful. FULL and INCREMENTAL job completed backing up all files and folders. Thank you.

Full Backup:

Incremental Backup:

Thanks and Regards Senthil Kumar.D.S |DCS ITO Storage & BUR CUG: #5-55-3243| Desk:91-44-66373243| Mobile:91-9894632192 DLF SEZ IT park| Tower-1B-L4|1/124 Sivaji Garden, Mount Poonamallee Road | Chennai| India 600089www.mphasis.com If no response from me, please mark your reply to : [email protected] For any escalations, please contact: [email protected] & [email protected] Please consider the environment before printing this email

From: D s, Senthil Kumar [mailto:[email protected]] Sent: Sunday, July 17, 2011 9:44 PMTo: S, AnandhanCc: HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Bestshore NBU BUR TeamSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Anandhan,

We have initiated the backup on USPLV7ETSDC1. Once the backup completed we will update you. Thank you.

Thanks and Regards Senthil Kumar.D.S |DCS ITO Storage & BUR CUG: #5-55-3243| Desk:91-44-66373243| Mobile:91-9894632192 DLF SEZ IT park| Tower-1B-L4|1/124 Sivaji Garden, Mount Poonamallee Road | Chennai| India 600089www.mphasis.com If no response from me, please mark your reply to : [email protected] For any escalations, please contact: [email protected] & [email protected] Please consider the environment before printing this email

From: S, Anandhan [mailto:[email protected]] Sent: Sunday, July 17, 2011 3:19 PMTo: Sharma, Manu; Halgali, Ashok; Bestshore NBU BUR TeamCc: HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana LauraSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Team,

Symantec patch has updated on USPLV7ETSDC1 today. Please fire the back up and update the status.

Best Regards,

Anandhan .S

My Support Hours : 21:30 to 6:30 AM (CST)

ITO-DCS- Wintel-US West Region Team

|+91 44 6612 0000 Ext :4101 Work phone | +91 99625 69710 Mobile | [email protected] | [email protected] Millennia Business Park,102, Campus 1C,143, Dr MGR Road, North Veeranam Salai, Perungudi, Chennai 600 096, India.

If I am not available or there is no response, please mark a copy to : [email protected]

For any escalation, please contact Balaji [email protected] or +91 99401 18940

www.mphasis.com

MphasiS races past USD 1 billion mark in FY 2010

_____

From: S, Anandhan Sent: Sunday, July 10, 2011 6:57 PMTo: Sharma, Manu; Halgali, Ashok; 'Bestshore NBU BUR Team'Cc: HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Wadsworth, Mark; Lubbers, Mark ASubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Manu,

Ok. Thanks for your confirmation. I will update same patch to USPLV7ETSDC1 on next week. I will let you know once I done.

Best Regards,

Anandhan .S



|+91 44 6612 0000 Ext :4101 Work phone | +91 99625 69710 Mobile | [email protected] RMZ Millennia Business Park,102, Campus 1C,143, Dr MGR Road, North Veeranam Salai, Perungudi, Chennai 600 096, India.



www.mphasis.com


_____

From: Sharma, Manu Sent: Sunday, July 10, 2011 5:30 PMTo: Sharma, Manu; S, Anandhan; Halgali, Ashok; Bestshore NBU BUR TeamCc: HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Wadsworth, Mark; Lubbers, Mark ASubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

The change made by you is successful. FULL and INCREMENTAL job completed backing up all files and folders.

Thanks & Regards,

Manu Sharma

Specialization Associate Infrastructure Engineer

MphasiS ITO

Cell : +91 9610418681, 9011096475 | [email protected] | [email protected]| [email protected]

EON Cluster C, Kharadi Infrastructure Pvt Ltd - SEZ, Plot No 1, Survey 77, MIDC, Kharadi, Pune - 411014

If I am unavailable or there is no response, please send a copy of the email to Bestshore NBU BUR Team

For any escalation, please contact : [email protected] [email protected]

From: Sharma, Manu [mailto:[email protected]] Sent: Sunday, July 10, 2011 5:00 PMTo: S, Anandhan; Halgali, Ashok; Bestshore NBU BUR TeamCc: HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Wadsworth, Mark; Lubbers, Mark ASubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Full backup didnt missed any file. Backup completed successfully. Will initiate an incremental now to check it.

Thanks & Regards,

Manu Sharma

Specialization Associate Infrastructure Engineer

MphasiS ITO

Cell : +91 9610418681, 9011096475 | [email protected] | [email protected]| [email protected]

EON Cluster C, Kharadi Infrastructure Pvt Ltd - SEZ, Plot No 1, Survey 77, MIDC, Kharadi, Pune - 411014


For any escalation, please contact : [email protected] [email protected]

From: S, Anandhan [mailto:[email protected]] Sent: Sunday, July 10, 2011 8:17 AMTo: Halgali, Ashok; Bestshore NBU BUR TeamCc: HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Wadsworth, Mark; Lubbers, Mark ASubject: FW: 7-11 | System state backup Partial success | 100-02-8841017

Hi Team,

Symantec fix has been installed on USPLD7ERDC04.please fire the backup and update the status.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: S, Anandhan Sent: Saturday, July 02, 2011 6:54 AMTo: Halgali, AshokCc: 'Bestshore NBU BUR Team'; Gupta, Arun; Vasudevan, Selvadurai; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; 'Torda, Douglas'; Wadsworth, Mark; Lubbers, Mark ASubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Team,

The change to implement Symantec fix has been rescheduled to next week end due to change freeze. I kept the reference the SC # 100-02-8841017 on suspend , till next week end/next scheduled date .

Thanks.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: S, Anandhan Sent: Friday, July 01, 2011 7:49 PMTo: Halgali, AshokCc: 'Bestshore NBU BUR Team'; Gupta, Arun; Vasudevan, Selvadurai; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Ashok,

Change record RI# 100-03-820126 has created and implementation scheduled on Saturday CST .

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: Halgali, Ashok Sent: Friday, July 01, 2011 8:35 AMTo: S, AnandhanCc: Bestshore NBU BUR Team; Gupta, Arun; Vasudevan, Selvadurai; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Anandhan,

Please do update with the status of the same.

Thanks & Regards,

Ashok Halgali

Lead Infrastructure Engineer | ITO BUR | Mphasis an HP Company

EON Cluster C, Kharadi Infrastructure Pvt Ltd - SEZ

Plot No 1, Survey 77, MIDC, Kharadi,Pune 411014

E-mail:- [email protected]; ashok.halgali@mphasis. com


For any escalation, please contact : [email protected] or [email protected]

From: S, Anandhan [mailto:[email protected]] Sent: Thursday, June 30, 2011 2:27 PMTo: Gupta, Arun; Vasudevan, Selvadurai; HP-DL USW_SW_Wintel_Best_ShoreCc: Bestshore NBU BUR TeamSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

HI Arun,

I will create a change record to update Symantec recommendations on DC04 and will let you know the status.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: Gupta, Arun Sent: Wednesday, June 29, 2011 2:53 PMTo: S, Anandhan; Vasudevan, Selvadurai; HP-DL USW_SW_Wintel_Best_ShoreCc: Bestshore NBU BUR TeamSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Team

Could you please update on below mail.

ThanksArun Gupta

Principal Infrastructure Engineer | ITODCS-BUR | Mphasis an hp Company2nd Floor, Cluster C, EON, Kharadi, Pune 411 014Phone: +9120 301-32296 | Mobile: +919923364551 | CUG: #5-52-32296eMail: [email protected] ; [email protected]; [email protected]

If no response from me, please mark your reply to : [email protected]

For any escalations, please contact: [email protected] & [email protected]

From: Gupta, Arun [mailto:[email protected]] Sent: Tuesday, June 28, 2011 2:49 PMTo: S, Anandhan; Jayakumar, Narendran; Vasudevan, SelvaduraiCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Anandhan

We have raised a case with Symantec and they are suggesting to install the EEB :- NB_6.5.4_ET1765302_1_329235.zip.

Refer the attached e-mail from Symantec and do the needful for the client USPLD7ERDC04.

ThanksArun Gupta




From: Gupta, Arun [mailto:[email protected]] Sent: Wednesday, June 22, 2011 10:27 AMTo: S, Anandhan; Jayakumar, Narendran; Vasudevan, SelvaduraiCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Anandhan

We will check and raise a case with Symantec for USPLD7ERDC04

ThanksArun Gupta

Principal Infrastructure Engineer | ITO BUR | Mphasis an hp Company2nd Floor, Cluster C, EON, Kharadi, Pune 411 014 |Phone: +9120 301-32296 | Mobile: +919923364551 | CUG: #5-52-32296eMail: [email protected] ; [email protected]; [email protected]

If no response from me, please mark your reply to :

NetBackup DL : [email protected]


From: S, Anandhan Sent: Tuesday, June 21, 2011 4:13 PMTo: Gupta, Arun; Jayakumar, Narendran; Vasudevan, SelvaduraiCc: 'Bestshore NBU BUR Team'; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Arun,

MS says there is no issue on USPLD7ERDC04, since we are able to take backup thru ntbackup command. Please raise a case with Symantec to fix the issue.

Regarding USPLV7ETSDC1, We are still working with MS to find out reason for system state back failure.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: S, Anandhan Sent: Monday, June 20, 2011 3:54 PMTo: Gupta, Arun; Jayakumar, Narendran; Vasudevan, SelvaduraiCc: 'Bestshore NBU BUR Team'; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Ok. Arun.

I am taking up this issue with MS and I will update the status.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: Gupta, Arun Sent: Monday, June 20, 2011 3:46 PMTo: S, Anandhan; Jayakumar, Narendran; Vasudevan, SelvaduraiCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Anandhan

Backup completed partially again. Please find the error messages as below.

06/20/2011 01:22:04 - Warning bpbrm (pid=3579) from client pld7erdc04bu.plano.esc.eds.com: WRN - old TIR info file 'C:\Program Files\VERITAS\NetBackup\tir_info\D\NetBackup_file_info.Domain_Controllers' is missing. Backing up everything in 'D:'

06/20/2011 01:22:04 - Warning bpbrm (pid=3579) from client pld7erdc04bu.plano.esc.eds.com: WRN - old TIR info file 'C:\Program Files\VERITAS\NetBackup\tir_info\D\NetBackup_file_info.Domain_Controllers' is missing. Backing up everything in 'D:'

06/20/2011 01:52:59 - Warning bpbrm (pid=3579) from client pld7erdc04bu.plano.esc.eds.com: WRN - old TIR info file 'C:\Program Files\VERITAS\NetBackup\tir_info\C\NetBackup_file_info.Domain_Controllers' is missing. Backing up everything in 'C:'

06/20/2011 01:53:00 - Warning bpbrm (pid=3579) from client pld7erdc04bu.plano.esc.eds.com: WRN - old TIR info file 'C:\Program Files\VERITAS\NetBackup\tir_info\C\NetBackup_file_info.Domain_Controllers' is missing. Backing up everything in 'C:'

06/20/2011 00:58:05 - Warning bpbrm (pid=4230) from client plv7etsdc1bu.plano.esc.eds.com: WRN - can't open object: Shadow Copy Components:\System Service\Event Logs\Event Logs (BEDS 0xE0009421: No component files present on the snapshot.)

06/20/2011 00:58:05 - Warning bpbrm (pid=4230) from client plv7etsdc1bu.plano.esc.eds.com: WRN - can't open object: Shadow Copy Components:\System Service\Windows Management Instrumentation\WMI (BEDS 0xE0009421: No component files present on the snapshot.)

06/20/2011 00:58:05 - Warning bpbrm (pid=4230) from client plv7etsdc1bu.plano.esc.eds.com: WRN - can't open object: Shadow Copy Components:\System Service\Windows Internet Name Service\C:_WINDOWS_system32_wins\wins (BEDS 0xE0009421: No component files present on the snapshot.)

0

ThanksArun Gupta





From: Gupta, Arun [mailto:[email protected]] Sent: Monday, June 20, 2011 11:24 AMTo: S, Anandhan; Jayakumar, Narendran; Vasudevan, SelvaduraiCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Anandhan

We have restarted the backups on both clients : pld7erdc04bu.plano.esc.eds.com & plv7etsdc1bu.plano.esc.eds.com, will update you the status.

ThanksArun Gupta





From: S, Anandhan [mailto:[email protected]] Sent: Sunday, June 19, 2011 10:11 AMTo: Jayakumar, Narendran; Vasudevan, SelvaduraiCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Narendran,

I have updated MS hot fixes on both servers today. I am able to take system state backup thru netback command. please reinitiate back up job and let us update status.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: Jayakumar, Narendran Sent: Saturday, June 18, 2011 5:47 PMTo: S, Anandhan; Vasudevan, SelvaduraiCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_ShoreSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Anandhan/Selvadurai,

Could you please update further.

Best Regards, Narendran JayakumarITO | DCS - BURMphasiS, an HP companyDLF IT Park, 1B Tower, Shivaji Garden, Ramapurum, Chennai-600089, Tamil Nadu, India

Mobile: +91-98845 24131eMail: [email protected] | [email protected]

"Make hay while the sun shines"

From: Jayakumar, Narendran [mailto:[email protected]] Sent: Tuesday, June 14, 2011 6:40 AMTo: S, AnandhanCc: Bestshore NBU BUR Team; Vasudevan, SelvaduraiSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Anandhan,

No luck on USPLD7ERDC04 SS backup Could you please provide the VSS output on USPLD7ERDC04. If everything looks good, I will open a ticket with Symantec for further trouble shooting.

Thanks for your support.




From: S, Anandhan Sent: Monday, June 13, 2011 4:20 PMTo: Jayakumar, NarendranCc: 'Bestshore NBU BUR Team'; Vasudevan, SelvaduraiSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi team,

I am able to take system state backup through Ntbackup command now. please fire the back again on USPLD7ERDC04.

I am still working with EMSC for USPLV7ETSDC1.i will update you the status ASAP.

Best Regards,

Anandhan .S






www.mphasis.com


_____

From: S, Anandhan Sent: Tuesday, June 07, 2011 10:46 AMTo: Jayakumar, Narendran; Vasudevan, SelvaduraiCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; 'Bestshore NBU BUR Team'; Torda, Douglas; Martorano, Ana Laura; Gupta, ArunSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi,

We have uploaded necessary logs file to EMSC , waiting for update from them. We will update you once we got it.

Thanks

Best Regards,

Anandhan .S






www.mphasis.com


From: Jayakumar, Narendran Sent: Tuesday, June 07, 2011 1:11 AMTo: Vasudevan, SelvaduraiCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, Douglas; Martorano, Ana Laura; S, Anandhan; Gupta, ArunSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Selva,

Please let me know if there are any update on your case.





From: Jayakumar, Narendran [mailto:[email protected]] Sent: Friday, June 03, 2011 11:14 PMTo: Vasudevan, SelvaduraiCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, Douglas; Martorano, Ana Laura; S, Anandhan; Gupta, ArunSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Selva,

Appreciate your response. And, I could see the errors in the output.

Lets wait for the case solution and please keep us posted. Thank you.




From: Vasudevan, Selvadurai Sent: Friday, June 03, 2011 8:54 PMTo: Jayakumar, NarendranCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; 'Bestshore NBU BUR Team'; Torda, Douglas; Martorano, Ana Laura; S, Anandhan; Gupta, ArunSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Narendran,

Please check the attachment for the output of the VSSadmin list writers of the servers uspld7erdc04 and usplv7etsdc1.

We have raised a EMSC case IM001216237 to check the backup failure issue on these servers. We will keep you updated once we have an update from EMSC.

Thanks & Regards,Selvadurai vasudevanUS West Region - Wintel Team

MphasiS ITO#+91 9884728866 mobile | Email | [email protected] Millenia Business Park, 102, Campus 1C, 143, Dr MGR Road, North Veeranam Salai, Perungudi, Chennai 600 096, IndiaIn the event that you are unable to contact me, please email the team DL at [email protected] HPVR : https://uspl.webmail.eds.com/exchweb/bin/redir.asp?URL=https://uspl.webmail.eds.com/exchwehttps://www.rooms.hp.com/attend/default.aspx?key=RH97KBHLZ2Our support hours are : 09:00 AM to 06:00 PM IST (11:30 PM EST to 08:30 AM EST)For any escalation, please contact : [email protected] (OR) +91- 9940118940.

www.mphasis.com

From: Jayakumar, Narendran Sent: Friday, June 03, 2011 2:52 AMTo: Vasudevan, SelvaduraiCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; 'Bestshore NBU BUR Team'; Torda, Douglas; Martorano, Ana Laura; S, Anandhan; Gupta, ArunSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Selva,

Let us know the output VSSadmin list writers on these servers. Also, please make sure that the output is stable.




From: Vasudevan, Selvadurai Sent: Thursday, June 02, 2011 4:50 PMTo: Gupta, ArunCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; 'Bestshore NBU BUR Team'; Torda, Douglas; Jayakumar, Narendran; Martorano, Ana Laura; S, AnandhanSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Arun,

We have tried manually running system state backup using the NTbackup utility on the server uspld7erdc04. The backup got completed successfully. Please find the attachment for the details.

We are checking the issue on the server usplv7etsdc1. We will keep you updated.

Please provide us with the complete error details.



www.mphasis.com

From: Gupta, Arun Sent: Thursday, June 02, 2011 3:40 PMTo: Vasudevan, Selvadurai; S, AnandhanCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, Douglas; Jayakumar, Narendran; Martorano, Ana LauraSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Selvadurai

Backups for both client failed with same error.

06/02/2011 02:09:28 - Warning bpbrm (pid=11075) from client pld7erdc04bu.plano.esc.eds.com: WRN - can't open object: System State: (WIN32 -536805685: Unknown error)




ThanksArun Gupta

Principal Infrastructure Engineer | ITO BUR | Mphasis an hp Company

From: Gupta, Arun [mailto:[email protected]] Sent: Thursday, June 02, 2011 12:42 PMTo: Vasudevan, Selvadurai; S, AnandhanCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, Douglas; Jayakumar, Narendran; Martorano, Ana LauraSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi Selvadurai

We have initiated the backups for both clients, will update you once complete.

ThanksArun Gupta





From: Vasudevan, Selvadurai [mailto:[email protected]] Sent: Thursday, June 02, 2011 12:34 PMTo: Martorano, Ana Laura; Jayakumar, Narendran; S, AnandhanCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, DouglasSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello All,

I have rebooted both the servers uspld7erdc04 & usplv7etsdc1 with CSR approval. Kindly initiate the Backups and update the status.



www.mphasis.com

From: Martorano, Ana Laura [mailto:[email protected]] Sent: Wednesday, June 01, 2011 11:18 PMTo: Jayakumar, Narendran; S, AnandhanCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, DouglasSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Im working to get approval from 7-Eleven. I will let you know as soon as I get it, we cannot reboot this servers just anytime.

Regards,

Ana Laura Martorano

Customer Service Representative - 7-Eleven

Hewlett-Packard Company

+1 972 378 3369/ Phone

+54 911 3231 4885/ Mobile

[email protected]/ Email

Please consider the environment before printing this email

From: Jayakumar, Narendran Sent: mircoles, 01 de junio de 2011 02:32 p.m.To: S, Anandhan; Martorano, Ana LauraCc: Hara, Jim; USW_SW_Wintel_Best_Shore; Bestshore NBU BUR Team; Torda, DouglasSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Ana,

Could you please provide your approval to Wintel to proceed with the reboot of the mentioned servers.





From: Torda, Douglas [mailto:[email protected]] Sent: Tuesday, May 31, 2011 7:41 PMTo: S, Anandhan; Martorano, Ana LauraCc: Hara, Jim; HP-DL USW_SW_Wintel_Best_Shore; Bestshore NBU BUR TeamSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Ana and Anandhan Did this get done?

Best Regards,

Douglas Torda

Account Delivery Executive -- 7-Eleven Account

Hewlett-Packard Company

+1 972 605 3883 / Tel

+1 214 402 9279 / Mobile

[email protected] / Email

5400 Legacy Drive

Plano, TX 75024

USA

Please consider the environment before printing this email.

From: S, Anandhan Sent: Monday, May 30, 2011 8:08 AMTo: Martorano, Ana LauraCc: Torda, Douglas; Hara, Jim; USW_SW_Wintel_Best_Shore; Bestshore NBU BUR TeamSubject: FW: 7-11 | System state backup Partial success | 100-02-8841017

Hi Ana,

We need your approval to reboot the servers for fixing the backup issue. After registered VSS, system state backup is still getting failed.

Server:

1.uspld7erdc04

2.usplv7etsdc1

Thanks

Best Regards,

Anandhan .S






www.mphasis.com


From: Gupta, Arun Sent: Monday, May 30, 2011 2:57 PMTo: S, AnandhanCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Hara, Jim; Jayakumar, NarendranSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi

Last backup for the client : plv7etsdc2bu.plano.esc.eds.com completed successfully, for other two clients we are getting same errors

pld7erdc04bu.plano.esc.eds.com


plv7etsdc1bu.plano.esc.eds.com




ThanksArun Gupta





From: Jayakumar, Narendran [mailto:[email protected]] Sent: Friday, May 27, 2011 3:58 PMTo: S, AnandhanCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Hara, JimSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Thanks, Anandhan. I will update you on Monday once this weekend Full backup completes.




From: S, Anandhan Sent: Friday, May 27, 2011 3:07 PMTo: Jayakumar, NarendranCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Hara, JimSubject: FW: 7-11 | System state backup Partial success | 100-02-8841017

Hi ,

VSS has been re-registered on USPLV7ETSDC1,DC2 and uspld7erdc04 . Please fire the back again and let us update the status.

Thanks

Best Regards,

Anandhan .S






www.mphasis.com


From: Jayakumar, Narendran Sent: Friday, May 27, 2011 2:33 AMTo: S, AnandhanCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Martorano, Ana Laura; Hara, JimSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello Anandhan,

We need this to be fixed asap.




From: Jayakumar, Narendran [mailto:[email protected]] Sent: Wednesday, May 25, 2011 11:16 PMTo: S, AnandhanCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Martorano, Ana Laura; Hara, JimSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello,

The VSS seems to be inconsistent as per VSSadmin output. Please re-register it to fix the System state backup issue.




From: S, Anandhan Sent: Wednesday, May 25, 2011 7:26 PMTo: Jayakumar, NarendranCc: 'Bestshore NBU BUR Team'; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Martorano, Ana Laura; Hara, JimSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi ,

Herewith attached output of net backup and VSSadmin.

Thanks

Best Regards,

Anandhan .S






www.mphasis.com


From: Jayakumar, Narendran Sent: Wednesday, May 25, 2011 5:20 PMTo: S, AnandhanCc: 'Bestshore NBU BUR Team'; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Martorano, Ana Laura; Hara, JimSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hello,

When the ntbackup of system state throw an error which shall be related to volume shadow component.

I would like to see the output of VSSadmin and ntbackup for these systems.




From: S, Anandhan Sent: Wednesday, May 25, 2011 4:55 PMTo: Jayakumar, NarendranCc: Bestshore NBU BUR Team; HP-DL USW_SW_Wintel_Best_Shore; Martorano, Ana Laura; Torda, Douglas; Martorano, Ana Laura; Hara, JimSubject: RE: 7-11 | System state backup Partial success | 100-02-8841017

Hi,

While running system state backup, system is throwing an error. As per MS recommendations , This error does not jeopardize the integrity of the backup.

VSS service is running and all writers are good.

To avoid this issues in future , we can proceed to update hot fix recommended by MS ,if it required on next available window.

Please refer the MS article http://support.microsoft.com/kb/822132.

Thanks

Best Regards,

Anandhan .S






www.mphasis.com


From: Jayakumar, Narendran [mailto:[email protected]] Sent: Wednesday, May 25, 2011 12:57 PMTo: HP-DL USW_SW_Wintel_Best_ShoreCc: Bestshore NBU BUR TeamSubject: 7-11 | System state backup Partial success | 100-02-8841017

Hello Team,

The system state backups on the below clients gets partial success. SS backup shouldnt be partial success which may cause issue during restoration.

VSS inconsistency is one of the cause. So, we need your support to get this fixed. Perform the following to find the VSS consistency. Moreover, we dont have access to these servers.

Volume shadow service should be in started state and automatic.

Perform the VSSadmin list writers cmd and check for error free output.

Perform ntbackup of system state and should get complete without any errors.

If VSS found to be inconsistent, please re-register the VSS and update us, this requires a reboot. Also, please share the output of the above activities. Thank you for your support.

pld7erdc04bu.plano.esc.eds.com

System State:\


System State:\


System State:\

Netbackup related errors:











image007.pngimage007.png

image008.gifimage008.gif

image009.jpgimage009.jpg




image013.gifimage013.gif

image001.pngimage001.png

RE: 7:11 | Archive backup failing for the client M0326PL2FromHalgali, AshokToFarberow, JeffCcBestshore NBU BUR Team; Farberow, Jeff; Yeh, Yvonne; HP-DL Oracle Tech 7-Eleven DBA; Muralidharan, Sinoj; Gupta, Arun; Martorano, Ana [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]

Hi Jeff,

As per below technote, we did an exclude to /dev directory & the backup ran fine. We have added the / partition to the original policy & backup is complete.

Deleted the test policy too.

http://www.symantec.com/business/support/index?page=content&id=TECH73719&actp=search&viewlocale=en_US&searchid=1311695996023" http://www.symantec.com/business/support/index?page=content&id=TECH73719&actp=search&viewlocale=en_US&searchid=1311695996023

Thanks & Regards,

Ashok Halgali

Lead Infrastructure Engineer | ITO BUR | Mphasis an HP Company

EON Cluster C, Kharadi Infrastructure Pvt Ltd - SEZ

Plot No 1, Survey 77, MIDC, Kharadi,Pune 411014

E-mail:- [email protected]; ashok.halgali@mphasis. com


For any escalation, please contact : [email protected] or [email protected]

_____________________________________________From: Gupta, Arun [mailto:[email protected]] Sent: Friday, July 22, 2011 9:38 PMTo: Farberow, Jeff; Farberow, Jeff; HP-DL Oracle Tech 7-Eleven DBA; Muralidharan, SinojCc: Bestshore NBU BUR Team; Yeh, YvonneSubject: RE: 7:11 | Archive backup failing for the client M0326PL2

Hi Jeff

We have removed the / from OS policy of m0326pl2. We will investigate further on this and let you know.

ThanksArun Gupta




_____________________________________________From: Farberow, Jeff [mailto:[email protected]] Sent: Friday, July 22, 2011 9:15 PMTo: Farberow, Jeff; Gupta, Arun; HP-DL Oracle Tech 7-Eleven DBA; Muralidharan, SinojCc: Bestshore NBU BUR Team; Yeh, YvonneSubject: RE: 7:11 | Archive backup failing for the client M0326PL2

Well Sinoj had a idea and it worked

We ran a test and took out the parameter that was added in bp.conf that fixed the error we were getting when it was trying to back up the / filessytem

IGNORE_XATTR=1

After taking that out the archive job ran fine. Dont have any idea how these problems are even related.

So until you talk to symmantec lets leave the parameter out of the bp.conf and you guys go back into the o/s policy and take / back out of the policy.

Reply back when you have done this

_____________________________________________

From: Farberow, Jeff

Sent:Friday, July 22, 2011 10:06 AM

To:Gupta, Arun; Oracle Tech 7-Eleven DBA; Muralidharan, Sinoj

Cc:Bestshore NBU BUR Team; Yeh, Yvonne

Subject:RE: 7:11 | Archive backup failing for the client M0326PL2

Im at a loss

Based on the technote one of the options was to copy nblog.conf from a working client to this client. I copied it from plslc080. We still have the problem.

The date on tile original file on m0326pl2 was

-rw-r--r-- 1 root bin 3655 May 19 2010 nblog.conf.orig\\

Thats the same size and date as the one I copied from plslc080 so I cant even see how this file was possibly corrupted

-rw-r--r-- 1 root bin 3655 May 19 2010 nblog.conf

Another option it mentioned was to look at /etc/vx/vrtslog.conf and make sure permissions were right and contents were right.

File on m0326pl2 had correct permissions and contents wer exactly as listed in technote below.

review the contents of the file/etc/vx/vrtslog.confthe correct contents and permissions are thus:/etc/vx/vrtslog.conf-rw-r--r-- 1 root root 382 vrtslog.confand contains correct information:################################################ Caution! Do not update/modify file by hand.# Use vxlogcfg tool to update/modify this file###############################################51216.Configuration=/usr/openv/netbackup/nblog.conf51216.Names=NB,nb50936.Configuration=/etc/vx/VxICS/icsul.conf50936.Names=VxICS, ics

I even tried to copy that file from m0318pl2 since the archive job is running fine over there. We still had the same error

The technote is basically saying

Its a permissions problem when it tries to delete the files at the end of the archive job.

I log in as root and could delete the files without any problem.

So I think unless you guys have any other ideas you will have to open a ticket with symmantec

_____________________________________________

From: Gupta, Arun

Sent:Friday, July 22, 2011 6:15 AM

To:Oracle Tech 7-Eleven DBA

Cc:Bestshore NBU BUR Team; Yeh, Yvonne; Farberow, Jeff

Subject:7:11 | Archive backup failing for the client M0326PL2

Hi Team

Archive backup for the policy : m0326pl2_ring_Archive is failing. We are getting error code 3 in NetBackup. Please see the below explanation for error code 3.

bash-3.00$ sudo ./bperror -S 3

valid archive image produced, but no files deleted due to non-fatal problems

The backup portion of the archive command reported problems so the files were not deleted.

Please check the issue and let us know. Also please refer below tech-notes from Symantec for similar kind of issue.

http://www.symantec.com/business/support/index?page=content&id=TECH57828&actp=search&viewlocale=en_US&searchid=1311332018812

http://www.symantec.com/business/support/index?page=content&id=HOWTO50780&actp=search&viewlocale=en_US&searchid=1311332018812

We have a SC ticket 100-02-9099891 assigned in our queue. Please let us know the queue name so that we can re-direct the same.

ThanksArun Gupta




RE: 7:11 : SQL backup for the policies 711CAESQL03_SQL & 711CAESQL07_SQL failed FromKarnik, JaydeepToGupta, Arun; IN DBA-CSCcBestshore NBU BUR Team; S, Anandhan; Martorano, Ana LauraRecipientsRECIPIENTS/CN=DZ1MTL; [email protected]; [email protected]; [email protected]; [email protected]

Hi Arun,

Inst3 & Inst7 are failed over to1 & running on SQ1.

Thanks & regards,

Jaydeep Karnik | SQL Server Database Administrator | M P H A S I S an HP Company |

Level 2,A 136, Tower IV, CyberCity, Magarpatta, Pune Maharashtra -411028- India |

Tel: +91 (20) 40142111 | CUG - #55142111|Mobile: +919552599629|Page:[email protected]

Mailto:[email protected] | Web: www.mphasis.com

From: Gupta, Arun [mailto:[email protected]] Sent: Friday, July 08, 2011 4:24 PMTo: IN DBA-CSCc: Bestshore NBU BUR Team; S, Anandhan; Martorano, Ana LauraSubject: 7:11 : SQL backup for the policies 711CAESQL03_SQL & 711CAESQL07_SQL failed Importance: High

Hi Team

SQL backup for the policies : 711CAESQL03_SQL & 711CAESQL07_SQL (INST3 & INST7) are failing. Client name for both policy is : pls7eehsq2bu.plano.esc.eds.com .

Please check and let us know. If these instances have failed over from SQ2 to SQ1 then let us know so that we will change the client name in backup policies accordingly.

ThanksArun Gupta




Daily Operational 7 11

Documents

Transcript of Daily Operational 7 11