03 A practical approach to implement a high available · A practical approach to implement a high...

19.10.2017

1

ISP Symposium 2017: IBM Spectrum Protect: The Times They Are A-Changin‘ – Customers Talks

26.-29. September 2017, Cologne Marriott Hotel, Germany

A practical approach to implement a high

available data protection solution with virtualized

IBM Spectrum Protect Servers in a Disk based

environment

How to set up and operate an IBM Spectrum

Protect System virtualized without Tape, but

Container Storage Pools and Replication


26.-29. September 2017, Cologne Marriott Hotel, Germany 2

Personal Data

Remo Leuppi is a Storage and Backup Engineer for Baloise Group. His area of expertise includes HDS, EMC, NetApp storage knowhow and IBM

Tivoli Storage Manager. His the Projectmanager for the following implementation at Baloise Group

Some words about Baloise / Basler Versicherung

The Baloise Group is focused on continental Europe with insurance, pension solutions and banking services. Baloise Group is an attractive,

modern employer represents solidity since 1863. The Company with about 7400 employees is listed on SIX Swiss Exchange, Main Market

Segment.

• Remo Leuppi

• Baloise Group

• [email protected]

19.10.2017

2



Personal Data

Gerd Becker is a Project Manager for EMPALIS Consulting GmbH, an IBM Business Partner in Germany. He has more than 40 years of IT

experience, including over 20 years experience with storage management products such as DFSMS and Tivoli Storage Manager. His areas of

expertise include IBM Tivoli Storage Manager implementation projects and education at customer sites, including mainframe environments

(OS/390®, VSE, VM, and Linux® for zSeries®).

In the context of data protection he has certifications and experience in project management.

He holds several certifications, including technical and sales, and is an IBM Tivoli Certified Instructor. He has developed and taught several

storage classes for IBM Education Services in Germany, Switzerland and Austria and for qSkills in Nuernberg. He has been Chairman of the

Guide Share Europe (GSE) Storage-Usergroup for more than 15 years. He is author of the Redbooks „IBM Tivoli Storage Manager Technical

Guide 5.3, IBM Tivoli Storage Manager Technical Guide 6.1, IBM Tivoli Storage Manager Certification Guide 6.1, IBM Tivoli Storage Manager as

a Dataprotection Solution“, did the betatest for TSM Version since 5.3 to 7.1.x , IBM Spectrum Protect to 8.1.x and is member of the IBM

Spectrum Protect Early Access Program.

• Gerd Becker

• Empalis Consulting GmbH

• [email protected]



Agenda

• Baloise Group – existing Environment

• Propositions

• Initial Design, architectural Concept

• Implementation

• Project Experiences

• Outlook

• Hints and Tips

19.10.2017

3



Baloise Group Server Landscape



Baloise Group Matrix of competences

19.10.2017

4



Baloise Group TSM Workload

• MS SQL

• MySQL

• Domino

• Mainframe

• SAS

• DB2

Domain Summary Total Data GB Summary Total files

CLUSTER 14.223,36 598.948

DB2 41.423,64 2.333.921

DOMINO 1.209,95 2.344.888

DOMINO_DAOS 5.168,16 7.051.932

DOMINO_DB 103.762,05 408.609

FILER 75.275,08 276.778.124

LINUX-OS 30.456,52 45.521.779

ORACLE_DB 11.017,78 26.724

OS 98.098,31 51.764.118

SAP_DB 19.056,64 1.110.052

SAS 29,31 170.047

SHAREPOINT 24,65 623

SQL 49.295,91 883.235

Result 449.041,36 388.993.000



Propositions – where we come from

• 3 Node Cluster

• HP DL380 G8

• 128 GB Memory

• 2 TSM Instances

• Max 64 GB Memory

• Primary and Copy Pool

• Primary on HDS HUS-VM (DC1)

• Secondary on HDS HUS-VM

(DC3)

• DB / LOG / ARCH on NetApp • with 4 snapshots per day

• Snapshot replication to DC3

19.10.2017

5



Propositions – why we will change the world

Challenges and Motivation

• Hardware out-of-life

• Software out-of-support

• Data amount growth

• New Functions

• Less complexity

• Upgrading DataCenter 2• before only data position (storage)

• Now with compute power (storage and servers)



Propositions – guidelines for the new design

• Windows Operating System

• Disk based solution – no Tape

• Virtualized servers (if possible)

• High level backup and recovery performance

• High available environment – cluster solution

• Disaster protection – distant datacenters

• Ransomware attack protection – air gap (without tape ?)

• Hardware efficient solution

• Cost optimized solution

• Bidirectional replication of data

19.10.2017

6



Initial Design – TSM Architecture Concept

• TSM VM

• Win 2012 R2

• 64 GB Memory

• SAN Raw Device• DB (Flash)

• Activelog (Flash)

• Archivelog (SAS and NL-SAS)

• Datapool (SAS and NL-SAS)

• Hyper-V Cluster

• Win 2012 R2

• Ressource shared with DPM

• 4 Host DC1 / 2 Host DC2• DL380 G9

• 256 GB Memory



Initial Design – Hyper-V Architecture Concept

• Cluster DC1• 4 Host

• DL380 G9

• 256 GB Memory

• 2X 40 Gb Network

• 2x 16 Gb FC (1per Fabric)

• VM• OS Data on shared Datastore

• Virtual WWN for SAN Disk

• Hyper-V Datastore• CSV (NetApp)

• Mirrored to DC2

• Snapshot all 6h

• Presented on all Host

• TSM Data • HDS HUS VM

• Raw Device (FC)

• Direct to VM

19.10.2017

7



Implementation – Step by Step

Step overview:

1. Installing test TSM Instances – Test installation procedure and basic functions

2. Working with IBM Blueprint – where you can find information

3. Performance Test (Blueprint) – scripts for I/O testing and server setup, what we used and what not

4. Installing Mgmt Server (HUB and TSMManager) – Administrators view

5. Installing production TSM Instances – implementation guidelines

6. Generate Housekeep and other scripts managed by Mgmt-Server – Enterprise Management

7. Server-to-Server Connection – New instances, old instances

8. Data Migration Approach



Implementation – Step 1

Installation Procedure

Hardware preparation

Software Installation with the Installation Manager

Installation latest good level (July 2016) IBM Spectrum Protect Version 7.1.5

Setting up the Sever Instances Test1 (Basel), Test2 (Zürich)

Basic Tests, functionality of container pools and node replication

Testing benefits: Compression and Deduplication

19.10.2017

8



Implementation – Step 2

IBM Blueprint:

- Design for IBM Hardware

- Design for „real“ Servers, not virtualized

- Naming conventions not variable, will not fit to every environment

+ very good basic information

+ values can be adapted to every hardware

+ guidelines for the setup

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/IBM%20Spectrum%20Protect%20Blueprints



Implementation – Step 3 - Performance

Containerpool performance:Blueprint baseline:

19.10.2017

9



Implementation – Step 3 - Performance

DB performance:

Blueprint baseline:



Implementation – Step 4 – Admins view

IBM: Operations Center JamoDat: TSM-Managerpros

• Free of charge – included in the product

• Helpful for beginners

• New functions and features available with the

server-level

• Web based application

cons

• Dependency to the hub server

• Resource hungry

• Functionality limited – implemented via

commandbuilder

pros

• Easy to use

• Compatible to all server levels

• New functions and features near time

available with new server-level

• Long term reports included

• Resource saving

cons

• License fee

• Dependency between collector and

viewer

• Only Microsoft Windows based

• New functions like replication, cloud

rare implemented

19.10.2017

10



Implementation – Step 5 – Implementation Guidelines

• Windows 2012 R2 virtualized with Hyper-V

• Disk Storage FC direct attached to the Guest System (Raw Device Mapping for STGPools, Database and Log)

• Only one Spectrum Protect Instance per OS

• Fast Network Connection – 10 Gb per VM (Host 2x 40 Gb)



Implementation – Step 6 – Housekeep and Protection

Relationship between instances for Disaster

protection:

- Data: Protect Stgpool

- Metadata: Replicate node

- Database Data Instances: s2s virtual volumes

- Database management Instance: local copy

on Netapp, Snap Mirror to the remote site

- Recovery Planfiles Data Instances: s2s virtual

volumes

- Recovery Planfiles management instance:

local copy on Netapp, Snap Mirror to the

remote site

19.10.2017

11



Implementation – Step 7 – Server – to – Server Relationship

• Old Servers TSM Version 6.3

• New Servers IBM Spectrum Protect Version 7.1.5

• High Speed Network Connection (10 Gb)

• Disk Storage FC direct attached to the Guest System (Raw Device Mapping)

• Only one Spectrum Protect Instance per OS

• Fast Network Connection – DWDM Basel - Zurich

40 Gb between Hyperv Hosts

10 Gb between HyperV Host and SP Server



Implementation – Step 8 – Data Migration

Possible Approaches:

• „big bang“ Switch: dry out old systems during data retention

• Export / Import: move all data from all old systems to the new systems

• Mix of 1. and 2.: initial backup to the new systems, dry out old data with short retention, export/import data with long term retention and archive objects (mergefilespace)

Challenge:

• Initial backup with a huge amount of data

• Import not direct in container pool, must be a sequential pool

• Convert stgpool necessary, file-pool after conversion unusable

Solution:

• Migration Node by Node with migration plan

• Additional resources temporary available (File Stgpool)

• Automated procedure for export/import and Stgpool conversion

• No NDMP-Data to migrate

• No Backupsets to migrate

19.10.2017

12



Implementation – Step 8 – Data MigrationPossible Approaches:

• „big bang“ Switch: dry out old systems during data retention

• Export / Import: move all data from all old systems to the new systems

• Mix of 1. and 2.: initial backup to the new systems, dry out old data with short retention, export/import data with long

term retention and archive objects (mergefilespace)

Challenge:

• Initial backup with a huge amount of data

• Import not direct in container pool, must be a sequential pool

• Convert stgpool necessary, file-pool after conversion unusable

Solution:

• Migration Node by Node with migration plan

• Additional resources temporary available (File Stgpool)

• Automated procedure for export/import and Stgpool conversion

• No NDMP-Data to migrate

• No Backupsets to migrate

Migration: Control of the Import-Process

/* ------------------------------------------------------------------*/

/* Script Name: ++IMPORT_nodes */

/* Description: Workflow for Migration-process */

/* with Export / Import */

/* controlling script variable ZZ_node_xxxx */

/* 1. Check Status */

/* 2. Start Export without data */

/* 3. Connect Node in Import Domain */

/* 4. Start Export with Data */

/* 5. End Part 1 */

/*--------------------------------------------------------------------*/

/*----->>>>>>>>>>>>> convert Filepool if necessary <<<<<<<<----------*/

/*--------------------------------------------------------------------*/

/* 6. Check Status */

/* 7. connect node in target domain */

/* 8. associate schedule */

/* 9. delete node admin */

/* 10. >>>>>>> to do manually */

/* - >>>>>>> associate schedules */

/* - >>>>>>> change client option file */

/* - >>>>>>> generate password (dsmcutil) */

/* - >>>>>>> restart scheduler */



Project Experiences

1. Security Guidelines for Node access

2. New node registration Procedure

3. Disaster Recovery Procedure

4. Daily Ingest Procedures

5. Storage Pool savings (dedup and compression

6. Storage Pool protection

7. Node Replication

8. Overall Data Occupancy

9. Database Occupancy

19.10.2017

13



Project Experiences – Security Guidelines• Access to Clients without Password:

the password has to be stored encrypted –

password access generate

• The node password set during registration

will expire in a short term (5 days)

• No access to other client‘s data

• No web access with node admin

• Only Server administrators can do backups

and restores (via RDP)



Migration: activate Node after Export, Conversion and Registration

/* -----------------------------------------*/

/* Script Name: ++activate_node */

/* Description: Migration: aktivate Node after Export, Conversion and Registration */

/* Parameter1 node-Name */

/* Parameter2 Domaene */

/* Parameter3 Schedule */

/* example1: run ++activate_node node1 os*/

start:

upd node $1 dom=$2 replstate=ena maxnummp=5 passexp=5 cloptset=""

if (rc_ok) goto Step2

goto Fehler

Step2:

unlock node $1


goto Fehler

Step3:

remove admin $1


if (rc_notfound) goto step4

goto Fehler

Step4:

define association $2 $3 $1

if (rc_ok) goto Ende

if (rc_exists) goto Ende

goto Fehler

Ende:

issue message i "node $1 ready for use"

exit

Fehler:

issue Message e "node $1 not ready for use, please check"

exit

Project Experiences – New Node registration

Procedure to register new nodes:

Requirement open registration

• Graphical Interface for Node registration

• Creation of dsm.optnode-name, tcpserver, tcpport

• Run dsmc, enter password and contact

• Setup scheduler with password (passwordaccess generate)

• Node activation:- connect node to target domain- set passwordexpiration- remove node admin-define association to the schedule

19.10.2017

14



Project Experiences – Disaster RecoveryProcedure to prevent disaster situationsDifferent Levels of Disaster

• Temporary outagenode failover to the target siteonly restore and retrieveno backup, e.g. for database logs

• Permanent outage of the primary datacenterConfiguration change, Target <-> Sourcemust be manually done – tested scriptsway back probably complicated

• Replacement of the primary datacenterrecover virtualized Server (binaries and definitions)restore the databaserecover the container storage pools



Project Experiences – Daily ingest Procedures

Approximately 11 to 20 TB per Day

19.10.2017

15



Project Experiences – Storage Pool Savings



Project Experiences – Storage Pool Protection

SOURCE_POOL

TARGET_

SERVER START_TIME END_TIME Duration

GBYTES_TO_

PROTECT

GBYTES_

PROTECT

ED

EXTENTS_TO

_PROTECT

EXTENTS_

PROTECTE

D

EXTENTS_TO_

DELETE

EXTENTS_

DELETED

COMP_

STATE

CONTAINERPOOL TSMP031 09.08.2017 02:00 09.08.2017 03:59 1:59:17 1.231.713 1.231.713 16074343 16074343 2296814 2296814 Success

CONTAINERPOOL TSMP031 17.08.2017 02:00 17.08.2017 04:18 2:18:22 1.001.617 1.001.617 17059265 17059265 29706896 29706896 Success

CONTAINERPOOL TSMP031 12.08.2017 02:00 12.08.2017 08:51 6:51:13 964.971 964.971 13483738 13483738 28843040 28843040 Success





19.10.2017

16



Project Experiences – Node Replication



Project Experiences – Overall Occupancy

Savings with deduplication / compression

Less amount of data to license

Upgrade V8.1.1 ???

19.10.2017

17



Project Experiences – Database OccupancyMassive database growth with container pool deduplication

DB Backup to virtual Volumes with compression

Performance and Tuning Guide:

For a rough estimate, plan for 100 GB of database

storage for every 50 TB of data that is to be

protected in deduplicated storage pools.

Protected data is the amount of data before data

deduplication, including all versions of objects

stored.



Outlook – Future and Functions

• Upgrade Spectrum Protect Server to Version 8.1.1 with eFix

8.1.1.9

• Upgrade Operations Center to Version 8.1.1

• Update Spectrum Protect Clients to Version 8

• Enhanced Security with Version 8.1.2 +

19.10.2017

18



Outlook – Obscurities and Questions

• Why does the node replication takes longer than protect

stgpool ?

• When will we be able, to do backups to the target server in

case of a disaster?

See RFE 107699 and 109500



Hints and Tips

With Air-Gap against ransomware !

• Disk only solution

before

after

19.10.2017

19



Questions



Thank you

Remo Leuppi

Dipl. Techniker HF

Wirtschaftsinformatik

Corporate IT

Basler Versicherungen,

Aeschengraben 21, Postfach,

4002 Basel

Telefon +41 58 285 76 70,

Mobile +41 76 451 94 08

[email protected] -

www.baloise.com

03 A practical approach to implement a high available · A practical approach to implement a high...

Documents

Transcript of 03 A practical approach to implement a high available · A practical approach to implement a high...