mailto:emanuela.caramagna@emc.com

2014 EMC Proven Professional Knowledge Sharing 2

Table of Contents

Introduction ........................................................................................................................................................ 3

First phase: Analysis ......................................................................................................................................... 4

Second phase: Planning & Design .................................................................................................................... 5

Greenplum backup on Data Domain using NFS or DD*boost ......................................................................... 5

Standard database backup using proxy host with Oracle RMAN & Data Domain in VTL ................................ 7

NDMP VNX backup using snapshot and Data Domain NDMP server with NetWorker ................................... 9

Third phase: Implementation and Test ............................................................................................................. 11

Fourth phase: Monitoring and Change Management ....................................................................................... 12

Conclusion....................................................................................................................................................... 12

Glossary .......................................................................................................................................................... 13

Disclaimer: The views, processes, or methodologies published in this article are those of the author. They do

not necessarily reflect EMC Corporation’s views, processes, or methodologies.

2014 EMC Proven Professional Knowledge Sharing 3

Introduction

Backing up big data environments is a huge challenge for storage administrators faced with backup window,

SLA, and RTO/RPO requirements. Considerations are complex and change depending on the type of

environment, backup flow, and so on.

In this article, I will explain some general considerations along with a case study developed during a recent

customer site engagement. While not presuming to cover all possible cases, I can provide my direct

experience in the field.

This article will explain different process phases needed to define backup strategy and implement it on a

customer site, including;

analysis

planning and design

test and implementation

monitoring

2014 EMC Proven Professional Knowledge Sharing 4

First phase: Analysis

During this phase it is very important to collect all needed information about the environment to back up,

keeping in mind that backup performance is impacted by all elements involved during operation (storage, SAN,

LAN, Host, etc.). To do that, customers need to have clear answers to the following questions:

What type of data are we managing?

o Big Data appliance (Greenplum®, Teradata, Netezza, etc.)

o Traditional Database (Oracle, DB2, SQL, etc.)

o NAS (VNX, NetApp, etc.); Is NDMP applicable?

o Unstructured data on server

Which compliance is needed?

o Backup windows

o Success rate required

o RPO

o RTO

Size and data characteristics?

o Data size

o Regarding appliances: how many appliances, how many instances, full/incremental backup

management of appliance, etc.

o Regarding standard databases: type of databases (Oracle Exadata, RAC, etc.), type of

database; transactional? not transactional?, archive log management, backup host

characteristics (CPU, RAM, SAN, connection, etc.), storage and SAN characteristics, backup

destination, etc

o Regarding NDMP: NAS model, disk structure, snapshot are allowed? If yes, number of backup

snapshots in parallel, number of file systems, number of files inside, LAN/SAN connections,

change rate

o Regarding unstructured data: number of file systems, number of files, backup host

characteristics (CPU, RAM, SAN, connection, etc.), storage and SAN characteristics, backup

destination, change rate

2014 EMC Proven Professional Knowledge Sharing 5

Second phase: Planning & Design

With all data collected during the analysis phase, it is necessary to design the solution in terms of:

Backup flow: backup direct from production, from snapshot, from proxy host

Requirements: LAN/SAN connectivity, storage, host, backup destination, backup server

Policy to implement: number of sessions, scripts needed, backup server and storage node

configurations, etc.

While this is the most critical phase, it is not possible to give a complete answer to all scenarios in one article

because all change in base of environment characteristics. For this reason, examples of one backup type

(Appliance, Standard database, NDMP) are provided below.

Greenplum backup on Data Domain using NFS or DD*boost

Parameters:

Dimension: 50TB for database (2 databases, 100 TB total), 10GB connection, 1 DD990 used in DD*Boost, 1

full backup every Friday for DB1 and every Sunday for DB2. RPO: 24 Hours, RTO: 2 days.

To manage this backup, consider size, backup windows, parallel segments, and boost reduction.

Parameters Value

Requirements Value

Size 50

TB/h 4.17

Backup windows 12

MB/s Read 1213.63

Parallel segments 60

MB/s LAN/Write NFS 1213.63

Boost reduction 3

MB/s LAN/Write Boost 404.54

MB/s Single channel NFS 20.23

MB/s Single channel Boost 6.74

2014 EMC Proven Professional Knowledge Sharing 6

In this case, you need to manage 1214MB/s in read from the Greenplum appliance. As well, you need

1214MB/s write to DDwith NFS protocol, or 406MB/s in write in the case of DD*Boost.

This means that you need one 10GB connection for DD*Boost and two 10GB connection for NFS

Considering a DD990, you need at least 5-6 shelves in order to attain the performance needed. Fifty percent of

DD990 sustained throughput will be used in NFS and 25% using DD*Boost.

In this case, restore considerations are more restrictive because RTO is 2 days.

Parameters Value

Requirements Value

Size 50

TB/h 1.04

Backup windows 48

MB/s Read 303.41

Parallel segments 60

MB/s LAN/Write NFS 303.41

MB/s Single channel NFS 5.06

In this scenario, no backup software is used but all is managed by Greenplum and Data Domain®.

2014 EMC Proven Professional Knowledge Sharing 7

Standard database backup using proxy host with Oracle RMAN & Data Domain in VTL

Parameters:

Dimension: 80TB in one database, backup window is 24 hours, 1 full backup every day, backup managed by

proxy host, RTO: 2 days, RPO: 2 hours (managed by archive logs direct from production) This real case is

based on a transactional database.

In this case, it is necessary to consider that backup windows are used for different operations; for writing data

to the Data Domain system, the window is 12 hours.

In this scenario, we need to consider storage performance, channel between storage and proxy host, proxy

host, and channel between proxy host and Data Domain system as show below.

2014 EMC Proven Professional Knowledge Sharing 8

Requirements are an initial indication of prerequisites based on NetWorker® Performance and Tuning Guide

and from Oracle SGA requirements. Based on my experience, the values provided are good for Linux on Intel

platform. In any case, it gives the customer an indication of proxy host requirements.

CPU Requirements: (MB/s/100)*2+% DB overhead

SGA requirements: fixed size+ (n°Channel*(16+(4*device block size))

Parameters Value

Requirements Value

Size (TB) 80

Read from storage TB/h 6.666666667

Backup windows hours 12

Read From Storage MB/s 1942

%CPU DB overhead 15%

Required CPU 45

N° channel 64

Standard Target Write MB/s 1942

Device Block size GB 0.5

MB/s Single Channel 30

Retention (day) 30

SGA PARAMETERS:

Number of datafile 10000

Fixed size 4096

Channel/block size calculation 1152

MIN TOT. SGA REQUIRED 5248

Networker Index Size (GB) 0.07

Min IOPS ( large server ) 1900

Disk Latency max 25 ms

In this scenario, NetWorker (a virtual backup server in this example), RMAN, and script for disk operations are

used to support backup operations.

Storage needs to sustain 1650 MB/s for read, proxy host that can manage 3300MB/s transfer rate, and Data

Domain DD990 used in VTL with a minimum of 8 shelves needed.

On the proxy host, four 8GB Fiber Channel ports are needed for storage connection and four 8GB Fiber

Channel ports are needed for write to Data Domain. Each part of the architecture is critical in this scenario and

needs to be tuned and set up with very high level of attention to obtain required performance.

2014 EMC Proven Professional Knowledge Sharing 9

Restore considerations are more restrictive in this case because RTO is 2 days:

Parameters Value

Requirements Value

Size (TB) 80

Read from storage TB/h 1.666666667

Backup windows hours 48

Read From Storage MB/s 485

%CPU DB overhead 15%

Required CPU 11

N° channel 64

Standard Target Write MB/s 485

Device Block size GB 0.5

MB/s Single Channel 8 .

NDMP VNX backup using snapshot and Data Domain NDMP server with NetWorker

Considerations for NAS backup are a bit more complex and require a good NAS configuration in terms of

volumes and file system structure.

Parameters: 10 million files (1M for volume), 30 TB of data (10 Volumes of 3TB), NAS with a maximum of eight

backup/restore operations in parallel, mix of standard file systems and deduplicated, 10 GB connection

available for backup, backup windows; 24 hours on weekend, one full backup every month and incremental

backup on other days.

2014 EMC Proven Professional Knowledge Sharing 10

Parameters Value

Requirements Value

Size (TB) 30

MAX SIZE per Day 5.7

Backup windows hours 24

TB/h 0.24

Daily change rate 10%

Read/write MB/s 69

Number of stream 8

Full single stream MB/S 36

Device Block size GB 0.5

BACKUP SERVER

Retention (day) 30

Index size (GB) 17

Number of datafile 10000000

CPU 8GHz

Number of Volume 10

RAM (GB) 32

max latency max 25ms

IOPS 2800.00

In this scenario Data Domain is used as a NDMP server and is configured on NetWorker server as storage

node. Production NAS will direct data to Data Domain system via 10GB LAN and Index to Backup server.

You need to create a VTL on Data Domain with a minimum of 8 drives and set up 8 as max parallelism on NAS

client defined on NetWorker.

It is important to consider NAS configuration and free space:

It is strongly recommended to back up data from snapshot in this scenario. This requires 10-15% of

free space in NAS volume in order to manage snapshot

If you have deduplicated volume and you write to a Data Domain system, it is recommended to send

data not deduplicated to backup

If you have a deduplicated file system, VBB is not supported and you must use DUMP to manage

backup operations

2014 EMC Proven Professional Knowledge Sharing 11

In this case we will manage this schedule:

In order to have better performance, you need to consider that if you have large volume with millions of files it

is better to divide the structure into different folders in order to manage full backups within the backup windows.

In this case, IOPS / CPU / RAM on backup server are very important because NDMP backup uses more

resources than a standard backup (120 IOPS for each NDMP session).

Third phase: Implementation and Test

Inside this phase, it is very important to plan some tests in order to verify infrastructure performance and tune

parameters such as:

Number of sessions for appliance and standard database

DAY FS1 FS2 FS3 FS4 FS5 FS6 FS7 FS8 FS9 FS10

1° Mon Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

1° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

1° Wed Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

1° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

1° Fri FULL Incr Incr Incr Incr Incr Incr Incr Incr Incr

1° Sat Incr FULL Incr Incr Incr Incr Incr Incr Incr Incr

1° Sun Incr Incr FULL Incr Incr Incr Incr Incr Incr Incr

2° Mon Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

2° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

2° Wed Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

2° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

2° Fri Incr Incr Incr FULL Incr Incr Incr Incr Incr Incr

2° Sat Incr Incr Incr Incr FULL Incr Incr Incr Incr Incr

2° Sun Incr Incr Incr Incr Incr FULL Incr Incr Incr Incr

3° Mon Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

3° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

3° Wed Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

3° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

3° Fri Incr Incr Incr Incr Incr Incr FULL Incr Incr Incr

3° Sat Incr Incr Incr Incr Incr Incr Incr FULL Incr Incr

3° Sun Incr Incr Incr Incr Incr Incr Incr Incr FULL Incr

4° Mon Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

4° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

4° Wed Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

4° Tue Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

4° Fri Incr Incr Incr Incr Incr Incr Incr Incr Incr FULL

4° Sat Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

4° Sun Incr Incr Incr Incr Incr Incr Incr Incr Incr Incr

2014 EMC Proven Professional Knowledge Sharing 12

Type of backup for DUMP or VBB for NDMP backup

Device block size

Tune host kernel parameters

Oracle and DB parameters, etc.

Tune storage configuration, FC/LAN connections, etc.

It is important to plan enough tests in order to have enough time to manage tuning phases and have complete

architecture tuning.

This is the most difficult phase, requiring a very high level of attention and time in order to avoid customer

dissatisfaction with the backup configuration.

Fourth phase: Monitoring and Change Management

Monitoring the daily situation and changes inside the environment is crucial.

A basic trace can be managed by monitoring Data Domain with EMC Data Protection Advisor in order to track

daily backup ingestion, deduplication ratio, and LAN/SAN performance. Additionally, it is possible to add FC

and LAN switches with detail on port usage and monitoring. Finally, adding a backup application and operating

system monitoring provides a complete scenario of all infrastructure.

It is strongly suggested to add notifications such as backup failure, backup windows not respected, growth

size, etc.

Conclusion

Though this is all very hard work, it is what is required to achieve cost-effective and efficient backup!

2014 EMC Proven Professional Knowledge Sharing 13

Glossary

RTO = recovery time objective

RPO = recovery point objective

SLA = service level agreement

DPA = data protection advisor

DD = Data Domain

NDMP = Network Data Management Protocol

NAS = Network Attached Storage

EMC believes the information in this publication is accurate as of its publication date. The information is

subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO

RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS

PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR

FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable

software license.