Post on 29-Nov-2014
description
Lahav Savir
• 15 years in on-line industry
• Architect and CEO @ Emind Systems (est. 2006)
• AWS solution provider
• Over 30 AWS customers
Hobbies (that’s the . . .)
• MTB cycling
• Mountain hiking
Backup scenarios
On premises to off-site
• File servers
• Backup files
• Data base dumps archiving
• Disaster recovery
On the cloud to other site
• File servers
• Large data volumes
• Data base dumps
• Large S3 buckets
Storage scenarios
Storage appliances
• NFS
• CIFS
Disks & Servers
• Windows shares
• Linux exports
• Linux servers
• Sun exports
Requirements
Backup • Keep a replica of the data off-site • Keep history of the data for X previous months • Secure transfer • Encryption of data sets • Large files • Delta transfer Deployment • Don’t impact existing setup • Don’t install any SW on servers • No additional hardware
Few more . . .
• Control bandwidth throughput
• Visibility and monitoring
• Simplicity
• Keep the costs down
– License
– Traffic
– Storage
Alternatives
• Windows
– Virtual drive to S3
– Sync application
– Cygwin / delta copy
• Linux
– s3fs (fuse)
– s3cmd
• Storage built-in integration tos3
– No monitoring
– No visibility to status
– No bandwidth control
– No feedback
Simple solution
• Sync Manager
– Linux appliance
– cifs-utils
– rsync
– s3cmd
– tc (traffic controller)
– net-snmp
– curl
Sync Configuration
• rsync (filer to filer) rsync;/filer/data1/; sync@192.168.61.130:/data1/A
rsync;/filer/data2/; sync@porticor_vpd:/data2
• s3 (filer to s3 with / without VPD) s3;/var/www/wordpress/;s3://bucket1/wordpress-d/;-
-no-delete-removed
s3;/mnt/srv1/;s3://bucket2/
Bandwidth control
• Tag user traffic iptables -t mangle -A OUTPUT -m owner --uid-owner $SYNCMGR_UID -j MARK --set-mark 0x1
• Create root qdisc for eth0 $TC qdisc add dev $IF root handle 1: htb default 30
• Add a class (bucket) with bandwidth restrictions $TC class add dev $IF parent 1: classid 1:2 htb rate $MAXRATE
• Then add a filter to force packets through the class $TC filter add dev $IF protocol ip parent 1:0 prio 1 handle 1 fw classid 1:2
Tip: use iftop to see it in action
Monitoring
## SNMP params
SNMPTRAP=true
SNMPTRAP_HOST=nms_server
SNMPTRAP_PORT=162
SNMPTRAP_COMMUNITY=public
SNMPTRAP_OID=.1.3.6.1.4.1.39731.2101
## support_router
SUPPRTR_NOTIF=true
SUPPRTR_PROJECT="SupportDispatcher“
SUPPRTR_SYNCMGR_CLIENT=Emind
SUPPRTR_BASEURL=https://support.emind.co/support_router/public/api.php
## snmpd.conf
rocommunity public
# send all Emind Enterprise ID requests to the subagent
pass .1.3.6.1.4.1.39731 /usr/local/emind/snmp_subagent
Cloud backup hosts
• ec2 instance (Linux server)
– EBS volumes
• s3 buckets
• Porticor VPB
– EBS volumes
– S3 proxy
Hosting on the cloud
• Public cloud
– Instance behind security groups with SSH keys
• VPC
– Instance behind VPN
• AWS VPN Gateway
• IPSec with CheckPoint in the VPC
• IPSec with Swan in the VPC
• SSL VPN with OpenVPN in the VPC
Restoring
• rsync back from storage rsync ; sync@192.168.61.130:/data1/A ; /filer/data1/
• 3scmd s3cmd get s3://bucket2/file /path/to/restore/file
Summary
• Simple and open solution
• No impact on customer infrastructure
• No additional HW required
• Control with full visibility
• Fully integrated with NMS
• Reliable
• Secure
AWS Tips
• Don’t forget to set AWS console to MFA
• Setup a VPN to your AWS server
• No public SSH
• Monitor traffic coming into your servers
• Multi-region / AZ for high availability
• Use ec2 tools
• Backup backup backup . . .
Questions ???
Thank you, Mail me: lahavs@emind.co
Lahav Savir
LinkedIn / Twitter / Facebook