Managing Big Data with Percona Server, XtraBackup, and Tungsten

Managing Big Data withManaging Big Data withPercona Percona Server,Server,XtraBackupXtraBackup, and Tungsten, and Tungsten

Robert Hodges CEO, Continuent,Vadim Tkachenko, Founder & CTO, Percona

© Copyright 2011 Percona and Continuent

AgendaAgenda

/ Introductions/ What Is The Big Data Problem?/ Percona Server/ XtraBackup/ Tungsten/ Commercial Services and Support


About PerconaAbout Percona

/ Percona is the oldest and largest independent MySQLsupport and consulting company

// Our products:Our products:•• Percona ServerPercona Server•• Percona XtraBackupPercona XtraBackup

// We provide consulting, support, training andWe provide consulting, support, training anddevelopment for MySQL and LAMPdevelopment for MySQL and LAMP


About ContinuentAbout Continuent

/ Continuent is the leading provider of data replicationand clustering for open source relational databases

/ Our Products:• Tungsten Replicator - High-performance, MySQL replication• Tungsten Enterprise - Commercial replication and data

management solution for MySQL and PostgreSQL

/ Our Services:• Consulting on Tungsten plus replication and clustering in general• Subscriptions for commercial products


What Is the Big DataProblem?


The Big Data Problem

/ Large quantities of data• Application performance• Slow server starts

/ High transaction loads• Resource contention• SQL processing overhead

/ Distributed data• Datasets replicated across servers and sites• Master failover and slave promotion• Slave lag

/ Continuous, 24x7 Operation• Hot backup and restore• System upgrades


Percona Server


Percona Server

/ Drop-in replacement for MySQL/ High performance/ Diagnostics/ Operations / data management/ NOSQL


High Performance

/ Tuned for multi-core multi-disks systems/ SSD / Flash optimizations


Diagnostic

/ Advanced statistic on query execution/ # User@Host: mailboxer[mailboxer] @ [192.168.10.165]

/ # Thread_id: 11167745 Schema: board

/ # QC_Hit: No Full_scan: No Full_join: No Tmp_table: Yes Disk_tmp_table: No

/ # Filesort: Yes Disk_filesort: No Merge_passes: 0

/ # Query_time: 0.000659 Lock_time: 0.000070 Rows_sent: 0 Rows_examined: 30 Rows_affected: 0 Rows_read: 30

/ # InnoDB_IO_r_ops: 1 InnoDB_IO_r_bytes: 16384 InnoDB_IO_r_wait: 0.028487

/ # InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000

/ # InnoDB_pages_distinct: 5 select count(distinct author_id) from art87.article87 force index (forum_id) whereforum_id = 240215 and thread_id = '710575'

/ Additional INFORMATION_SCHEMA tablesCLIENT_STATISTICSINDEX_STATISTICSINNODB_BUFFER_POOL_PAGESINNODB_BUFFER_POOL_PAGES_BLOBINNODB_BUFFER_POOL_PAGES_INDEXINNODB_BUFFER_POOL_CONTENTINNODB_INDEX_STATSINNODB_IO_PATTERNINNODB_RSEGINNODB_SYS_INDEXESINNODB_SYS_TABLESINNODB_TABLE_STATSPROCESSLISTTABLE_STATISTICSUSER_STATISTICS


Operations / management

/ Export / Import InnoDB tables• Move shard from one server to another• Try copy 100GB user1.table1 from Server1 to Server2

/ Fast warmup• Periodic dump and load at start content of buffer pool• Very important for big (memory, data) slaves

/ Integration with FlashCache• Hints for queries to ignore caching• FlashCache is Flash/SSD cache layer over regular HDD/RAID


NOSQL

/ HandlerSocket plugin• Access to data without SQL queries / overhead• Wide range of operations on PK/ index

• Batch, range ( < , >, <= X <= ) access

/ Outstanding performance• Reported 750,000 request/sec


Percona Server plans

/ Cloud support• Scalability + performance

/ Flash/SSD• Continue research and improve performance

/ Diagnostic• You never can have enough statistics


Introducing PerconaXtraBackup


Percona XtraBackup

/ OpenSource hot-backup for MySQL/InnoDB/ Free


Designed with big data in mind

/ Non-blocking for InnoDB tables/ Streaming directly to backup server / host/ Incremental ( differential ) backups/ Partial backups/ Parallel copy/ Minimal actions to setup slave


Non-Blocking

/ Copy InnoDB files in background/ Server continues to process workload


Streaming

/ Tar-compatible file• innobackupex-1.5.1 --stream=tar ./ > backup.tar

/ Compress• innobackupex-1.5.1 --stream=tar ./ | gzip - > backup.tar.gz

/ Copy to remote host• innobackupex-1.5.1 --stream=tar ./ | ssh vadim@desthost "cat - > /data/vol1/mysqluc/backup.tar“


Incremental/Differential

/ Differential• Copy ONLY changes since last full backup

/ Incremental• Copy ONLY changes since last incremental / differential


Partial Backup

/ Copy only selected databases / tables• Databases:

user1user2…user1000

• You want to copy only user567 database from server1 to server2• Create new database or import into existing


Setup slave

/ innobackupex-1.5.1 --stream=tar /tmp/ --slave-info | sshuser@DESTSERVER "tar xfi - -C /DESTDIR“

/ innobackupex-1.5.1 --apply-log --use-memory=2G/DESTDIR


Percona XtraBackup plans

/ Cloud/S3• Copy / restore data from cloud storage

/ Compression on table level/ Improve incremental and partial backup operations

© Copyright 2011 Percona and Continuent© Continuent 2011© Continuent 2010

Introducing Tungsten


What Is Tungsten?

/ Tungsten Replicator is a fast, open-sourcereplication engine for MySQL

/ Tungsten Enterprise is a complete datamanagement solution based on replication1. Ensure reliable, fast access to data2. Simplify common management tasks3. Increase performance and resource utilization…Without changing applications


Tungsten Replicator ArchitectureTungsten Replicator Architecture

MySQLMySQLTungstenReplicator(replicator)

MasterMaster

SlaveSlave

Binlogs TransactionHistory Log

(Transactions + Metadata)

replicator.properties

(Replicator configuration)

MySQLMySQL

TungstenReplicator(replicator)

TransactionHistory Log

replicator.properties

Tail binlog orlogin as client

Transport viaTCP/IPconnection

Apply using JDBC


Tungsten Replicator: So Many Features

/ Global transaction IDs/ Transaction filters/ Replicate from MySQL to PostgreSQL & Oracle/ Time-delay replication/ Backup and restore integration/ Cross-version replication: 5.5 -> 5.1 -> 5.0 -> 4.1/ Row and statement replication/ Automatic consistency checks/ Auto-provisioning/ Multi-master replication/ Parallel replication


Tungsten Replicator Pipelines (Slave)

THLTHLSlaveSlaveDBMSDBMS

ApplierApplierExtractorExtractor FiltersFilters ApplierApplierExtractorExtractor FiltersFilters

StageStage StageStagePipelinePipeline

Tungsten Replicator ProcessTungsten Replicator Process

Transaction History LogTransaction History Log

Master THLMaster THL(via network)(via network)


Apache/Mod_PHPlibmysqlclient.alibmysqlclient.a

Tungsten Enterprise ArchitectureTungsten Enterprise Architecture

Apache/Mod_PHP

DBMSDBMS

Replicator

DBMSDBMS

Replicator

DBMSDBMS

Replicator

libmysqlclient.alibmysqlclient.a

MasterMasterSlaveSlave SlaveSlave

ConnectorConnector

Manager ManagerManager


Tungsten Enterprise Management

Broadcast commands Broadcast commands and monitoring dataand monitoring data

BusinessBusinessRulesRules

Local DBMS ResourcesLocal DBMS Resources

Local DBMS Local DBMS ResourcesResources

Manager(Coordinator)

Manager

ManagerCctrl Client

Cctrl Client

Group Group CommunicationsCommunications

Cctrl Client

Local DBMS Local DBMS ResourcesResources


Tungsten Replicator vs. Enterprise

Feature Replicator EnterpriseSimple admin with trepctl X X

(MySQL) Row/Statement 5.0/5.1/5.5 X XMulti-master replication X XParallel apply (2.0) X XReplication to other DBMS types X XDistributed management XAutomated failover and recovery XZero-downtime upgrade XSQL routing and load-balancing XFloating IP support X

© Copyright 2011 Percona and Continuent© Continuent 2011© Continuent 2010

Moving Data:Tungsten Replicator


Big Data and Multi-Master Replication

/ Big data is spread across multiple servers and sites/ Many-to-one: consolidate updates from multiple

servers into one slave• Roll-up reporting from remote servers

/ Bi-directional: Updates from two masters propagateto each other

• High-volume merchant apps and distributed directories

/ Cross-site/Cross-Host: Processing distributed overmultiple hosts and sites

• Sharded, multi-tenant datasets with centralized data


Multi-Master Data Consolidation

MySQLMySQLTungstenReplicator

Master PipelineMaster Pipeline

MySQLMySQL

TungstenReplicator

Slave PipelineSlave Pipeline

MySQLMySQLTungstenReplicator

TungstenReplicator

MySQLMySQL TungstenReplicator

Direct Pipeline, Direct Pipeline, Multiple ServicesMultiple Services

MySQLMySQL

MySQLMySQL

master1

master2


Bi-Directional ReplicationBi-Directional Replication

nyc1nyc1

Site NYC Site NYC

nyc2nyc2

sjc1sjc1

sjc2sjc2

Replicator

njc-localnyc-remote

Replicatorsjc-local

Replicator

nyc-localsjc-remote

Replicatornyc-local

Site SJC Site SJC

Cool feature: Tungsten slaves rememberpositions on remote and local masters


Big Data and Failover

/ Slave promotion is a severe problem for high-volumeapplications with multiple slaves

/ Tungsten global transaction IDs eliminate binloghacks to reconfigure slaves

/ Tungsten epoch numbers prevent re-connectionwhen logs diverge after failover

/ Say goodbye to binlog hacks for failover!


MasterMaster

Replicator

Seqno

Epoch Transaction

1 1 UPDATE CUST..2 1 INSERT INTO

CALL_TAB…3 3 DELETE FROM

CALL_TAB…4 3 UPDATE CUST…

SlaveSlave

Replicator

Replicator Log

Transfer log from master

BackupBackupseqno=3seqno=3

Global ID and Epoch Number Concepts

Seqno

Epoch Transaction

1 1 UPDATE CUST..2 1 INSERT INTO

CALL_TAB…3 3 DELETE FROM

CALL_TAB…4 3 UPDATE CUST…5 3 UPDATE CALL_TAB…

Replicator LogBackup reloadable Backup reloadable

on any serveron any server

Epoch numbers preventEpoch numbers preventinconsistent slaves frominconsistent slaves from

connectingconnecting


MasterMaster

ReplicatorManager

SlaveSlave(Relay)(Relay)

ReplicatorManager

SlaveSlave

ReplicatorManager

SlaveSlave

ReplicatorManager

Floa

ting

IP

Main Cluster

BackupBackupBackupBackupBackupsBackups


DR Cluster

DR Site Enabling with Global IDs

Seqno 25Seqno 25Epoch 1Epoch 1

Seqno 21Seqno 21Epoch 1Epoch 1Seqno 28Seqno 28

Epoch 1Epoch 1



Old Old MasterMaster

ReplicatorManager

SlaveSlave(Relay)(Relay)

ReplicatorManager

New New MasterMaster

ReplicatorManager

SlaveSlave

ReplicatorManager

Floa

ting

IP

Main Cluster



DR Cluster




Seqno 35Seqno 35Epoch 1Epoch 1(epoch number(epoch numberpreventspreventsreconnect asreconnect asslave)slave)

Reconfiguration After Failover


/ Single-threaded MySQL replication suffers fromimpedance mismatch between CPU and I/O for high-volume data replication

/ Tungsten optimizes resource utilization for large-scale data transfer

/ Pipeline parallelism - Spread work across stages/ Service parallelism -- Spread work across different

replicators and replication services within replicators/ Parallel apply -- Spread apply operations across

multiple threads

Big Data and Resource Utilization


Parallel Slave Apply -- Internals View

THLTHL SlaveSlaveDBMSDBMS

ApplierApplierExtractorExtractor ApplierApplierExtractorExtractor

PipelinePipeline

Tungsten Replicator ProcessTungsten Replicator Process

Transaction Transaction History LogHistory Log

Master THLMaster THL(via network)(via network)

StageStageApplierApplierExtractorExtractor

StageStage StageStage

ApplierApplierExtractorExtractorApplierApplierExtractorExtractor

ParallelQueue

shard.listshard.listfilefile


db1 db2 db3db1 db2 db3db1 db2 db3db1 db2 db3

Replicator

Demo: Parallel Apply with TPC-B Tests

TPC-BTPC-BTestTest

TPC-BTPC-BTestTest

TPC-BTPC-BTestTest

Replicator

MasterSlave


Future Plans

Tungsten Replicator will go fullyGPL V2 in 2011!

Initial Replicator-Only Buildsavailable in February

Keep Watchingwww.continuent.com


Commercial Services andSupport


Continuent Product Subscriptions

/ Per-server subscriptions for Tungsten Enterprise;pricing based on number of DBMS servers

/ Software-only -- Software license; use consulting forsupport; bug-fixes are free

/ Software + Support -- Software license + 24x7support (SLAs negotiable)


Continuent ConsultingContinuent Consulting

/ Practical consulting by industry experts on anyaspect of Tungsten or replication/clustering for opensource databases

/ Hourly support contracts• Advice, troubleshooting, bug fixes

/ Projects• Cluster design, roll-outs, upgrades, special replication problems

/ Sponsored development• Implementation of features large and small

/ Offered for both open source and commercialproducts


Percona Software and Services

/ Consulting• Our specialties include MySQL architecture and design,

diagnosing and solving performance problems, recoveringcorrupted data, setting up high availability, clustering, and muchmore.

/ Support & Maintenance• Expert technical support for MySQL• Insurance against bugs in MySQL• Customized MySQL server binaries

/ Training• Public and custom onsite

/ Software• Percona Server with XtraDB 5.1 and 5.5 coming• Percona XtraBackup


Conclusion

/ Big data has a unique set of challenges for MySQLdeployments

• Large volumes• High transaction rates• Distribution across servers and sites• Constant availability

/ Percona Server, XtraBackup, and Tungsten addressbig data problems

/ Combine Percona and Continuent for your nextproject!


Continuent, Inc.560 S. Winchester Blvd., Suite 500 San Jose, CA 95128 Tel +1-866-998-3642 Fax +1-408-668-1009e-mail: [email protected]

Percona, Inc.+1-888-316-9775 or+1-208-473-2904 (USA)+44-208-133-0309 (UK)http://www.percona.com/contact/sales/

Contact InformationContact Information

Continuent Web Site:http://www.continuent.com

Percona Web Site:http://www.percona.com

Managing Big Data with Percona Server, XtraBackup, and Tungsten

Documents

Transcript of Managing Big Data with Percona Server, XtraBackup, and Tungsten