MariaDB Training - Downloads · MariaDB Performance Tuning MariaDB Training. MariaDB Performance...

Click here to load reader

  • date post

    11-Aug-2018
  • Category

    Documents

  • view

    244
  • download

    12

Embed Size (px)

Transcript of MariaDB Training - Downloads · MariaDB Performance Tuning MariaDB Training. MariaDB Performance...

  • MariaDB Performance TuningMariaDB Training

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Introduction

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Introducing MariaDB AbFounders from MySQL the

    Company and Community Funded by Founders, Employees,

    and Venture Capital Over 100 Employees Several former MySQL Employees

    and Community Members in over 14 Countries

    3

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Personal IntroductionsInstructor

    Name and Background

    Participants Name and Company MariaDB Experience How You Use MariaDB Needs Related to Course Topics

    4

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Class Schedule & Personal ConcernsStarting and Ending Times Planned Breaks On- Site

    Location of Rest Rooms Smoking Areas Snacks and Drinks

    LVC Classes Chat with Everyone

    5

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Course OutlineBasics & Core

    Tuning Overview Collecting Information

    Tweaking Schema Tuning Query Tuning Common Bottlenecks

    Server & Storage Engines Architecture Server Configuration MyISAM InnoDB

    Administration MariaDB Replication Maintenance Impact

    6

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Tuning Overview

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tuning TermsPerformance Efficiency Scalability Benchmark Throughput Risk

    Queuing SLA Response Time Capacity Resources Uptime

    8

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Value in Performance TuningEfficient Use of Hardware Resources

    Anticipate Needs

    Best Performance for Users Machines Cost less than User Time Competitive Advantage

    Maintain Spare Infrastructure Capacity Be Prepared for Application Development Needs (New

    Features) Allow for Unexpected Traffic Spikes or Other Changes

    9

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tuning LevelsApplication

    Primary Responsibility of Architects and Developers Secondary Responsibility of DBA for Input

    Database Primary Responsibility of DBA Tuning and Monitoring Secondary Responsibility of Developers Gain Insight and

    Understanding

    10

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tuning LevelsServer

    Primary Responsibility of System Administrator Secondary Responsibility of DBA Knows Well Databases

    Network Primary Responsibility of Network Administrator Secondary Responsibility of DBA Replication or Clusters

    11

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tuning Routine - When to TuneTune from Start in the SDLC

    Start Early to Ensure Schema is Well Constructed Test Queries on Real Data Watch for Bottlenecks Over Tuning without Production Data or Traffic is Counter

    Productive

    Conduct Periodic Reviews of Production Systems Watch for Schema, Query and Significant Changes Check Carefully New Application Features

    Monitor System Resources Disk, Memory, Network, CPU Eliminate the Obvious Potential Problems Be Proactive in Solving Problems

    Emergency Response Tuning

    12

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tuning Routine - When Not to TuneIdentify Objectives in Advance

    Adhere to Objectives

    Decide on Best Use of Resources Tune Appropriately Don't Excessively Tune

    Affect Adversely Data Integrity Where is Speed Most Important Where is Integrity Most Important Adhere to these Boundaries

    13

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Establish BaselineNumber of Queries Expected per Unit of Time Number of Concurrent Client Connections Expected

    Average Number of Clients Peak Number of Concurrent Clients

    Length of a Client Connection Expected In Milliseconds Size of Dataset Fit in Memory When all Requirements will Double

    14

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Construct TestsDetermine System Resource Limits and Design Test

    Should Run Normally Should Exceed Server Resource Limits Should Simulate Nightly Back-up Load without Users

    Noticing

    Examine Impact of Configuration Changes Remove or Restrict Access to CPU core or Hard Drives Start another Process via cron that Consumes Suddenly

    Memory Observe What Happens When a Caching Layer goes

    Offline

    Use Tests to Identify Easy Solutions

    15

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Examine Test ResultsTest System Identical to the Production System

    Operating System (Version, Libraries, Patches) Hardware (Number of Cores, Physical Memory, Disk Speed) Other Resources (Memory Available to MariaDB, Competing

    Services) Test System Simulating Real Data and Traffic

    Number of Clients (Concurrently, per Hour per Day, Peak Traffic)

    Any Caching that might Skew Results

    16

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    HardwareOne Service per Server is Ideal to Prevent Contention More CPU Cores is generally Good More Disk is usually Better

    Large Datasets, Fast Disks are Ideal

    RAM is usually Best Traffic Dependent More of Dataset in Memory, Fewer Slow Disk Operations

    17

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Collecting Information

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tools & StatisticsIdentify Accurately and Carefully Trouble Spots

    Guessing is Rarely Useful

    Gather Performance Stats with MariaDB and OS Tools SHOW Statements PERFORMANCE_SCHEMA CPU, Disk, Network, Memory, & Swap Stats

    Retain Snapshots of Multiple Stats Data from a Single Point Shows very Little

    Automate the Collection of Stats into Logs Can be Useful for Emergency Tuning

    19

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    SHOW PROCESSLISTSnapshot of mysqld Activity

    mysqld is Multi-Threaded, One Thread per

    Client Connection (i.e., query, transaction) a "process" is a "thread"

    Accumulate SHOW PROCESSLIST Snapshots to

    build History of Thread Activities

    20

    Time: Length of Time Thread has been in Current State State: Indicates Thread Activity

    SHOW PROCESSLIST \G

    ****** 1. row ******** Id: 9546 User: monyog Host: localhost:40600 db: NULL

    Command: Sleep Time: 192 State: Info: NULL Progress: 0.000

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Thread States

    21

    After Create

    Analyzing

    Checking Permissions

    Checking Table

    Cleaning Up

    Closing Tables

    Converting HEAP to MyISAM

    Copy to tmp Table

    Copying to Group Table

    Copying to tmp Table

    Copying to tmp table on Disk

    Creating Index

    Creating Sort Index

    Creating Table

    Creating tmp Table

    Deleting from Main Table

    Deleting from Reference Tables

    discard_or_import_tablespace

    End

    Executing

    Execution of init_command

    Flushing Tables

    Freeing Items

    FULLTEXT initialization

    Init

    Killed

    Locked

    Logging Slow Query

    Login

    Manage Keys

    NULL

    Opening Tables, Opening Table

    Optimizing

    Preparing

    Purging Old Relay Logs

    Query End

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Thread States

    22

    Reading from Net

    Removing Duplicates

    Removing tmp Table

    Rename

    Rename Result Table

    Reopen Tables

    Repair by Sorting

    Repair Done

    Repair with Keycache

    Rolling Back

    Saving State

    Searching Rows for Update

    Setup

    Sorting for Group

    Sorting for Order

    Sorting Index

    Sorting Result

    Statistics

    System Lock

    Table Lock

    Updating

    Updating Main Table

    Updating Reference Tables

    User Lock

    User Sleep

    Waiting for (tables|tables|table flush)

    Waiting for All Running Commits to Finish

    Waiting for Commit Lock

    Waiting for Global Read Lock

    Waiting for lock_type Lock

    Waiting for Release of Read Lock

    Waiting on Cond

    Waiting to Get Read Lock

    Writing to Net

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    SHOW STATUS - Global or SessionReturns List of Internal Counters GLOBAL for System-Wide Status Since Start-Up

    SESSION for Local to Client Connection

    FLUSH STATUS Resets Local Counters Monitor Changes to Counters to Identify Hot Spots Collect Periodically Status Snapshots to Profile Traffic

    23

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Status Counters

    24

    Aborted_clients

    Aborted_connects

    Binlog_cache_*

    Bytes_received

    Bytes_sent

    Com_*

    Compression

    Connections

    Created_tmp_disk_tables

    Created_tmp_files

    Created_tmp_tables

    Delayed_errors

    Delayed_insert_threads

    Delayed_writes

    Flush_commands

    Handler_*

    Innodb_*

    Key_*

    Last_query_cost

    Max_used_connections

    Not_flushed_delayed_rows

    Open_*

    Opened_files

    Opened_table_definitions

    Opened_tables

    Prepared_stmt_count

    Qcache_*

    Queries

    Questions

    Rpl_status

    Select_*

    Slave_open_temp_tables

    Slave_retried_transactions

    Slave_running

    Slow_launch_threads

    Slow_queries

    Sort_*

    Ssl_*

    Table_locks_immediate

    Table_locks_waited

    Tc_log_max_pages_used

    Tc_log_page_size

    Tc_log_page_waits

    Threads_cached

    Threads_connected

    Threads_created

    Threads_running

    Uptime*

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    PERFORMANCE_SCHEMASimilar to INFORMATION_SCHEMA, but Performance Tuning Monitors MariaDB Server Events

    Function Calls, Operating System Waits, Internal Mutexes, I/O Calls

    Detailed Query Execution Stages (Parsing, Statistics, Sorting) Some Features Storage Engine Specific

    Monitoring Lightweight and Requires No Dedicated Thread Not a Simple, Quick Fix

    Interpreting Output Requires knowledge of Internals Designed to be Used Iteratively with Successive Refinement

    25

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    PERFORMANCE_SCHEMA Tables

    26

    SET PROFILING=ON; SHOW PROFILES; SHOW PROFILE FOR QUERY 1;

    cond_instances

    events_waits_current

    events_waits_history

    events_waits_history_long

    events_waits_summary_by_instance

    events_waits_summary_by_thread_by_event_name

    events_waits_summary_global_by_event_name

    file_instances

    file_summary_by_event_name

    file_summary_by_instance

    mutex_instances

    performance_timers

    rwlock_instances

    setup_consumers

    setup_instruments

    setup_timers

    threads

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    External ToolsApacheBench htop httperf innotop iostat / vmstat / top maatkit MyBench MariaDB Benchmark Suite (sql-bench)

    mysqladmin (extended-status) mysqlslap mytop statpack Super Smack SysBench

    27

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Schema Tuning

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Naming ConventionConsistent, Descriptive Names Reduce Mistakes in

    Development and Maintenance Plural Form for Table Names (products) Singular Form for Column Names (name_last, email) Alphabetical Order for Link Tables (posts_users) Long Names are Fine Use Aliases in Queries

    29

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Choosing Data TypesUse Appropriate Data Type (INT for Numbers, VARCHAR) Use Accurate Types where required (DECIMAL vs. DOUBLE) Use Smallest Useful Type On-Disk Size does Not Equal In-Memory Size Variable Length Fields are often Padded Use NOT NULL, where Practical

    A NULL field uses slightly More Disk and Memory (Depends on Storage Engine)

    Use PROCEDURE ANALYSE( )

    30

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Data Types for Integers

    31

    Documentation on Integer Data Types: https://mariadb.com/kb/en/mariadb/data-types/

    Use UNSIGNED when Appropriate

    INT(n) Specifies Display Precision, Not Storage Precision

    Size and Precision is Storage Engine Dependent Define Handling of Out-of-Range Values with sql_mode

    Default Mode: Values are Truncated Silently Strict Mode: Errors are Generated

    TINYINT

    SMALLINT

    MEDIUMINT

    INTEGER,INT

    BIGINT

    https://mariadb.com/kb/en/mariadb/data-types/

  • trainin[email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    String Data Types

    32

    Documentation on String Data Types: https://mariadb.com/kb/en/mariadb/string-data-types/

    All String Data Types have a Character Set CHAR(n) number of characters, not bytes, wide VARCHAR(n) Changes to CHAR in Implicit Temporary Tables and mysqld internal buffers

    TEXT Not Supported by the MEMORY Storage Engine; Implicit Temporary Tables may Convert to MyISAM

    CHAR

    VARCHAR

    TINYTEXT

    TEXT

    MEDIUMTEXT

    LONGTEXT

    https://mariadb.com/kb/en/mariadb/string-data-types/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Character Set & Collation

    33

    Documentation on Character Sets and Collation: https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

    SELECT * FROM table1 ORDER BY col1 COLLATE latin1_german2_ci;

    Character Set may be Global or for Schema, Table or Column

    Multi-Byte Character Sets Increase Disk Storage and Working Memory Requirements (e.g, UTF-8 Requires 3 or 4 bytes per Character)

    Collations affect String Comparison (Character Order) Collations can be Changed for Query:

    https://mariadb.com/kb/en/mariadb/data-types-character-sets-and-collations/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Date & Time Data Types

    34

    DATE from 1000-01-01 to 9999-12-31 (YYYY-MM-DD)

    TIME from -838:59:59 to 838:59:59 DATETIME Same Range with Time (YYYY-MM-DD HH:mm:ss) TIMESTAMP Unix timestamp, in seconds from

    1970-01-01 Many Apps Store UNIX_TIMESTAMP() values in unsigned integer

    field

    YEAR Accepts YY or YYYY

    DATE

    TIME

    DATETIME

    TIMESTAMP

    YEAR

    Documentation on Date and Time Data Types: https://mariadb.com/kb/en/mariadb/date-and-time-data-types/Documentation on Microseconds in MariaDB: https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

    SELECT CURTIME(4); +---------------+ | CURTIME(4) |

    +---------------+ | 05:33:09.1061 | +---------------+

    https://mariadb.com/kb/en/mariadb/date-and-time-data-types/https://mariadb.com/kb/en/mariadb/microseconds-in-mariadb/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Listing Acceptable Values

    35

    Documentation on ENUM: https://mariadb.com/kb/en/mariadb/enum/ Documentation on SET: https://mariadb.com/kb/en/mariadb/set-data-type/

    CREATE TABLE colors (primary_colors ENUM('red', 'yellow', 'blue'));

    ENUM is an Enumerated List of String Values uses a 2-byte integer index

    CREATE TABLE colors

    (primary_colors SET('red','yellow','blue'));

    INSERT INTO colors VALUES('red'), ('red,blue');

    Simple Example for an ENUM Column

    Simple Example for a SET Column

    SET is a Specified List of String Values Can Hold Multiple Specified Values

    https://mariadb.com/kb/en/mariadb/enum/https://mariadb.com/kb/en/mariadb/set-data-type/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Keys and IndexesMake PRIMARY KEY as Small as Practical InnoDB Stores Copy of PRIMARY KEY with each Index Entry

    Secondary Indexes Grow with PRIMARY KEY

    Use Surrogate PRIMARY KEY, Not Natural Ones AUTO_INCREMENT is like a Sequence AUTO_INCREMENT UNSIGNED NOT NULL Careful of AUTO_INCREMENT Contention for Pre-Plugin InnoDB

    Consider Partial (Prefix) Index for String Indexes With Good Index Selectivity, Won't Affect Performance

    Much Without Good Selectivity is a Different Problem

    36

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    File StorageStore Files in BLOB Useful

    Included in Back-Up and Replication Guaranteed to be Consistent with Other Data

    Store Files in BLOB has Problems Database gets Too Large MariaDB Can't do Partial Field Read MyISAM Reads Entire File InnoDB can Skip if Field Not Referenced in Query

    Can Minimize by putting Files in Separate Table (1:1 relationship)

    Store Files in File System and File Name and Path in Table Database is Smaller and More Efficient Organize Separate Back-Up, possibly with External Locking

    for Consistency

    37

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Tuning Tables OverallMinimize Table Size on Disk and in Memory Archive Table Data if Possible and Appropriate Remove Duplicate or Unused Indexes

    No need to Index a Column if its First Field in Another Index

    Use Appropriate Data Types Smaller is Better Consider Sharding Large Tables across Multiple

    Servers

    38

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Normalization of TablesProcess of Optimizing and Factoring a Schema

    1:1, 1:n and n:n 1NF / 2NF / 3NF

    Makes a Data Set as Smaller by Eliminating Redundant or Duplicated Data

    Problem if Data is Smaller than Key

    Ensures Data Set Consistency and Integrity Often Improves Concurrency by Reducing Locking

    Overhead - Not Always Increases Number of Tables and Associated

    Maintenance

    39

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Denormalization of TablesComplete Normalization can Slow Queries More Complex JOIN Queries are Drains and Harder to Maintain

    Adding Redundant Data Back to Simplify Problems with JOIN Queries

    Combine 1:n Relationships Add Pre-Computed Columns Use Fake Materialized Views

    Denormalization Adds Redundant Data to Schema Write Queries become More Complex or Numerous as Multiple

    Locations must be Maintained

    It's Easier to Normalize First, Then Denormalize when Appropriate

    40

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    ViewsMERGE

    Query and Original View SELECT are Merged Often the Fastest Option, as Base Table Indexes Used

    TEMPTABLE View Data Materialized to Temporary Table, then Query

    Executed Can be Slower, as Temporary Table Doesn't have Indexes

    UNDEFINED Algorithm is Chosen on Fly Usually Defaults to TEMPTABLE

    41

    CREATE VIEW ALGORITHM = ...

    AS SELECT ...

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Query Tuning

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Query Tuning OverviewTry Not to Query Tune on Production Server Use Test Server with same Hardware and OS Use Copy of Production Data Set Query Frequency is as Important as Query Speed

    Moderately Slow Queries are often a Bigger Problem than a Rarely Run Very Slow Query

    43

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    IndexingIndexes Improve Read Performance

    Without Index, MariaDB Must Read Every Row a Full Table Scan

    With Index, MariaDB can jump to Requested Rows Reduced I/O and Improving Performance Index Increase cost of Writes

    Index Improve Writes since they Require Reads Find Balance

    Index for Speed, but Avoid Indexing Excessively or Arbitrarily Remove Dead or Redundant Indexes

    44

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Index SizeKeep Indexes as Small as Practical

    Faster since More Likely to Fit in Memory Rebuilds Faster after Writes PRIMARY KEY should be Minimum Useful Size

    Use Partial Prefix Indexes for String Columns May Slow Searches Slightly, but Reduce Index Size

    Use Index Cardinality (Uniqueness Measure) Only If Necessary Re-evaluate as Data Grows

    Low Cardinality Indicates many Duplicates High Cardinality is More Useful

    45

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    B-Tree IndexesBalanced Binary Search Trees holding Pointers to

    Rows Precision depends on Storage Engine MyISAM stores within .MYD File Pointers to Rows (Offsets) InnoDB Primary Keys store Row Other Indexes Store Copy

    of Key

    Good for many Operations Equality (= ) Range Access (> < >=

  • [email protected]

    Version 3.1, Slide Copyright 2017. MariaDB Ab. Commercial in Confidence

    Advanced MariaDB

    InnoDB Adaptive Hash IndexInnoDB performs like In-Memory

    Database with AHI Leads to High CPU Usage

    Mutex for Adaptive Hash Index

    47

    SHOW ENGINE INNODB MUTEX;

    +--------+------------------------------+---------------+

    | Type | Name | Status |

    +--------+------------------------------+---------------+

    | InnoDB | &rseg->mutex | os_waits=1 |

    | InnoDB | &dict_sys->mutex | os_waits=1 |

    | InnoDB | &log_sys->mutex | os_waits=210 |

    | InnoDB | &buf_pool->flush_state_mutex | os_waits=5095 |

    | InnoDB | &buf_pool->LRU_list_mutex | os_waits=41 |

    | InnoDB | combined &block->mutex | os_waits=11 |

    | InnoDB | &new_index->lock | os_waits=2 |

    | InnoDB | &new_index->lock | os_waits=16 |

    | InnoDB | &new_index->lock | os_waits=1 |

    | InnoDB | &dict_operation_lock | os_waits=221 |

    | InnoDB | &log_sys->checkpoint_lock | os_waits=2920 |

    | InnoDB | &btr_search_latch_arr[i] | os_waits=6 |

    | InnoDB | combined &block->lock | os_waits=62 |

    +--------+------------------------------+---------------+

    [mysqld] innodb_adaptive_hash_index=OFF

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Composite IndexesSpans Multiple Columns Increases Search Speed Best when All Columns in Index

    Specified in Query Not Used with Latter Columns of Index

    48

    CREATE TABLE people (id INT AUTO_INCREMENT KEY,

    name_first CHAR(20), name_last CHAR(20), city CHAR(20), KEY names(name_first,name_last,city));

    SELECT * FROM people WHERE name_first = 'Bob' AND name_last = 'Smith';

    SELECT * FROM people WHERE name_last = 'Smith';

    Index Used

    Index Not Used

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Joining TablesJOIN is a Cartesian Product of Table Subsets with

    Variations The Optimizer can Help

    Orders Tables for Minimum Cost Fewest Rows Examined Converts OUTER to INNER when Possible

    Ensure Columns in ON and USING Clauses are Indexed

    Restrict GROUP BY and ORDER BY to Columns in same Table

    Consider JOIN Decomposition

    49

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Query AnalysisUse the Slow Log to find Problem Queries (--

    log-slow-queries)

    Use mysqldumpslow Utility for Manageable Reports

    Use EXPLAIN to see how MariaDB Executes a Troublesome Query and if Indexes are Used

    Use EXPLAIN EXTENDED and SHOW WARNINGS to see how MariaDB Rearranges a Query before Execution

    50

    Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/

    EXPLAIN SELECT * FROM employees WHERE MONTH(birth_date) = 8 \G id: 1 select_type: SIMPLE

    table: employees type: ALL possible_keys: NULL key: NULL

    key_len: NULL ref: NULL rows: 299587 Extra: Using where

    EXPLAIN EXTENDED SELECT ; SHOW WARNINGS;

    https://mariadb.com/kb/en/mariadb/explain/https://mariadb.com/kb/en/mariadb/explain/#explain-extendedhttps://mariadb.com/kb/en/mariadb/mysqldumpslow/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Fields from EXPLAINField Descriptionid Query identifierselect_type Type of SELECT performed

    table The table in use (each table in a JOIN will use one row)

    *type The table access strategy

    possible_keys Any available indexes that could resolve the query

    key Index MariaDB chose from possible_keyskey_len The portion of the chosen index to be used (width in bytes)ref Columns from the index, or a constant, usedrows Number of rows the query is expected to touchExtra Extra information relating to the access strategy

    51

    Documentation on EXPLAIN: https://mariadb.com/kb/en/mariadb/explain/ Documentation on EXPLAIN EXTENDED: https://mariadb.com/kb/en/mariadb/explain/#explain-extended

    * See Next Slide for List of Types

    https://mariadb.com/kb/en/mariadb/explain/https://mariadb.com/kb/en/mariadb/explain/#explain-extended

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Scan or Index Method Types

    Field Description

    const A constant value can be read once and cached

    eq_ref One index access per outer query row

    ref Multiple index accesses per outer query row

    index_merge Multiple indexes used, merging into a single result set

    range Multiple index accesses to return all rows within a range

    index Full index scan (every index entry is read sequentially)

    all Full table scan (every record is read sequentially)

    Method Used to Access Table

    52

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    EXPLAIN select_typeType of SELECT being Performed

    53

    Type of SELECT Description

    SIMPLE Normal SELECT Statement

    PRIMARY First (Outermost) SELECT in JOIN

    UNION Subsequent Query in JOIN

    DEPENDENT UNION Subsequent Query Dependent on an Outer Query Result

    UNION RESULT Query is the Results Set from a UNION

    SUBQUERY First SELECT in a Sub-Query

    DEPENDENT SUBQUERY A Sub-Query that must be Re-Executed for Each Combination of Values in Outer Query

    DERIVED Sub-Query in a FROM Clause

    UNCACHEABLE SUBQUERY

    Sub-Query that must be Re-Executed for Every Iteration of an Outer Query

    UNCACHEABLE UNION Subsequent Query in a UNION belonging to an Uncacheable Subquery

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    EXPLAIN Extraconst row not found

    Impossible HAVINGNo matching min/max rowNot existsSelect tables optimized awayOpen_trigger_onlyUsing filesortUsing join buffer Using

    intersect(...)

    Distinct Impossible WHEREno matching row in const

    tableRange checked for each

    record (index map: N)Skip_open_table Open_full_tableUsing indexUsing index conditionUsing sort_union(...)Using temporary

    Using WHERE with pushed condition

    Full scan on NULL keyImpossible WHERE noticed after

    reading const tablesNo tables used Scanned N

    databasesOpen_frm_only

    unique row not found Using index for group-by Using UNION(...) Using where

    54

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    MySQL OptimizerOptimizer calculates Best Plan

    Can't Take too Long or Defeats Purpose

    Many Possible Plans for Large Joins Greedy Searches used to Reduce

    Possibilities Possible to Adjust Decision Parameters

    Limit Number of Tables Considered (optimizer_search_depth)

    Skip Plans Based on Row Count Estimates (optimizer_prune_level)

    55

    Documentation on optimizer_search_depth: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_search_depth Documentation on optimizer_prune_level: https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_prune_level Article on Setting Optimizer Search Depth: https://mariadb.com/resources/blog/setting-optimizer-search-depth-mysql

    https://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_search_depthhttps://mariadb.com/kb/en/mariadb/server-system-variables/#optimizer_prune_levelhttps://mariadb.com/resources/blog/setting-optimizer-search-depth-mysql

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Optimizer HintsGiving Hints can Affect the Optimizers

    Choices Index Choices USE INDEX IGNORE INDEX FORCE INDEX

    Buffering Methods for GROUP BY SQL_BUFFER_RESULT SQL_SMALL_RESULT SQL_BIG_RESULT

    56

    SELECT * FROM people FORCE INDEX names WHERE name_first = 'Bob';

    Documentation on Forcing Indexes: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/ Documentation on Indexing with GROUP BY: https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/#forcing-an-index-to-be-used-for-order-by-or-group-by

    https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/https://mariadb.com/kb/en/mariadb/index-hints-how-to-force-query-plans/#forcing-an-index-to-be-used-for-order-by-or-group-by

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Slow Query LogContains Queries that took Longer

    than long_query_time Seconds to Execute Off by Default

    Activate when First Using an Application to Monitor Slow Queries

    Default Location is in Data Directory

    Use mysqldumpslow to Process Log

    Use also pt-query-digest as an Alternative to Process Log

    57

    Documentation on Slow Query Log: https://mariadb.com/kb/en/mariadb/slow-query-log-overview/ Documentation on mysqldumpslow: https://mariadb.com/kb/en/mariadb/mysqldumpslow/ Documentation on pt-query-digest: https://www.percona.com/doc/percona-toolkit/2.1/pt-query-digest.html

    [mysqld] log-slow-queries = ON long-query-time = 2

    log-queries-not-using-indexes log-slow-admin-statements log-slow-vervosity = query_plan,explain

    Excerpt from /etc/my.cnf.d/server.cnf

    https://mariadb.com/kb/en/mariadb/slow-query-log-overview/https://mariadb.com/kb/en/mariadb/mysqldumpslow/https://www.percona.com/doc/percona-toolkit/2.1/pt-query-digest.html

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    ORDER BY NULLGROUP BY Triggers an Implicit ORDER BY

    (filesort)

    ORDER BY NULL Avoids It

    58

    EXPLAIN SELECT Continent, COUNT(*) FROM country GROUP BY Continent \G

    ... Extra: Using temporary; Using filesort

    EXPLAIN SELECT Continent, COUNT(*) FROM country

    GROUP BY Continent ORDER BY NULL\G ... Extra: Using temporary

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    ORDER BY RAND( )MariaDB Cant Use an Index for Ordering Entire Results Set must be Generated in a Temporary

    Table Combined with GROUP BY, it can kill servers

    Used Often be Developers when an Application is New or Small

    Doesn't Necessarily Scale Well

    59

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    LIMIT ClausesLIMIT is Evaluated at End of Query Plan

    ORDER BY with LIMIT needs to Sort All Rows

    LIMIT(o, n) will still Parse o + n Rows LIMIT in Sub-Queries Before Sorting

    Scan in Index Order and Apply LIMIT vs. Sort and LIMIT

    60

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    SELECT * StatementSELECT with * Performs Badly

    Increases Size of Results Set Memory and Network Overhead on Client and Server

    Increases Disk Activity for Storage Engines that can Read Partial Records (not MyISAM, but InnoDB)

    Causes Development and Usage Problems Ambiguity with JOIN tables having same Column Names Problems when Schema has been Changed

    61

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Pagination Common ScenariosScenario 1

    Processes Data Set Twice Depends on Complexity of WHERE Clause Worse if User Controlled Filtering

    Allowed

    Scenario 2 The SQL_CALC_FOUND_ROWS Option Causes

    Entire Results Set to be Generated The ORDER BY and LIMIT Clauses become

    a Drain as Data Grows

    62

    SELECT SQL_CALC_FOUND_ROWS FROM table1

    JOIN table2 USING(id) ORDER BY col2 LIMIT 4;

    SELECT FOUND_ROWS();

    SELECT COUNT(*)

    FROM table1 WHERE col1 = 'something' AND col2 = 'something else';

    SELECT * FROM table1

    JOIN table2 WHERE col1 = 'something' ORDER BY col2 LIMIT 100;

    Scenario 1

    Scenario 2

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Pagination Limit Size of Data SetScenario 1

    Archive Old Data to Another Table

    Add an Index to Column (e.g., col_time)

    Scenario 2 The First Column is Assumed to be

    PRIMARY KEY Value Can Be a Constant, Re-

    Generated Periodically

    63

    SELECT * FROM table1 WHERE id < 1000;

    SELECT * FROM table1 WHERE col_time > NOW()

    - INTERVAL 6 MONTH;

    Scenario 1

    Scenario 2

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Pagination Reduce Data Set in ORDER BY and LIMITScenario 1

    First column Assumed to be PRIMARY KEY The ORDER BY and LIMIT Clauses often Use a

    Temporary Table Retrieving Only PRIMARY KEY keeps

    Temporary Table Small Allows 'Using Index' Especially with InnoDB

    Scenario 2 A Safe, Fixed Size Range Access

    64

    SELECT id FROM table1 WHERE col1 = 'something'

    ORDER BY col2 LIMIT 10;

    SELECT col2, col4 FROM table1

    JOIN table2 USING(id) WHERE id IN(10, 14, 28);

    Scenario 1

    Scenario 2

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Sub-QueriesAdvantages of Sub-Queries before MariaDB 5.3 for Certain

    SQL Constructs SELECT...FROM (( (SELECT filter|order|limit) JOIN SELECT

    order|limit ) JOIN column)

    ...WHERE exists (SELECT... LIMIT1) to remove DISTINCT or GROUP BY

    SELECT ( SELECT ) FROM to remove DISTINCT or GROUP BY

    Disadvantages of Sub-Queries before MariaDB 5.3 for Certain SQL Constructs IN or NOT IN that can be Rewritten with JOIN or LEFT JOIN

    WHERE IS NULL Replay for Each Row of Outer Table, No Materialization

    VIEW is a Sub-Query with Additional Security Features

    65

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Common Bottlenecks

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Lack of CachingCaching can Occur at Many Levels

    67

    Application NetworkUser Session Data

    Pre-Compiled, Static ContentProxy Server

    Content Distribution Network memcached

    MariaDB OS

    Query Cache Storage Engine

    File System Cache

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Using Caches Questions to ConsiderDo Users Care if Counter on Web Page is Real Time? Can Any Content be Re-Generated Periodically in a

    Controlled Fashion and Served from a Cache? Can the Application Scale Cheaply, Simply by Sharing

    Caches between Servers? Can the Queries be Rewritten to be Cacheable in the

    Query Cache, or their Results put in memcached?

    For MyISAM tables, does the Server have Sufficient Memory to Operate a File System Cache?

    68

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Using CacheQuery Cache has to be Enabled to Use Use memcached if controlling Server - If not, try MEMORY tables

    Web Application Use for Pagination (Page Count and id = N) Use if Data Not Specific to User and Not Required to

    Update in Real Time (Static, or Mostly Static) Use if User Specific Data lasts Longer than One Page

    Request

    69

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Slow at StartSlow at Start could be from Cold Index & Data

    Caches Try Preloading

    InnoDB Clustered Index

    InnoDB without PRIMARY Indexes For Speed, Try Preloading PK Ranges

    in Parallel MyISAM Key Cache

    70

    SELECT count(*) FROM tbl WHERE idx_col LIKE '%0%';

    LOAD INDEX INTO CACHE;

    SELECT count(*) FROM tbl WHERE non_idx_col=0;

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Temporary Tables on DiskWhen MEMORY Engine Can't Handle an Implicit

    Temporary Table, it Converts to MyISAM on Disk Table contains TEXT or BLOB Columns - Not Supported Small Values for tmp_table_size or max_heap_table_size Record Size greater than 8K (VARCHAR Converting to CHAR) SHOW STATUS, created_tmp_disk_tables Incrementing SHOW PROCESSLIST, Threads in "Creating tmp table on disk"

    Move TEXT and BLOB Columns to another Table Make Large VARCHAR Columns as Small as Possible

    Increase tmp_table_size and max_heap_table_size

    71

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Persistent ConnectionsSome Client Languages and Connection Pool Solutions

    use Persistent Connections (PHP mysql_pconnect( )) Not very Effective a New Client Connection Simpler Client Session Buffers can Stay Allocated too Long if

    the Client Pool of Application is not Efficient, Increasing Memory Usage

    Transactions can be Held Open if Application is Built Poorly, Reducing Concurrency

    72

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Domain Name ServiceMariaDB uses DNS and Reverse DNS for IP and

    Hostname with Client Connections A Slow DNS Server can Slow MariaDB Client

    Connection Running a Local DNS Server Helps with Lookup Speed

    and Security Use --skip-name-resolve to Tell MariaDB Not to do

    DNS Lookup Then Use Only IP Addresses in Permissions Tables

    73

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Architecture

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    MariaDB Terrain

    mysql

    application (e.g., php)

    mysqladmin

    mysqld

    Archive

    InnoDB

    MyISAM

    cron

    DBA

    web user

    socket file

    tcp/ip

    socket file

    ssh & tcp/ip

    Memory

    SQL Tier ClientsStorage Engine

    TierServer Tier

    Handler Interface

    75

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Client Connections

    76

    TCP/IP Connections available on All Platforms --skip-networking (disables)

    Socket Files available on Unix Platforms - fastest choice Named Pipe and Shared Memory available on Windows

    --enable-named-pipe, --shared-memory

    MariaDB Connections require little resources and easy to open.

    Most use External Connection Pools (not needed usually) Set Global Client Connection Limit (max_connections=n)

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Client Libraries and the Like

    Information on MariaDB Connectors: https://mariadb.com/products/connectors-plugins

    Various Libraries for connecting to MariaDB Most are Wrappers for MariaDB C API, which uses MariaDB Network Protocol

    PHP API, Perl DBI, Python, Ruby MariaDB Connector for Java Native C/C++ driver MariaDB ODBC Drive Embedded MariaDB (libmysqld)

    77

    https://mariadb.com/products/connectors-plugins

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    MariaDB Query Process

    Thread Cache

    Archive

    InnoDB

    Aria

    Memory

    Storage Engines

    Query Cache

    SQL Parser

    Optimizer

    Clients

    78

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Thread Cache

    Thread Cache

    Query Cache

    SQL Parser

    Optimizer

    Storage Engines

    Thread is Assigned to Each Connection

    Threads may be Reused from the Thread Cache or Created

    79

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Connection Pool

    Thread Cache

    Query Cache

    SQL Parser

    Optimizer

    Storage Engines

    Main Thread Listens for Client Connections User Authentication based on Host, User, and

    Password Client Buffers for Session Variables and Network

    Communications

    80

    thread_handling = pool-of-threads thread_pool_size = 3

    Excerpt from /etc/my.cnf.d/server.cnf

    Documentation on Thread Pool: https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/#using-thread-pool-scheduler

    https://mariadb.com/kb/en/mariadb/thread-pool-in-mariadb/#using-thread-pool-scheduler

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Query Cache

    Thread Cache

    Query Cache

    SQL Parser

    Optimizer

    Storage Engines

    Documentation on Query Cache: https://mariadb.com/kb/en/mariadb/query-cache/

    SHOW VARIABLES LIKE 'query_cache_type';

    Possible to Stipulate in a Query

    Stores SELECT Query Result Sets

    Useful for High Read, Low Write Servers

    Cache Purged when Related Data Changed

    SELECT SQL_CACHE * FROM ...

    SELECT SQL_NO_CACHE * FROM ...

    81

    No Longer Recommended

    https://mariadb.com/kb/en/mariadb/query-cache/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    SQL Parser

    Thread Cache

    Query Cache

    SQL Parser

    Optimizer

    Storage Engines

    Converts SQL Text to Binary Format Parses SQL into Tokens (i.e., keyword, table, field, value) Applies Grammar Rules to check Validity (Lexical

    Scanner, Grammar Rules Module) Construct a Parse Tree to be passed to Optimizer

    82

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Optimizer

    Thread Cache

    Query Cache

    SQL Parser

    Optimizer

    Storage Engines

    The Optimizer Reads the Parse Tree and Determines an Execution Plan

    Locate any Related Indexes Compare Efficiency of Index Access to Table Scan Determine JOIN order of Table

    Eliminate Unnecessary Tables and WHERE clauses

    Find Indexes that can handle GROUP BY and ORDER BY

    83

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Information Schema (information_schema)Generated as Needed

    No On-Disk Presence Read-Only Tables

    Some Accessible with SHOW statement

    CHARACTER_SETSCOLLATION_CHARACTER_SET_APPLICABILITYCOLLATIONSCOLUMN_PRIVILEGESCOLUMNSENGINESEVENTSFILESGLOBAL_STATUS

    GLOBAL_VARIABLESKEY_COLUMN_USAGEPARTITIONSPLUGINSPROCESSLIST SCHEMATAPROFILINGREFERENTIAL_CONSTRAINTSROUTINESSCHEMA_PRIVILEGES

    SESSION_STATUSSESSION_VARIABLESSTATISTICSTABLE_CONSTRAINTSTABLE_PRIVILEGESTABLESTRIGGERSUSER_PRIVILEGESVIEWS

    84

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Methods of Accessing Information Schema

    SHOW STATUS LIKE '%qcache%';

    Using SHOW Statement

    SELECT * FROM information_schema.global_status

    WHERE VARIABLE_NAME LIKE '%qcache%';

    Accessing Directly

    85

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Information Schema InnoDB Tables

    SHOW TABLES LIKE 'INNODB%'; SHOW TABLES LIKE 'XTRADB%';

    SELECT r.trx_id AS waiting_trx_id, r.trx_mysql_thread_id AS waiting_thread,

    r.trx_query AS waiting_query, b.trx_id AS blocking_trx_id, b.trx_mysql_thread_id AS blocking_thread, b.trx_query AS blocking_query FROM information_schema.innodb_lock_waits w

    INNER JOIN information_schema.innodb_trx b ON b.trx_id = w.blocking_trx_id INNER JOIN information_schema.innodb_trx AS r ON r.trx_id = w.requesting_trx_id;

    Locked & Locking QueriesINNODB_LOCKS

    INNODB_TRX

    INNODB_BUFFER_POOL_STATS

    INNODB_TABLE_STATS

    INNODB_INDEX_STATS

    86

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Server Configuration

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Global Variables

    GLOBAL Variables are System Wide Settings

    Drawn from Configuration File at Start

    Some can be Changed Dynamically by a SUPER user with SET

    88

    SET GLOBAL tmp_table_size = 32*1024*1024;

    SHOW GLOBAL VARIABLES;

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Session VariablesSESSION Variables are Client Specific

    Settings

    Default Values Same as Global Variables, but may be Changed for Client

    89

    SET SESSION tmp_table_size = 64*1024*1024;

    SHOW GLOBAL VARIABLES;

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Memory FootprintCheck if Server is Dedicated to MariaDB Check How much Memory does OS and Other Services

    Need MariaDB Allocates Global and per Session Buffers Global Buffers are Once-Only Allocations at Start (MyISAM

    Key Cache, InnoDB Buffer Pool, etc.)

    Session Buffers are Allocated Multiple Times per Client Connection, Depending on Queries

    MariaDB's Footprint needs to have Space, Depending on how many Concurrent Client Connections Expected

    90

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Global Memory Buffers Allocated once at mysqld Start

    91

    io_read io_write

    Global Memory Buffers are Allocated once when mysqld Starts table_open_cache table_definition_cache query_cache_size thread_cache

    Permissions Tables Global Memory Buffers are also Allocated for Storage Engines

    MyISAM: key_buffer_size InnoDB: innodb_buffer_pool_size, innodb_additional_mem_pool_size, innodb_log_buffer_size

    Storage Engines may Allocate internally Other Memory

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Session Memory

    92

    Buffers Allocated as required for each Client Connection Some Allocated more than once for Joins and Sorting Released when Query is Finished or Client Session Closed Persistent Connections need to be Reset Client Session Memory Variables:

    thread_stack read_buffer_size join_buffer_size bulk_insert_buffer_size

    binlog_cache_size net_buffer_length

    read_rnd_buffer_size max_heap_table_size max_allowed_packet sort_buffer_size

    tmp_table_size myisam_sort_buffer_size

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Estimating Memory Usage

    93

    (sum(session_buffers) * max_connections)

    + sum(global_buffers) = total memory usage

    Worst Case, Maximum Memory Usage Scenario:

    Chances of All Clients Allocating All Possible Session Buffers at once is usually Small

    Normal Practice is to Commit Over Memory Requirements

    Out-of-Memory (OOM) may occur if mysqld is near System Memory Limits

    Add Memory or Use a Crash-Safe Storage Engine

    Use SWAP space, but Avoid letting MySQL Swap for Normal Load

    Basic Formula for Estimating Memory Usage

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    File HandlesLogs and Replication Files need to be Open Client Connections each need a File (socket = file_handle) Storage Engines may have Specific Needs:

    MyISAM uses a file per table.frm, and multiple for table.MYI and table.MYD

    InnoDB needs only a few for Global Table-Space and Logs (unless using innodb_file_per_table)

    Implicit Disk-Based Temporary Tables are Files SELECT INTO OUTFILE uses a file

    94

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Temporary Disk SpaceMariaDB Uses Temporary Disk Space

    Large Implicit Temporary Tables for Resolving Queries Some Sort Operations (ORDER BY, GROUP BY)

    tmpdir defaults to system temporary location Some Unix platforms mount /tmp as tmpfs

    slave_load_tmpdir for Replicating LOAD DATA INFILE Defaults to Location of tmpdir Never use tmpfs Slave Exports from binlog to here, then LOAD DATA

    INFILE

    95

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Disk SpeedUnless a Data Set fits Memory, Disk Speed is Important Disk Reads are Alleviated by Caching Some Storage Engines Implement their own Caches Some Rely on OS Disk Caching Mechanism Disk Writes are Often a Bottleneck Battery Backed Writes are useful for Fast fsync( ) Consider Disk Access Patterns Multiple Disks to Separate

    Random & Sequential Disk IO Random: Most Data Access Can be Query Dependent Sequential: Transaction and Replication Logs

    96

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Group CommitInnoDB Flushes the Redo Log for a Transaction before its Committed Group Commit makes one Write to Logs for Multiple Transactions, instead of Individual Flushes Group Commit as of MariaDB 5.3

    97

    Graphs taken from Facebook Article show MySQL version 5.1.52Benchmark Articles on Group Commit with MariaDB: https://www.facebook.com/note.php?note_id=10150211546215933; https://www.facebook.com/note.php?note_id=10150261692455933 Documentation on Group Commit: https://mariadb.com/kb/en/mariadb/group-commit-for-the-binary-log/

    https://www.facebook.com/note.php?note_id=10150211546215933https://www.facebook.com/note.php?note_id=10150261692455933https://mariadb.com/kb/en/mariadb/group-commit-for-the-binary-log/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Server CPUSince mysqld is multi-threaded, it can

    use many CPU cores A Client Connection is a single thread

    a Query uses only One Core

    Query Concurrency is very Schema and Traffic Dependent

    98

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    MyISAM

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Advantages & Disadvantages of MyISAM

    Indexes are Cached by mysqld in key_buffer Don't Disable System Tables use MyISAM Storage

    Engine (InnoDB in Future)

    100

    Not Transactional

    Table-Level Locking

    Disadvantages

    Poor Crash Recovery

    (Relies on OS to Flush Data to

    Disk)

    Data Caching Relies on OS Disk Cache

    High Read, Low Write

    Traffic

    Concurrent Writes

    Advantages

    Many Column Data Types

    Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

    https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Architecture

    101

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Architecture

    102

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Key BufferOne Key Buffer by Default (key_buffer_size)

    For MyISAM Only, Set Key Buffer to 25 - 50% of Memory (max 4GB)

    Don't set key_buffer_size to 0 - Used for Internal Table Operations

    Consider Key Buffer Hit and Miss Rates Ideal Values are Application Specific Better to have More Reads than Writes

    Run Multiple Key Buffers Allows More than 4GB Dedicating a Key Buffer to a Single Table Prevents Index

    Flushing

    103

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Delayed Key WritesMyISAM Supports Delayed Key Writes Data is Written Immediately to Disk Index Writes are Postponed (delay_key_writes)

    ON needs DELAY_KEY_WRITE in Table Definition ALTER TABLE...DELAY_KEY_WRITE Rebuilds Whole Table

    After Crash, Index Files must be Rebuilt myisam_recover_options = DEFAULT, BACKUP, FORCE,

    QUICK

    104

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    InnoDB & MyISAM Row Formats

    105

    Fixed-Length Data Types Less Data File Fragmentation Heaviest Disk Space Usage

    Static

    Fixed and Variable Length Data Types More Data File Fragmentation More Efficient Disk Space Usage

    Dynamic

    Decompressed as Needed Minimal Disk Space Usage Read-Only

    Compressed

    INT, CHAR, etc.

    VARCHAR, TEXT, etc.

    105

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Concurrent InsertsMyISAM leaves Gaps in Data File when Deletes Rows

    No Automatic Defragmentation

    MyISAM tries to fill Gap with Data from INSERT MyISAM has no MVCC Need to Lock the Table to Avoid tripping a SELECT

    Concurrent Inserts allows INSERT to add Data Concurrently with SELECT

    Not Equivalent of InnoDB's MVCC, but can Improve Concurrency

    106

    concurrent_inserts = 0 Concurrent Inserts Disabled

    concurrent_inserts = 1 Allow Concurrent Inserts when a Table has no gaps (Default)

    concurrent_inserts = 2 Allow even When a Table has Gaps

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    TrapsMyISAM always Reads Whole Row from Disk, Not Partial

    Row Large, Infrequently Needed Fields may be Better

    Elsewhere in one-to-one Relationship MyISAM used for Implicit Temporary Tables on Disk

    Never Reduce MyISAM Resources to Zero (key_buffer_size, etc.)

    Ensure tmpdir Location has Plenty of Space - Cartesian Products can be Large

    Check Settings of myisam_sort_buffer_size and myisam_max_sort_file_size

    MyISAM Repairs use Key Buffer when too Small

    107

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    InnoDB

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Advantages & Disadvantages of InnoDB

    109

    Read Uncommitted Read Committed Repeatable Read (no fantom rows) Serializable

    Fully Transactional,

    ACID Compliant

    Data & Indexes Cached by mysqld

    in Buffer Pool

    Row-Level Locking for High-

    Concurrency

    Supports Four Isolation Levels

    Supports Foreign Keys and Multi-

    Version Concurrency

    Control

    Reliable Crash Recovery

    Slower than MyISAM, but

    Higher Concurrency

    Uses a Global Table-Space File

    and two Redo Log Files

    Advantages Disadvantages

    Isolation Levels

    Documentation for InnoDB & XtraDB: https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

    109

    https://mariadb.com/kb/en/mariadb/xtradb-and-innodb/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    InnoDB Architecture

    110

    Additional Memory Pool

    Buffer Pool *

    Data Files * **

    Redo Log (File 1/2)

    Redo Log (File 2/2)

    Memory

    Disk

    * Cached Data & Indexes, Stored in Pages ** Transaction Data Writes *** Data, Index, and Undo Log Files

    INSERT, UPDATE, DELETE

    1

    COMMIT

    CHECKPOINT

    Log Buffer * *

    2

    3

    110

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Buffer PoolFixed Size Pool of Memory Pool by InnoDB

    Row Data and Index Pages Adaptive Hash Index Insert Buffer Lock Structures Misc. Other Housekeeping

    111

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Buffer PoolLarger is Generally Better for innodb_buffer_pool_size

    For Servers with Only InnoDB Tables, 80% of Available Memory

    MyISAM is Used Internally and Needs Resources along with OS

    Dirty Buffer Pool Pages are Flushed in Background Allows Changes to be Merged and Written Sequentially to Disk Applications with Plenty of Writes, or Write Spikes, may need

    Adjusting Use innodb_max_dirty_pages_pct to Control Frequency of

    Flushing Dirty Pages

    InnoDB Plugin can Disable the Adaptive Hash Index Mostly Improves Performance, but Not Always

    112

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    IndexesInnoDB Indexes are Clustered

    Row Data is Stored with Primary Key

    Use Short Surrogate Primary Keys (INT UNSIGNED AUTO_INCREMENT)

    Secondary Index Entries hold a Copy of PK Large PK Increases All

    Table Index Statistics are Vague for InnoDB Gathered by a Series of Random Index Dives It's Possible for Stats to be Inaccurate and Cause

    Optimizer to Choose Bad Query Plan Run Periodically ANALYZE TABLE to Regenerate Stats

    113

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    InnoDB Clustered Index

    114

    root

    node node

    leafleaf leaf

    PAGE (16KB) PAGE (16KB) PAGE (16KB)

    B-Tree

    HEADER PRIMARY KEY ROW DATA

    leaf

    114

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    InnoDB Secondary Index

    115

    root

    node node

    leafleaf leaf

    PAGE (16KB) PAGE (16KB) PAGE (16KB)

    B-Tree

    HEADER VALUES OF INDEXED COLUMNS PRIMARY KEY

    leaf

    115

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Transaction LogConverts Transactions from Random I/O to Sequential

    Transaction Data is Written to Transaction Log for Durability and Speed - Not Directly to Data Files

    Log Files Rotate as a Circular Buffer with Data Merged to Disk in Background

    116

    fsync() Frequency

    Variable Setting Write and Flush Action

    innodb_flush_log_at_trx_commit = 0 Flush on Checkpoint

    innodb_flush_log_at_trx_commit = 1 Flush at Commit (Default)

    innodb_flush_log_at_trx_commit = 2 Write at Commit, but No Flush

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Transaction LogLarge innodb_log_file_size (Default 5MB) Results

    Usually in Less I/O and Better Performance Crash Recovery Delay Increases with Log File Size

    Increasing innodb_log_buffer_size (Default 1MB) May Reduce Log File I/O for Large Transactions

    117

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Isolation LevelsInnoDB Supports All Four Transaction Isolation Levels*

    READ UNCOMMITTED READ COMMITTED REPEATABLE READ SERIALIZABLE

    * Listed in Order of Increasing Performance Potential

    Increasing Isolation Protection means More Locks Needed

    Reduces Query Concurrency Increases Memory Usage

    Set Isolation Level on Per-Session Basis for Performance for Transactions Requiring More Protection

    Leave Bulk of Traffic to Operate at Lower Isolation Level

    118

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    The InnoDB CoreVariable innodb_thread_concurrency Sets Number of

    Threads that may Access simultaneously InnoDB Core

    Set Based on Application and Traffic Start about Double number of CPU Cores Increase Slowly to Watch for System Spikes or General

    Thread Thrashing, Reduced Through-Put

    119

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    The InnoDB Core

    Variable innodb_flush_method Controls How Data is Written to Disk

    Default is a Normal fsync( ) Operation

    Leads to "Double Buffering" of Data (Buffer Pool and OS Disk Cache)

    O_Direct Allows InnoDB to Use Direct I/O

    Bypasses any OS Cache Changing Flush Method should be Tested Thoroughly Some Combinations of Device, Filesystem, OS may

    Not Benefit

    120

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    TablespacesInnoDB Stores Tables in Tablespace Files

    Single Global Table Space is Used by Default Use innodb_file_per_table to Use Separate

    Tablespaces Easier Maintenance More File Handles Required Two Additional for Each

    Table

    121

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    InnoDB Plugin

    New Pluggable version of Original Built-In InnoDB

    Better Diagnostics Benchmarking Shows Faster & Higher

    Throughput Optional Compressed File Format

    Configurable IO Threads on All Platforms

    MariaDB 5.5 Uses InnoDB Plugin by Default

    122

    CREATE TABLE ROW_FORMAT=COMPRESSED;

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    InnoDB General TipsDo Bulk Data Loads

    Do in PK Order (Sequential Ascending) Batch to Allow Background Threads Keep Pace

    Avoid Large ROLLBACK Operations No Equivalent of Log and Insert Buffers Exist to Reduce

    Rollback Disk I/O

    RAID with Battery Back-Up Unit is Preferable Durability Benefit and some Drivers Allow Disk Sync

    Operations Return after Writing Only to BBU Cache

    Use when possible Newer InnoDB Plugin or XtraDB Many former Bottlenecks have been Fixed Supports Compression

    123

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    MariaDB Replication

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Purpose of MariaDB Replication

    125

    Load Balancing (Scaling SELECT Queries) Move Slow, Heavy Queries to Slave Take Slave Off-Line to Make Back-ups

    Multiple Data Centers Need Fast Reads Gain Redundancy (High Availability)

    Fail Over - Promote Quickly a Slave to Master Fail Over Isnt Automatic Requires External Monitoring

    Minimal Downtime for Upgrades or Schema Changes Apply Changes to a Slave Promote Slave to Master and Redirect Traffic Apply Changes to Master and Switch

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Replication Terrain

    126

    mysqld Data Storage

    Master

    Binary Log

    mysqld Data Storage

    Slave 1

    Relay Log

    Data Storage

    Slave 2/Master 2

    Relay Log

    Binary Log

    mysqldSlave 2A

    Slave 2B

    Slave 2C

    INSERT UPDATE DELETE

    CREATE ALTER DROP

    Dump Thread

    Client Threads

    SQL Thread

    IO Thread

    IO Thread

    IO Thread

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    MariaDB Replication Factors

    127

    One Master, Multiple Slaves No true Multi-Master Solution, but Circular Replication

    Close to Real Time, but Asynchronous Semi-Synchronous Replication Mode Crash-Safe Slaves with Transactional Storage Engines

    A Slave may also be a Master Set log_slave_updates in Configuration File Apply Optionally Replication Filtering Rules or Storage

    Engine Changes on Intermediate Slaves

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Replication Threads

    128

    Master binlog Dump Thread Pushes binlog Events to Slave Visible in SHOW PROCESSLIST as "Binlog Dump"

    Slave IO Thread Visible in SHOW SLAVE STATUS Requests and receives binlog events from the Master Writes them to the local relay log

    Slave SQL Thread Visible in SHOW SLAVE STATUS Reads the Relay Log and Executes Queries on Local Data Checks the Query Result Codes Match those Recorded by Master

    Slave Multiple Execution Threads Multi-Threaded Slave separates events based on Database Names Updates are Applied in Parallel, Not Sequence

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Parallel Replication

    129

    Documentation on Parallel Replication: https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

    Replication Process on Slaves Events Received from Master by IO Thread and Queued

    in Relay Log Each Relay Log Entry is Retrieved by the SQL Thread Each Transaction is Applied to the Slave

    On Non-Parallel Systems, Application Performed Sequentially by SQL Thread

    On Parallel Systems, Application Performed in Pool of Separate Replication Worker Threads

    https://mariadb.com/kb/en/mariadb/documentation/replication/standard-replication/parallel-replication/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Topologies

    130

    Master to Slave Simplest Solution and Used Most Widely Allows Off-Loading of SELECT Traffic to Slave

    Master1 to MasterN ... to Master1 (circular) Servers Replicate in a Circle, with binlog Events

    Traversing the ring until Originating Server Does Not Alleviate Heavy Write Load Needs Careful setup of server-id and

    auto_increment_offset, auto_increment_increment settings

    Master to Slave to Slaves Can build Complex Trees Useful for Replication Rules or Storage Engine Changes

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Master Configuration

    131

    GRANT REPLICATION SLAVE ON *.* TO 'maria_replicator'@'52.34.19.24' IDENTIFIED BY 'rover123';

    Enable Binary Log Choose Binary Log Format Set server-id in Configuration File to Unique Value

    Create Replication User Account on Master

    Make a Consistent Snapshot of Data on Master

    mysqldump -p -u admin_backup --master-data --flush-logs \ --all-databases > full-dump.sql

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Slave Configuration

    132

    Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/ Documentation on CHANGE MASTER TO: https://mariadb.com/kb/en/mariadb/change-master-to/

    CHANGE MASTER TO

    MASTER_HOST='35.161.145.71', MASTER_PORT=3306, MASTER_USER='maria_replicator', MASTER_PASSWORD='rover123';

    Set server-id in Configuration File to Unique Value

    Add read-only in Configuration File to Prevent Writes Set Optionally Replication Rules Covered Later in Class Restart MariaDB

    Load Data from Master

    Execute START SLAVE on Slave

    mysql -p -u root < full-dump.sql

    https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/https://mariadb.com/kb/en/mariadb/change-master-to/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Monitoring Replication

    133

    Documentation on SHOW MASTER STATUS: https://mariadb.com/kb/en/show-master-status/ Documentation on SHOW SLAVE STATUS: https://mariadb.com/kb/en/mariadb/show-slave-status/

    SHOW MASTER STATUS;Check Regularly Status on Master Includes binlog number and position

    Check More Often Status of Replication on Slave

    SHOW SLAVE STATUS \G

    Slave_IO_State: Waiting for master to send event Slave_IO_Running: Yes

    Slave_SQL_Running: Yes Last_Errno: 0 Last_Error: Seconds_Behind_Master: 300

    https://mariadb.com/kb/en/show-master-status/https://mariadb.com/kb/en/mariadb/show-slave-status/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Replication Files

    134

    Documentation mysqlbinlog: https://mariadb.com/kb/en/mariadb/using-mysqlbinlog/

    Binary Log Files (Master) Master Records Write-Queries to File Rotated when Flushed or Periodically to New Log File File

    Name Pattern (.000001)

    Relay Log File (Slave) Record of Master binlog Events Rotated when Flushed or Periodically File Name Pattern (.

    000001)

    Replication Configuration Recorded in master.info (Slave) Name of Relay Log File Recorded in relay-log.info

    (Slave)

    https://mariadb.com/kb/en/mariadb/using-mysqlbinlog/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Slave Configuration Files

    135

    52.89.128.176 maria_replicator rover123 3306 60

    0 ...

    master.info

    196803 mariadb-bin.000040

    196513 30217 9 ...

    relay-log.info

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Replication File Maintenance & Back-Ups

    136

    Replication Files Updated & Purged Automatically Dont Edit or Move Manually

    Include Replication Files when Making Binary Back-ups

    Use --raw option with mysqlbinlog to Back-up Binary Log

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Binary Log FormatStatement Based (SBR) Original Queries are Replicated

    Least Data Sent over Wire and Tested for Years Non-Deterministic Statements Executed on Slave Slave Load is

    Increased vs. RBR

    Row Based (RBR) - Table Rows are Replicated Only Non-Deterministic Statements Executed on Master Slave

    Load is Reduced vs. SBR More Data sent over Wire Not Supported by All Engines

    Mixed (default) - Smart Switching between SBR and RBR Checksum (--binlog-checksum) in Binary and Relay Logs to detect Errors

    Includes Errors in Memory, Disk, Network and Database Can be Implemented for each Slave

    137

    Documentation on Binary Log Format: https://mariadb.com/kb/en/mariadb/binary-log-formats/

    [mysqld] binlog-format=MIXED

    https://mariadb.com/kb/en/mariadb/binary-log-formats/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Slave Filtering Rules Database Level

    138

    Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

    Exclude Specific Databases (e.g., mysql)

    CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (), REPLICATE_DO_DB = (sales,inventory);

    CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (mysql);

    Include Specific Databases

    Excluding can Cause Problems with Joins

    SET GLOBAL replicate_ignore_db = ''; SET GLOBAL replicate_do_db = 'sales,inventory';

    SET GLOBAL replicate_ignore_db = 'mysql';

    https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Slave Filtering Rules Table Level

    139

    Documentation on Slave Options: https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

    Ignore Specific Tables

    CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB = (employees), REPLICATE_DO_TABLE = (employees.names, employees.contacts);

    CHANGE REPLICATION FILTER REPLICATE_IGNORE_TABLE = (employees.salary);

    CHANGE REPLICATION FILTER

    REPLICATE_IGNORE_DB = (sales), REPLICATE_DO_TABLE = (sales.europe_%), REPLICATE_IGNORE_TABLE = (sales.europe_uk_%);

    Include Specific Tables

    Wildcards for Multiple Tables

    https://mariadb.com/kb/en/mariadb/replication-and-binary-log-server-system-variables/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    MariaDB Replication - Asynchronous

    140

    Master Doesnt Wait for Slaves IO Thread may be Slow to Receive binlog Packets

    Network Congestion or Disconnects

    SQL Thread may be Slow in Processing Relay Log Events

    Load on Slave or Network Problems

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Semi-Synchronous

    141

    Documentation on Semi-Synchronous Replication: https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

    Implemented with an optional Plug-In A COMMIT on Master can Wait for a Slave to

    Acknowledge its Received Transaction Master Waits for Slave to Write Transaction

    to Relay Log, Not to Execute Transaction Slave SQL Thread may still Lag

    One Slave Response needed for Master to Continue (Semi-Synchronous, Not Synchronous)

    Can Affect Significantly Performance of Master

    INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';

    INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';

    https://mariadb.com/kb/en/mariadb/semisynchronous-replication/

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Lagging Slave

    142

    Documentation on Time-Delayed Replication: http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.html

    When Slave SQL Thread is Slow or Disabled due to Errors, Slave is said to Lag behind Master

    Slave SQL Thread must Execute Serially Queries that were Executed in Parallel on the Master

    Slave Multiple Execution Threads Multi-Threaded Slave separates events based on

    Database Names Updates are Applied in Parallel, Not Sequence

    Time-Delayed Replication - Set Provides a Buffer to Stop Replication of Mistakes

    CHANGE MASTER TO MASTER_DELAY = seconds;

    http://dev.mysql.com/doc/refman/5.6/en/replication-delayed.htm

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Troubleshooting Problems with Replication

    143

    Check Slave Error Log for Errors affecting Replication Look for Disconnects from Network Problems Binary or Relay Log Event Corruption will cause Slave

    SQL Thread to Stop Different Query Error Codes on Slave indicate its Not

    Synchronized with Master

    Tools like Maatkit can help with Replication Troubleshooting and Recovery

    May Need to Rebuild Slave from a fresh Snapshot (Back-up of Master or Another Slave)

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Maintenance Impact

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    OverviewMaintenance Tasks can Impact Heavily

    a System Backups, Table and Index Optimization,

    Schema DDL, etc. Some can Alleviate such Impact

    145

    Type of Operation PerformedHot Operation Warm Operation Cold Operation

    No Locks Required Possible High Disk/Network I/O Runs usually Concurrently with Normal Traffic

    System must be in Read-Only Mode Impact Depends on Application and Timing Related to Traffic

    System must be Stopped Completely Results in Complete Down Time for Application

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Back-UpsBinary Back-Up is Usually Faster than Logical Back-Up

    Faster Often for Creation and Restoration May be Transparent to Application, Depending on

    Method Often a Warm Operation (Read Lock Only) Consider Storage

    Engine

    Incremental Back-Ups can be very Low Impact Synchronized with Binary Log Full Back-Up Daily and Periodic binlog Snapshots

    Require Only FLUSH LOGS Slower to Restore than Full Back-Up

    146

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Table & Index OptimizationOPTIMIZE is Warm Operation (Read Lock)

    Perform during Low Traffic or Scheduled Maintenance Time

    OPTIMIZE LOCAL is Non-Replicated Version Can be Applied to Slaves one-by-one so Overall

    Cluster Impact is Minimal and Predictable

    ANALYZE is Warm operation (Read Lock) Impact is Storage Engine Specific MyISAM is Equivalent to Full Table Scan InnoDB is Fast Series of Random Index Dives

    147

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Schema AlterationsMost DDL Commands are Warm Operations DDL Involves Generally a Table Rebuild

    Table is Read Locked DDL is Performed on Copy of Table Two Tables are Swapped Lock is Released

    pt-online-schema-change Might Help

    DDL Operations can be a Long Process Best Performed in Scheduled Maintenance Time Test Carefully on Staging System Changes can be Made on Slave First, then Promoted to Master

    148

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Conclusion

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    On-Line ResourcesDocumentation Knowledge Base Open-Source Community

    Post Questions on Forums Ask Questions on IRC FreeNode

    150

    MariaDB Documentation and Knowledge Base: http://mariadb.com/kb

    http://mariadb.com/kb

  • [email protected]

    Version 3.2, Slide Copyright MariaDB Ab. Commercial in ConfidenceMariaDB Performance Tuning

    Other Training CoursesPublic In-Person Courses Private In-Person Courses Live Virtual Training Courses Self-Paced On-Line Courses

    151

    Training Courses: https://mariadb.learnupon.com/store Upcoming Courses: https://mariadb.learnupon.com/store/sessions

    https://mariadb.learnupon.com/storehttps://mariadb.learnupon.com/store/sessions

  • MariaDB Performance Tuning

    Introduction

    Tuning Overview

    Collecting Information

    Schema Tuning

    Query Tuning

    Common BottlenecksArchitecture

    Server Configuration

    MyISAM

    InnoDB

    MariaDB Replication

    Maintenance Impact

    Thanks for Participating

    MariaDB Performance Tuning: Introduction: MariaDB Performance Tuning_2: Introduction_2: MariaDB Performance Tuning_3: Introduction_3: MariaDB Performance Tuning_4: Introduction_4: undefined: undefined_2: undefined_3: MariaDB Performance Tuning_5: Introduction_5: undefined_4: undefined_5: undefined_6: undefined_7: undefined_8: undefined_9: undefined_10: undefined_11: undefined_12: Field: Description: const: eqref: ref: range: index: Full index scan every index entry is read sequentially: all: Full table scan every record is read sequentially: Type of SELECT: Description_2: SIMPLE: Normal SELECT Statement: PRIMARY: First Outermost SELECT in JOIN: UNION: Subsequent Query in JOIN: Subsequent Query Dependent on an Outer Query Result: UNION RESULT: Query is the Results Set from a UNION: SUBQUERY: First SELECT in a SubQuery: DEPENDENT SUBQUERY: A SubQuery that must be ReExecuted for Each Combination of Values in Outer Query: DERIVED: SubQuery in a FROM Clause: UNCACHEABLE: SubQuery that must be ReExecuted for Every Iteration of an Outer Query: UN