Geek Sync: Field Medic’s Guide to Database Mirroring

60
Field Medic’s Guide to Database Mirroring August 18, 2015

Transcript of Geek Sync: Field Medic’s Guide to Database Mirroring

Page 1: Geek Sync: Field Medic’s Guide to Database Mirroring

Field Medic’s Guide to Database Mirroring August 18, 2015

Page 2: Geek Sync: Field Medic’s Guide to Database Mirroring

Hi, I’m Kendra Little

Page 3: Geek Sync: Field Medic’s Guide to Database Mirroring

What you’ll learn today

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Page 4: Geek Sync: Field Medic’s Guide to Database Mirroring

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Page 5: Geek Sync: Field Medic’s Guide to Database Mirroring

Mirroring: Your friend since 2008 Became supported in SQL Server 2005 SP1

An important performance enhancement called ‘Log Stream Compression’ was added in SQL Server 2008

Works in Standard Edition (synchronous only) •  “Basic Availability Groups” are a planned feature for SQL

Server 2016 that we believe will be in Standard Edition, but…

Page 6: Geek Sync: Field Medic’s Guide to Database Mirroring

(continued) Mirroring is much easier to manage than Availability Groups

• No Windows Failover Cluster required • Much simpler to document • Management can all be done using SSMS/TSQL

Mirroring has perks, too! • Automatic page repair • Doesn’t require a domain / can be used with multiple domains

(using certificates)

Page 7: Geek Sync: Field Medic’s Guide to Database Mirroring

Mirroring is ‘deprecated’ … but this doesn’t mean it’s dead

It will be removed in a future version of SQL Server • What version that is hasn’t been disclosed. Features may stay

‘deprecated’ for many many versions without being removed.

Fully supported in SQL Server 2012 and 2014

Well known, widely documented, established feature • Supported by many vendors • Typically stable

Page 8: Geek Sync: Field Medic’s Guide to Database Mirroring

Not ending a TSQL Statement in a “;”

Also deprecated…

Page 9: Geek Sync: Field Medic’s Guide to Database Mirroring

Mirroring modes Choose between safety and performance

“High Safety” mode • Synchronous • Automatic failover is optional • Witness – use a witness if you’re using automatic failover. (Can

be Express Edition)

“High performance” mode • Asynchronous • Enterprise Edition only • No automatic failover • Do not use a witness with this mode

Page 10: Geek Sync: Field Medic’s Guide to Database Mirroring

An example: Async

Page 11: Geek Sync: Field Medic’s Guide to Database Mirroring

Database Mirroring

Page 12: Geek Sync: Field Medic’s Guide to Database Mirroring

Problems mirroring doesn’t solve

Databases are mirrored individually, and fail over individually • No distributed transaction support (not in AGs prior to 2016,

feature is planned at date of this presentation) •  If you want them to fail over as a group, you have to write

custom code and make sure it happens

You must keep SQL Agent Jobs and logins in sync and manage where they run

Mirroring lots of databases on one instance can be difficult due to thread requirements (Also true for AGs)

Page 13: Geek Sync: Field Medic’s Guide to Database Mirroring

Mirroring is the right choice when…

One mirror is enough

Individual database failover is enough (not a group)

You aren’t ready to manage the complexity of a cluster

And you may not have many other choices if you’re using SQL Server 2008 or SQL Server 2008R2

Page 14: Geek Sync: Field Medic’s Guide to Database Mirroring

Demo: automatic page repair in action

Page 15: Geek Sync: Field Medic’s Guide to Database Mirroring

This database is mirrored

Page 16: Geek Sync: Field Medic’s Guide to Database Mirroring

High safety with no witness – no automatic failover

Page 17: Geek Sync: Field Medic’s Guide to Database Mirroring

The view from the GUI

Page 18: Geek Sync: Field Medic’s Guide to Database Mirroring

We change to high performance mode

Page 19: Geek Sync: Field Medic’s Guide to Database Mirroring

Let’s corrupt a page

dbo.CriticalTable has only one data page

The table has a single row where the ‘val’ column is quoting a famous sloth meme

We’re going to corrupt that page and see if mirroring can fix the issue

Page 20: Geek Sync: Field Medic’s Guide to Database Mirroring

Here’s the offset of the page in the data file

We have to shut down the SQL Instance this time to edit the page

Page 21: Geek Sync: Field Medic’s Guide to Database Mirroring

Open the data file with a hex editor

Page 22: Geek Sync: Field Medic’s Guide to Database Mirroring

CTRL+G, set offset as dec relative to begin

Page 23: Geek Sync: Field Medic’s Guide to Database Mirroring

Type over the values to corrupt

Page 24: Geek Sync: Field Medic’s Guide to Database Mirroring

Save and click YES

Page 25: Geek Sync: Field Medic’s Guide to Database Mirroring

Start the SQL Server back up

Page 26: Geek Sync: Field Medic’s Guide to Database Mirroring

Try reading the page

Page 27: Geek Sync: Field Medic’s Guide to Database Mirroring

Try again

Page 28: Geek Sync: Field Medic’s Guide to Database Mirroring

Verify what happened

Page 29: Geek Sync: Field Medic’s Guide to Database Mirroring

The fine print Automatic page repair only kicked in when corruption was detected

• Maximum of 100 rows per database • Rows in sys.dm_db_mirroring_auto_page_repair are cleared

on instance restart

You still need • alerts and monitoring •  to set page_verification to CHECKSUM for the database •  to run CHECKDB (to detect corruption where reads haven’t

been happening) •  to investigate root cause when this happens

But isn’t this much better than repairing the page yourself?

Page 30: Geek Sync: Field Medic’s Guide to Database Mirroring

This feature is also in AlwaysOn Availability Groups. Mirrors are just easier to manage!

Page 31: Geek Sync: Field Medic’s Guide to Database Mirroring

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Page 32: Geek Sync: Field Medic’s Guide to Database Mirroring

Setting up logins You can script these all out from SQL Server Management Studio

If you allow SQL Authentication, this can be complex though • You have to look up the passwords yourself, and you don’t

want to save them in a script in plain text • The SID of the login matters so that the login will map up with

the database users, and the SID doesn’t script automatically

Robert Davis has published a free script that helps you automate this and solve both of these problems: http://www.sqlsoldier.com/wp/sqlserver/transferring-logins-to-a-database-mirror

Page 33: Geek Sync: Field Medic’s Guide to Database Mirroring

‘Orphaned users’ If you do have a login where the SID wasn’t transferred over to the mirror, the users associated with the login will be ‘orphans’ – and they won’t work

You can detect and fix this without recreating the login if it happens using sp_change_users_login:

http://msdn.microsoft.com/en-us/library/ms174378(SQL.90).aspx

Page 34: Geek Sync: Field Medic’s Guide to Database Mirroring

CLR Assemblies: Trustworthy? CLR Assemblies are part of the database and are mirrored

But they may not work when you first fail over without extra steps • The TRUSTWORTHY database property isn’t automatically

restored with the database. If you need this, it must be enabled the first time you fail over

• Not all CLR assemblies require this-- you can see your current setting in the sys.databases DMV

Page 35: Geek Sync: Field Medic’s Guide to Database Mirroring

Setting up SQL Server Agent jobs

The easy way to script out all jobs in SSMS:

View: Object Explorer Details

In the Object Explorer Details pane, you highlight all jobs and script them out in a single script

But beware… •  If job steps use log files, make sure the same path exists

where you’re installing it • Make sure you enable/disable the right jobs on the mirror

Page 36: Geek Sync: Field Medic’s Guide to Database Mirroring

Making jobs work auto-magically after failover

You need custom code to help your jobs out: •  If an automatic failover occurs, the right jobs need to run

automatically, too • You need to accomplish this without having jobs fail all the time

if they’re running against the mirror– because that will destroy monitoring for failed jobs

Each job needs to check “Am I the principal?” and act accordingly

This goes for your application jobs as well as maintenance

Page 37: Geek Sync: Field Medic’s Guide to Database Mirroring

Special setup: Certificates If you’re using certificates that are signed in the master database, these need to be set up on the mirror also

Mirroring doesn’t replace the need to securely back up those certificates in a place where you’re sure you won’t use them!

Page 38: Geek Sync: Field Medic’s Guide to Database Mirroring

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Page 39: Geek Sync: Field Medic’s Guide to Database Mirroring

Split brain in High Safety Mode Split brain is as bad as it sounds: everything shuts down

Let’s say you’re using high safety mode with a principal, a mirror, and a witness

•  If the principal can’t see either the mirror or the witness, it can’t know if the database is online at the witness. It will take the database offline.

•  If the mirror can’t see the principal or the witness, it freaks out too, and puts the database offline.

The database is offline everywhere.

We’ve lost quorum!

Constant communication between the principal, mirror, and witness is critical

Page 40: Geek Sync: Field Medic’s Guide to Database Mirroring

Split brain The database

could be online somewhere else. I’d better shut it

down!

Derp derp derp derp

The database could be online

somewhere else. I’d better shut it

down!

Page 41: Geek Sync: Field Medic’s Guide to Database Mirroring

Suspended mirror ‘Suspended’ status can happen after things go wrong

• Forced service • SQL Server bugs in some versions (Examples:

http://support.microsoft.com/kb/2403218 , http://support.microsoft.com/kb/975681 )

‘Suspended’ means that data is not being transferred from the principal to the mirror

• The log will grow on the principal • You’re no longer protected by mirroring

To fix, on either partner run: ALTER DATABASE YourMirrorDB SET PARTNER RESUME;

Page 42: Geek Sync: Field Medic’s Guide to Database Mirroring

Suspended mirror I’ll save all the changes in my

transaction logs for when you come back!

What’s going on?

Page 43: Geek Sync: Field Medic’s Guide to Database Mirroring

Disconnected witness We’ve seen this if there have been network interruptions to the witness, or problems in DNS

Mirroring may keep running (depending on whether or not the communication between the primary and the mirror is interrupted)

If you have a disconnected witness, you may fix this by restarting the endpoints on the witness

To prevent unplanned failovers, I prefer to: • Switch to high performance mode and remove the witness • Repair the witness • Add the witness back and switch to high safety mode

Page 44: Geek Sync: Field Medic’s Guide to Database Mirroring

Disconnected witness I sure hope there’s no network

interruptions…

Me too!

Derp derp derp derp

Page 45: Geek Sync: Field Medic’s Guide to Database Mirroring

Forcing up the mirror Example: you’re using asynchronous mirroring (“high performance” mode). This doesn’t have automatic failover.

The principal server is lost!

You need to bring things up on the mirror. To do this, you run: •  ALTER DATABASE YourMirrorDB SET PARTNER

FORCE_SERVICE_ALLOW_DATA_LOSS;

Discussion: • What do you do to identify how much data was lost? • What other steps need to be done immediately if you have to

do this?

Page 46: Geek Sync: Field Medic’s Guide to Database Mirroring

Mirroring and Virtualization Technically, it’s supported by VMware:

“AlwaysOn Availability Groups and database mirroring can be used in combination with vSphere vMotion, vSphere HA, and DRS to maximize SQL Server availability.”

… SQL Server on VMware Availability and Recovery Options

http://www.vmware.com/files/pdf/solutions/SQL_Server_on_VMware-Availability_and_Recovery_Options.pdf

Page 47: Geek Sync: Field Medic’s Guide to Database Mirroring

But it’s not all that simple Communication is critical

Network interruptions can cause big problems: • Unplanned failovers in mirroring • Loss of quorum

Even planned movements of virtual machines can cause packet loss and communication drops (and we’re not even talking about storage migration) – this can cause failovers

And you need to make sure that all components in mirroring are never on the same host

Page 48: Geek Sync: Field Medic’s Guide to Database Mirroring

Your biggest takeaways Monitoring is critical

•  If you have automatic failover, you need to make sure the witness and mirror are healthy

•  If you ignore failures, the likelihood of split brain taking everything down goes up

Placement of the witness is important •  If you’re using multiple datacenters, the witness should be in

the primary datacenter

Page 49: Geek Sync: Field Medic’s Guide to Database Mirroring

Field medics need to know…

Rescue scenarios

Why mirroring matters Special setup for mirrors

Monitoring mirroring 3

12

4

Page 50: Geek Sync: Field Medic’s Guide to Database Mirroring

sys.database_mirroring This DMV helps you quickly see which databases are mirrored

For each database that’s mirrored, you can see: • Mirroring mode • Who’s the partner and what their state is • Who’s the witness and what their state is • Endpoint address

Page 51: Geek Sync: Field Medic’s Guide to Database Mirroring

Database Mirroring Monitor Job This is created automatically if you use SQL Server Management Studio to set up mirroring

• Runs every one minute by default • Calls the built in sp_dbmmonitorupdate procedure (which you

can use in your own code, if you want)

This job allows you to see the current mirroring status by using: • The Database Mirroring Monitor tool • The sp_dbmmonitorresults procedure

Page 52: Geek Sync: Field Medic’s Guide to Database Mirroring

Launching the monitor Right click on the database name in SSMS

Snoop around under ‘Tasks’

Page 53: Geek Sync: Field Medic’s Guide to Database Mirroring
Page 54: Geek Sync: Field Medic’s Guide to Database Mirroring

Performance counters Per database mirrored

Tracking and baselining these is helpful in case you need to troubleshoot performance problems

Some highlights from the full list: •  Log Send Queue KB •  Redo Queue KB •  Pages sent / sec •  Redo bytes / sec •  Transaction Delay

http://msdn.microsoft.com/en-us/library/ms189931.aspx

Page 55: Geek Sync: Field Medic’s Guide to Database Mirroring

Thresholds and alerts • Set in the Database Mirroring Monitor (hard to see) • These are set per database mirrored • You must set up alerts (with Database Mail and operators) to

get notified, and it’s a lot to tackle if you mirror multiple dbs

Page 56: Geek Sync: Field Medic’s Guide to Database Mirroring

It’s better to buy monitoring You can build your own, but:

• Complex and time consuming to set up alerts for state changes and performance alerts yourself

• You’ll get lots of duplicate alerts (especially if you’re mirroring multiple databases)

• The Database Mirroring Monitor shows a limited amount of history… troubleshooting past failovers can be tricky

Page 57: Geek Sync: Field Medic’s Guide to Database Mirroring

The medic’s advice for mirroring

Page 58: Geek Sync: Field Medic’s Guide to Database Mirroring

Become a mirroring pro Create a test environment

• Run through setup multiple times • Practice planned failover, switching modes

Implement and test monitoring • Notification of state changes (pausing, failovers) • Create a delay by performing multiple index rebuilds on the

primary – do you get notifications of latency? • Shut the witness down – do you get a notification

Make it fail • Practice forcing service on a mirror • Practice recovering afterwards

Page 59: Geek Sync: Field Medic’s Guide to Database Mirroring

Here’s How …

Idera Can Help You.

Page 60: Geek Sync: Field Medic’s Guide to Database Mirroring

www.idera.com