Webinar manage MySQL like a devops sysadmin

113
Manage MySQL like a devops sysadmin Webinar – Oct 2012 Frédéric Descamps

Transcript of Webinar manage MySQL like a devops sysadmin

Page 1: Webinar manage MySQL like a devops sysadmin

Manage MySQL like a devops sysadmin

Webinar – Oct 2012

Frédéric Descamps

Page 2: Webinar manage MySQL like a devops sysadmin

www.percona.com

Who am I ?

Frédéric Descamps

@lefred

http://about.be/lefred

Managing MySQL since 3.23 (as far as I remember)

devops believer

Page 3: Webinar manage MySQL like a devops sysadmin

www.percona.com

To start with...

Let's use devops like

slides...

Page 4: Webinar manage MySQL like a devops sysadmin

4

Let's use devops like slides...

Page 5: Webinar manage MySQL like a devops sysadmin

5

from

Page 6: Webinar manage MySQL like a devops sysadmin

6

Page 7: Webinar manage MySQL like a devops sysadmin

7

to

Page 8: Webinar manage MySQL like a devops sysadmin

8

Page 9: Webinar manage MySQL like a devops sysadmin

9

devopsis a movement willing to help

aligning IT with company needs

(starting by aligning devs and ops)

@endemics - Gildas Le Nadan - “The Devops movement”, Mixit Lyon 2011

Page 10: Webinar manage MySQL like a devops sysadmin

10

devops is about communication !

Page 11: Webinar manage MySQL like a devops sysadmin

11

devops is about communication !

betweendev and ops

Page 12: Webinar manage MySQL like a devops sysadmin

12

devs ops

Small and frequent successful iterations

This is one of the most fundamental ideas in devops: the feedback loop !

Page 13: Webinar manage MySQL like a devops sysadmin

13

We can say that devops is bringing

together people and practices from

development and operations

Page 14: Webinar manage MySQL like a devops sysadmin

14

We can say that devops is bringing

together people and practices from

development and operations

I am a happy dev !I am a happy op !

Page 15: Webinar manage MySQL like a devops sysadmin

15

devops now

=● being professional● improving the collaboration by increasing trust and respect

Page 16: Webinar manage MySQL like a devops sysadmin

16

/me likes !

HOW can I reach that goal ?

Page 17: Webinar manage MySQL like a devops sysadmin

17

devops requires

visibility !

Page 18: Webinar manage MySQL like a devops sysadmin

18

Everybody should have a status overview

Is there any problem ? How are the test going ? How is performance ?

Page 19: Webinar manage MySQL like a devops sysadmin

19

devops is about CAMS

Page 20: Webinar manage MySQL like a devops sysadmin

20

CULTUREAUTOMATION

MEASUREMENTSHARING

@botchagalupe @damonedwards http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

Page 21: Webinar manage MySQL like a devops sysadmin

21

What techniques can I use ?

Page 22: Webinar manage MySQL like a devops sysadmin

22

MonitoringInfrastructure as

CodeContinuous

Improvement

Page 23: Webinar manage MySQL like a devops sysadmin

23

What about DBAs ?

Page 24: Webinar manage MySQL like a devops sysadmin

24

Join the team, follow the

movement !

Page 25: Webinar manage MySQL like a devops sysadmin

25

You are already between devs

and ops

Page 26: Webinar manage MySQL like a devops sysadmin

26

You are already between devs

and ops

I am a happy dba !

Page 27: Webinar manage MySQL like a devops sysadmin

27

But we are still closer to the

dark side of the force: sysadmins

Page 28: Webinar manage MySQL like a devops sysadmin

28

But we are still closer to the

dark side of the force: sysadmins

Page 29: Webinar manage MySQL like a devops sysadmin

29

How can we apply devops

ideas to our dba job ?

Page 30: Webinar manage MySQL like a devops sysadmin

30

1. communication2. configuration

management3. testing4. visibility

Page 31: Webinar manage MySQL like a devops sysadmin

31

1. Communication

Usually DEVS communicatewith managers, quality managers, products managers, ...

Page 32: Webinar manage MySQL like a devops sysadmin

32

OPS “communicate” with support, ISP's and... their servers

Page 33: Webinar manage MySQL like a devops sysadmin

33

DBAs don't communicate

Page 34: Webinar manage MySQL like a devops sysadmin

34

The interaction between developers, sysadmins, dbas and all the actors

should begin from the start of any projects

Page 35: Webinar manage MySQL like a devops sysadmin

35

Page 36: Webinar manage MySQL like a devops sysadmin

36

Page 37: Webinar manage MySQL like a devops sysadmin

37

Discuss everything database related with developers and

sysadmins:

● disks and raid design● schema design & partitions● additional columns & indexes● tables repair, ...

Page 38: Webinar manage MySQL like a devops sysadmin

38

2. configuration management

Can we automate DBAs job ? Should we ? Even if we have only one database ?

Page 39: Webinar manage MySQL like a devops sysadmin

39

Page 40: Webinar manage MySQL like a devops sysadmin

40

Page 41: Webinar manage MySQL like a devops sysadmin

41

A lot of things to configure● binaries● databases ● configs● cron jobs● monitoring● grants● replication

Page 42: Webinar manage MySQL like a devops sysadmin

42

Documentation is needed and must be up to date

Page 43: Webinar manage MySQL like a devops sysadmin

43

Needs to be reproducible

quickly without missing anything !

Page 44: Webinar manage MySQL like a devops sysadmin

44

Keep track of any changes

Mind the gap

Page 45: Webinar manage MySQL like a devops sysadmin

45

3. testing

Test every changes

every new package... EVERYTHING

Page 46: Webinar manage MySQL like a devops sysadmin

46

4. visibility

Expose monitoring, trending, test results and project status

to EVERYONE in your company!!

Everybody should know why the database is slooooow !

Page 47: Webinar manage MySQL like a devops sysadmin

47

Use all these metrics to plan ahead all the changes !

NOT ALONE BUT WITH EVERYBODY

Page 48: Webinar manage MySQL like a devops sysadmin

48

notions practices

ideas methodologies

Page 49: Webinar manage MySQL like a devops sysadmin

49

We are DBAs we work with data,

we need something more

concrete !

Page 50: Webinar manage MySQL like a devops sysadmin

50

We are DBA's we work with data, we need

something more concrete !

Page 51: Webinar manage MySQL like a devops sysadmin

51

TRUE !

Page 52: Webinar manage MySQL like a devops sysadmin

52

communication

Page 53: Webinar manage MySQL like a devops sysadmin

53

Use an agile method to

handle your projects

Page 54: Webinar manage MySQL like a devops sysadmin

54

For operations a good method is

kanban

Page 55: Webinar manage MySQL like a devops sysadmin

55

Page 56: Webinar manage MySQL like a devops sysadmin

56

Page 57: Webinar manage MySQL like a devops sysadmin

57

Use dashboards and make them

available to EVERYONE

Page 58: Webinar manage MySQL like a devops sysadmin

58

share all your dashboards with

everybody in your company

Page 59: Webinar manage MySQL like a devops sysadmin

59

configuration management

Page 60: Webinar manage MySQL like a devops sysadmin

60

consider your infrastructure as

code

Page 61: Webinar manage MySQL like a devops sysadmin

61

Use puppet (or any other

configuration management tool) to deploy

everything related to your database

Page 62: Webinar manage MySQL like a devops sysadmin

62

Save your puppet recipes into a version control system

Page 63: Webinar manage MySQL like a devops sysadmin

63

but also save your database

schema changes in that system

Page 64: Webinar manage MySQL like a devops sysadmin

64

test

Page 65: Webinar manage MySQL like a devops sysadmin

65

Page 66: Webinar manage MySQL like a devops sysadmin

66

What should you test as a devops

dba ?

Page 67: Webinar manage MySQL like a devops sysadmin

67

Test if you can deploy

everything and if you didn't forget

anything

Page 68: Webinar manage MySQL like a devops sysadmin

68

Test also the performance !Is it faster with this change ?

Page 69: Webinar manage MySQL like a devops sysadmin

69

use Percona Toolkit

Page 70: Webinar manage MySQL like a devops sysadmin

70

automate your tests

Page 71: Webinar manage MySQL like a devops sysadmin

71

visibility

Page 72: Webinar manage MySQL like a devops sysadmin

72

monitoring

Page 73: Webinar manage MySQL like a devops sysadmin

73

Monitor your database with an alerting system (nagios).

Disk/IO, CPU load, Network traffic, disk usage

Page 74: Webinar manage MySQL like a devops sysadmin

74

But do you want to be woken up at night

because replication of the machine taking backups is 60sec

behind ?

Page 75: Webinar manage MySQL like a devops sysadmin

75

NO!

Page 76: Webinar manage MySQL like a devops sysadmin

76

Then do you want to be woken up in the middle

of the night because they are 1000 more selects than usual ?

Page 77: Webinar manage MySQL like a devops sysadmin

77

NO!

Page 78: Webinar manage MySQL like a devops sysadmin

78

Your alerting should answer those questions:

Is the database server running ?Does it allow connections ?

Is replication lagging on production slaves ?

Page 79: Webinar manage MySQL like a devops sysadmin

79

trending

Page 80: Webinar manage MySQL like a devops sysadmin

80

The most interesting facts for a DBA are the trending

results !

Use cacti ... with Percona Monitoring Plugins

They provide very good metrics

Page 81: Webinar manage MySQL like a devops sysadmin

81

Correlate changes in the trending results

with your configuration changes...

everything should be tracked

Page 82: Webinar manage MySQL like a devops sysadmin

82

others ?

Page 83: Webinar manage MySQL like a devops sysadmin

83

You can also use other “devops”

oriented tools like :

Page 84: Webinar manage MySQL like a devops sysadmin

84

puppet-dashboard

with kanban plugin

Page 85: Webinar manage MySQL like a devops sysadmin

85

If there is only one thing you

remember from this presentation

Page 86: Webinar manage MySQL like a devops sysadmin

86

Yes, even a schema change

MUST be considered like a

migration !

Page 87: Webinar manage MySQL like a devops sysadmin

87

review ittest itplan itdo it

Page 88: Webinar manage MySQL like a devops sysadmin

88

and now that we have a

devdbaops team

Page 89: Webinar manage MySQL like a devops sysadmin

89

we can start working on

really interesting tasks for

everyone

Page 90: Webinar manage MySQL like a devops sysadmin

90

the smooth

migration

Page 91: Webinar manage MySQL like a devops sysadmin

91

When this goal is achieved

DEVS have less pressureOPS have less pressureDBAs have less pressure

Page 92: Webinar manage MySQL like a devops sysadmin

92

collaboration =

less pressure

Page 93: Webinar manage MySQL like a devops sysadmin

93

Page 94: Webinar manage MySQL like a devops sysadmin

94

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present future

Page 95: Webinar manage MySQL like a devops sysadmin

95

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present future

Page 96: Webinar manage MySQL like a devops sysadmin

96

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present

Page 97: Webinar manage MySQL like a devops sysadmin

97

v 1.9application v 2.0

db schema

v 1.9 v 2.0

present Issue !!

Page 98: Webinar manage MySQL like a devops sysadmin

98

v 1.9application v 2.0

db schema

v 1.9 v 2.0

Page 99: Webinar manage MySQL like a devops sysadmin

99

v 1.9application v 2.0

db schema

v 1.9 v 2.0

Issue !!

Page 100: Webinar manage MySQL like a devops sysadmin

100

v 1.9application v 2.0

db schema

v 1.9 v 2.0

Page 101: Webinar manage MySQL like a devops sysadmin

101

This can only be the result of

Page 102: Webinar manage MySQL like a devops sysadmin

102

collaboration between DEVS,

DBAs & OPS

Page 103: Webinar manage MySQL like a devops sysadmin

103

demo

Page 104: Webinar manage MySQL like a devops sysadmin

104

gitpuppet

Percona serverJenkins

VirtualBoxvagrant

mcollective

Page 105: Webinar manage MySQL like a devops sysadmin

105

Change the schema (add an index for example)

Commit !

Page 106: Webinar manage MySQL like a devops sysadmin

106

Launches a new virtual machine from scratch using vagrant

Install everything and populate a test db

Page 107: Webinar manage MySQL like a devops sysadmin

107

Start some basic tests with cucumber-puppet to see if all puppet code was correctly performed

No error is a success

Otherwise is a fail

Page 108: Webinar manage MySQL like a devops sysadmin

108

Then start some tests with percona-playback and/orpt-query-digest and compares to old values

Faster is a success

Slower is a fail

Page 109: Webinar manage MySQL like a devops sysadmin

109

Photo Credit: http://www.flickr.com/photos/kuma-kigurumi/http://www.flickr.com/photos/airforceone/2125791018/http://www.flickr.com/photos/akosvaradi/2617404213/http://www.flickr.com/photos/doberdad/2264130166http://www.flickr.com/photos/tomsaint/2714401733http://www.flickr.com/photos/10335017@N07/4570943373/http://www.bxel.net/metro-de-bruxelles/metro_plan_bruxelles.jpghttp://www.flickr.com/photos/youngkimstudio/6216328782http://www.flickr.com/photos/26676383@N00/3690701110/http://www.flickr.com/photos/blubbla/209864856http://www.flickr.com/photos/chrishuffman/2336990347http://geekandpoke.typepad.com/.a/6a00d8341d3df553ef0153922fa2ee970b-pi

Projects links: http://puppetlabs.com/resources/overview-2/http://puppetlabs.com/puppet/related-projects/dashboard/http://theforeman.org/http://git-scm.com/http://www.redmine.org/http://www.redmine.org/projects/redmine/wiki/PluginKanbanhttp://vagrantup.comhttps://www.virtualbox.org/http://jenkins-ci.org/http://www.percona.com/software/percona-toolkit/http://code.google.com/p/mysql-cacti-templates/

Page 110: Webinar manage MySQL like a devops sysadmin

www.percona.com

You can find all these recipes here !

Page 111: Webinar manage MySQL like a devops sysadmin

www.percona.com

Speaking at

In London, December 3rd & 4th

Page 112: Webinar manage MySQL like a devops sysadmin

www.percona.com

Speaking at

In London, December 3rd & 4th

40% off passes using the discount

code “MySQL-UK”

Page 113: Webinar manage MySQL like a devops sysadmin

[email protected]@lefred