1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant...

1 © 2008 Julian Dyke juliandyke.com

Automatic StorageManagement

Julian Dyke

Independent Consultant

Web Version - December 2008


Objectives

1. Understand how Oracle database files are stored in ASM

2. Calculate how long ASM rebalance operations will take


Agenda

ASM Instances ASM Disk Groups

Metadata Extent Distribution Rebalancing Redundancy


ASMInstances


ASM Single Instance Architecture

OracleClusterware

ASMInstance

RDBMSInstance

Server

OCSSDDaemon

Only

DedicatedStorage


ASM Single Instance Background Processes

PMON

SMON

CKPT

LGWR

DBW0 VKTM PSP0 MMAN RBAL

GMON

X000

DIAG

DIA0

Fixed Area

Variable Area

ASM Cache

Oracle 11.1


ASM RAC Architecture

Oracle Clusterware

ASMInstance

RDBMSInstance

Node 1

Oracle Clusterware

ASMInstance

RDBMSInstance

Node 2

Oracle Clusterware

ASMInstance

RDBMSInstance

Node 3

Oracle Clusterware

ASMInstance

RDBMSInstance

Node 4

Storage Network

Private Network

PublicNetwork

SharedStorage


ASM RAC Architecture

PROD1 TEST1

CLUSTERWARE

+ASM1

PROD2 TEST2

CLUSTERWARE

+ASM2

PROD

TEST

Clusterware

ASM Instances

RDBMS Instances

Database Files


ASM RAC Instance Background Processes

PMON

SMON

CKPT

LGWR

DBW0 VKTM PSP0 MMAN RBAL

GMON

X000

DIAG

DIA0

Fixed Area

Variable Area

ASM Cache

Oracle 11.1

LMON LMS0 LMD0 LCK0 DIAG

MARK KATE

10

© 2008 Julian Dyke juliandyke.com

ASMDisk

Groups


ASM Disk Groups and Disks

Disk Group 1

Disk Group 2

Disk Group 3

Disk 1

Disk 4

Disk 2

Disk 6

Disk 5

Disk 7

Disk 3

12


ASM Disk Groups and Disks

Disk Group 1

Disk 1 Disk 4

Disk Group 2

Disk 2 Disk 5 Disk 6 Disk 7 Disk 3

Disk Group 3

13


ASM Disk Groups, Disks and Database Files

Disk Group 1

Disk 1 Disk 4

Disk Group 2

Disk 2 Disk 5 Disk 6 Disk 7

Disk 3

Disk Group 3

File 1

File 2File 3

File 4

File 5

File 5

File 6

14


File Extents versus Allocation Units File Extent

Logical unit of ASM file Map to allocation units

One to many mapping

Allocation Unit Physical unit of ASM disk Oracle 10.2 and below

Always 1MB Can be increased using _asm_ausize

Oracle 11.1 and above Variable size 1MB, 2MB ,4MB, 8MB, 16MB, 32MB, 64MB

15


X$KFFXP Maps file extents to allocation units Only populated in ASM instance Columns include

Column Name Description

GROUP_KFFXP Disk Group Number

NUMBER_KFFXP File Number

COMPOUND_KFFXP Disk Group Number || File Number

INCARN_KFFXP Incarnation Number

PXN_KFFXP Physical Extent Number (within file)

XNUM_KFFXP Logical Extent Number (within file)

LXN_KFFXP 0=primary, 1=first mirror, 2=second mirror

DISK_KFFXP Disk Number

AU_KFFXP Allocation Unit Number (within disk)

SIZE_KFFXP Size (# allocation units)

16


ASM Metadata Metadata is stored in first 256 files in ASM disk group

Space is initially allocated when disk group is created Can be subsequently extended

Metadata allocation units are divided into blocks Each block is 4096 bytes Block size specifed using _asm_blksize

Metadata files includeFile# Description

0 Metadata Header

1 File Directory

2 Disk Directory

3 Active Change Directory

4 Continuing Operations Directory

5 Template Directory

6 Alias Directory

9 Attribute directory (optional)

12 Staleness registry (optional)

17


ASM Metadata

Active ChangeDirectory

Continuing OperationsDirectory

Template Directory

Alias Directory

Disk Directory

File Directory

Partner Status Table

Metadata Header

Free Space Table

Allocation Table

Disk Header

Metadata Header

18


ASM Metadata Initial Allocation (Single Instance)

File# AU Description # AUs

0 0 Disk Header, Free Space Table, Allocation Table 1

1 Partner Status Table 1

1 File Directory 1

2 Disk Directory 1

3 Active Change Directory 42

4 Continuing Operations Directory 2

5 Template Directory 1

6 Alias Directory 1

Active Change Directory Records changes to metadata Used during recovery of instance or operation failures

Continuing Operations Directory Maintains state of active operations

19


ASM Metadata Block Types

Type Description

1 KFBTYP_DISKHEAD

2 KFBTYP_FREESPC

3 KFBTYP_ALLOCTBL

4 KFBTYP_FILEDIR

5 KFBTYP_LISTHEAD

6 KFBTYP_DISKDIR

7 KFBTYP_ACDC

8 KFBTYP_CHNGDIR

9 KFBTYP_CODBGO

10 KFBTYP_TMPLTDIR

11 KFBTYP_ALIASDIR

12 KFBTYP_INDIRECT

Type Description

13 KFBTYP_PST_NONE

14 KFBTYP_HASHNODE

15 KFBTYP_COD_RBO

16 KFBTYP_COD_DATA

17 KFBTYP_PST_META

18 KFBTYP_PST_DTA

19 KFBTYP_HBEAT

20 KFBTYP_SR

21 KFBTYP_STALEDIR

22 KFBTYP_VOLUMEDIR

23 KFBTYP_ATTRDIR

20


KFED Utility In Oracle 10.2 and above the kfed utility can be used to inspect and edit the

contents of ASM blocks

[oracle@server3 ~]$ $ORACLE_HOME/bin/kfed -has/mlib ASM Library [asmlib='lib']aun/um AU number to examine or update [AUNUM=number]aus/z Allocation Unit size in bytes [AUSZ=number]blkn/um Block number to examine or update [BLKNUM=number]blks/z Metadata block size in bytes [BLKSZ=number]ch/ksum Update checksum before each write [CHKSUM=YES/NO]cn/t Count of AUs to process [CNT=number]d/ev ASM device to examine or update [DEV=string]o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT]p/rovnm Name for provisioning purposes [PROVNM=string]s/eek AU number to seek to [SEEK=number]te/xt File name for translated block text [TEXT=string]ty/pe ASM metadata block type number [TYPE=number]

This utility should only be used under the guidance of Oracle Support

21


KFED Utility For example to dump blocks in aliases directory in DISKGROUP1 Find group number

SELECT disk_kffxp, au_kffxp FROM x$kffxp WHERE group_kffxp = 1AND number_kffxp = 6AND lxn_kffxp = 0;

Disk Allocation Unit

0 49

Find disk name

SELECT group_number FROM v$asm_diskgroupWHERE name = 'DISKGROUP1';

Alias directory is stored in file number 6

SELECT path FROM v$asm_diskWHERE group_number = 1AND disk_number = 0;

Path/dev/oracleasm/disks/VOL1

22


KFED Utility Example (continued)

Allocation unit is 1MB Block size is 4096 Therefore there are 256 blocks per allocation unit Starting block offset = 256 * 49 = 12544

for (( f = 12544 ; f < 12544 + 256 ; f++ ))do

kfed op=read blkn=$f dev='/dev/oracleasm/disks/VOL1' > blk${f}done

23


Extent Distribution

24


Extent Distribution Creating a disk group:

CREATE DISKGROUP diskgroup1 EXTERNAL REDUNDANCYDISK '/dev/oracleasm/disks/VOL1';

Dropping a disk group:

DROP DISKGROUP diskgroup1 INCLUDING CONTENTS

25


Extent Distribution1 disk

Disk 0

Metadata

0

1

2

3

4

5

6

7

Metadata Data

26


Extent Distribution2 disks

Disk 0 Disk 1

Metadata

0

2

4

6

8

10

12

14

1

3

5

7

9

11

13

15

Metadata Data

27


Extent Distribution4 disks


Metadata

0

4

8

12

16

20

24

28

2

6

10

14

18

22

26

30

1

5

9

13

17

21

25

29

3

7

11

15

19

23

27

31

Metadata Data

28


Extent Distribution1 large disk - 1 small disk

Disk 0 Disk 1

Metadata

0

1

3

4

6

7

9

10

2

5

8

11

Metadata Data

29


Extent Distribution1 large disk - 3 small disks

Disk 1 Disk 2

1

6

11

16Metadata

0

3

5

8

10

13

15

18

Disk 0 Disk 4

2

7

12

17

4

9

14

19

Metadata Data

30


Extent Distribution2 large disks - 2 small disks

Disk 1 Disk 2

3

9

15

21Metadata

0

2

6

8

12

14

18

20

Disk 0 Disk 4

1

4

7

10

5

11

17

23

13

16

19

22

Metadata Data

31


Rebalancing

32


Extent Distribution Adding a disk:

ALTER DISKGROUP diskgroup1 ADD DISK '/dev/oracleasm/disks/VOL2' REBALANCE POWER 0;

Dropping a disk:

ALTER DISKGROUP diskgroup1 DROP DISK 'DISKGROUP1_0002' REBALANCE POWER 0;

Rebalancing a disk group:

ALTER DISKGROUP diskgroup1 REBALANCE POWER 1;

33


RebalancingAdding disks - 1 disks to 2 disks

Disk 0 Disk 1

0

1

2

3

4

5

6

7

5

7

0

1

2

3

4

5

6

7

5

7

1

3

1

3

STOP

Metadata Data

34




0

1

2

3

4

5

6

7

2

6

1

5

0

4

0

1

2

3

4

5

6

7

1 0

45

2

6

STOP

Metadata Data

35



Disk 0 Disk 1 Disk 2

1

3

5

7

9

11

13

15

0

2

4

6

8

10

12

14

2

5

8

11

0

1

2

3

4

5

0

31

4

14

17

16

17

Metadata Data

36




1

3

5

7

9

11

13

15

0

2

4

6

8

10

12

14

1

5

9

13

0

4

8

12

0

1

2

3

4

5

6

7

1 0

45

2

6

STOP

Metadata Data

37


RebalancingDropping disks - 3 disks to 1 disk


0

3

6

1

2

4

5

7

4 5

0

3

1

2

4

5

2

5

1

4

8

7 8

1 2

STOP

Metadata Data

38


RebalancingMoving disks - 2 disks to 2 disks


1

3

5

7

9

11

13

15

4

6

8

10

12

14

10

6

2

0

4

8

0

1

2

3

4

5

6

7

0

4

2

6

STOP

2

0

12

14

Metadata Data

39


RebalancingV$ASM_OPERATION Contains details of ongoing rebalance operations

Column Name Data Type

GROUP_NUMBER NUMBER

OPERATION CHAR(5)

STATE VARCHAR2(4)

POWER NUMBER

ACTUAL NUMBER

SOFAR NUMBER

EST_WORK NUMBER

EST_RATE NUMBER

EST_MINUTE NUMBER

ERROR_CODE VARCHAR2(44)

Estimate of remaining time

40


RebalancingPower Limit Power limit can be 0 to 11

0 disables rebalance operation 1 to 11 specifies number of ARBn background processes used for

rebalance

In Oracle 10.2 RBAL manages rebalance operation Each ARBn background process is allocated a range of 128 allocation

units to rebalance When complete another range is requested

AD lock is taken while an allocation unit is being rebalanced Rebalance operations take much longer than theoretically necessary.

Possible reasons include: Locking GES updates with other ASM instances Updates to RDBMS instance

41


RebalancingSummary EST_MINUTES column of V$ASM_OPERATION is reasonably accurate

Allow a few minutes for SAN cache to stabilize Check regularly for changes to estimate

ASM rebalance operations do not affect workload Locks are only taken briefly

Lock mechanism has changed in Oracle 11.1 SAN cache and I/O performance will be affected

In Oracle 10.2 rebalancing is fastest if Other ASM instances are shutdown RDBMS instance is shutdown

Estimated completion time will be affected by: Use of SAN cache and I/O by rest of workload Rate of change by applications to blocks in ASM files being rebalanced

42


Redundancy

43


Redundancy ASM Supports three levels of redundancy

External Redundancy Implemented externally using storage layer Most common configuration in production

Normal Redundancy Two copies of each extent maintained in separate failure groups Used with extended clusters Used occasionally in production e.g. CERN Increases CPU overhead on servers

High Redundancy Three copies of each extent maintained in separate failure groups Very rare in production

44


ASM Failure Groups - External Redundancy

Disk Group


45


ASM Failure Groups - Normal Redundancy

Failure Group 1 FailureGroup 2

Disk Group

Disk 1 Disk 2 Disk 3 Disk 4 Disk 5 Disk 6

46


ASM Failure Groups - High Redundancy

Failure Group 1

Disk Group

Disk 1 Disk 2

Failure Group 2

Disk 1 Disk 2

Failure Group 3

Disk 1 Disk 2

47


Normal Redundancy1 Disk Per Failure Group

Disk 0 Disk 1

0

1

0

1

2

3

2

3

10

11

Failure Group 1

8

9

67

0

1

2

3

45

9

10

7

8

5

6

0

1

234

11

Failure Group 2

Primary SecondaryMetadata

Data Primary Secondary

48


Normal Redundancy2 Disks per Failure Group


0

3

0 1

23

1

2

4

7

5

6

4

7

5

6

20

23

Failure Group 1 Failure Group 2

16

19

12

15

0

3

4

7

8

11

18

21

14

17

10

13

1

2

56

9

22

19

20

15

16

11

12

03

47

8

2321

22

17

18

13

14

1

2

5

69

10

Primary SecondaryMetadata

Data Primary Secondary

49


High Redundancy1 Disk per Failure Group


0 0

Failure Group 1

0 0

Failure Group 2 Failure Group 3

0111

2 2 2

3 33

444

5 5 50

11 1

22 2

3 3 3

44 4

55 5

6 6 6

77 7

88 8

9 9 9

Primary Secondary TertiaryMetadata

Data Primary Secondary Tertiary

50


References Oracle Automatic Storage Management (Oracle Press)

Nitin Vengurlekar Murali Vallath Rich Long

What ASM and ZFS Can Do For You Jason Arneil - Nominet

A Closer Look Inside Oracle ASM Luca Canali - CERN

Implementing ASM Without HW Raid Luca Canali - CERN

51


Thank you for listening

[email protected]

1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant...

Documents

Transcript of 1 © 2008 Julian Dyke juliandyke.com Automatic Storage Management Julian Dyke Independent Consultant...