Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus...

35

Transcript of Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus...

Page 1: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords
Page 2: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Understanding Oracle RAC (12.1.0.2) Internals: The Cache Fusion Edition Subtitle

Markus Michalewicz Director of Product Management Oracle Real Application Clusters (RAC) November 19th, 2014 @OracleRACpm http://www.linkedin.com/in/markusmichalewicz http://www.slideshare.net/MarkusMichalewicz

Page 3: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

3

Page 4: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 4

The Secret to Horizontal Scalability of Any System – A straight vertical line

Simplification 1 Simplification 2

Page 5: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 6

Three Alternative Architectures – The Same Idea Applies

Note: “germany”, “argentina” and “brazil” are node names. They do not indicate a stretched cluster.

Page 6: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

5

Cache Fusion Overview

(Dynamic Re-) Mastering

Handling Contentions

RAC Meets Multitenant

RAC Meets In-Memory

7

Page 7: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

5

Cache Fusion Overview

(Dynamic Re-) Mastering

Handling Contentions

RAC Meets Multitenant

RAC Meets In-Memory

8

Page 8: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• Requester

• Listener

• Service

• Holder

• Master

• Session PQ/PX:

9

Oracle RAC Combines it All & Adds Services

Instance

Block, unless otherwise stated.

Page 9: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• In an Oracle environment, listeners establish connections with an instance.

• In an Oracle RAC environment two types of listeners are used (for this purpose):

1. SCAN and SCAN Listener

2. Local Listeners per node

• Services are used to load balance work.

• Local listeners will establish connections on instances where the service is offered.

• Once a session is established within an instance, the sessions stays until closed.

10

Listeners and Services Direct Work to an Instance

Page 10: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• Does the application scale on a single (SMP) system?

11

Scaling Principles

write write

write write

write write

write write

write write

write write

write write

write write

• If it does, it is likely to scale horizontally.

• Scalability is measured considering the whole system.

Page 11: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• Local Cache Hit

• Access time: nanoseconds

12

• Remote Cache Hit

• Access time: microseconds

• Data is on disk

• Flash cache access time: microseconds

• Disk controller cache: access time: micros.

• Spinning disk access time: milliseconds

Getting Access to Data

Whether local or remote, cache hits are always faster than reading from spinning disks

Page 12: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 13

Dynamic Retrieval of Data

+ +

2

3

1

• In order to fulfill a given request, Oracle RAC can decide to:

• Assemble data spread across instances and disks

• Perform a (full) disk read for (parts of) the data

• Use assembly and (full) disk reads dynamically

• The decision which access path to use is dependent on various factors (e.g. IO capacity, network utilization)

• Respective parameters are monitored

• The access path can change accordingly and dynamically.

Once data has been shipped to an instance, it resides in the cache for further (local) access. Updates will be

communicated as required (via messages).

Page 13: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Case 1 (all local)

• All entities in one instance • Data holding instance

• Session “holding” instance

• Mastering Instance

14

Case 2 (local / remote)

• Some entities in one instance

•Data holding instance

• Session “holding” instance

• Two way communication

Case 3 (all distributed)

• All entities are dispersed

• Three way communication to obtain a given block of data

• Message operations have been subject to improvements since Oracle RAC 10g times.

Maximum Three Way Communication write write write

Message Message

Block

Page 14: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

5

Cache Fusion Overview

(Dynamic Re-) Mastering

Handling Contentions

RAC Meets Multitenant

RAC Meets In-Memory

15

Page 15: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• In Oracle RAC, every object is mastered (as part of Global Cache Services / GCS) by an instance.

• The Global Resource Directory (GRD) is used to “keep track” of mastering.

• Illustration above is symbolic for this matter.

16

• If a session requests write access to a block that is not mastered by the instance hosting the session, the master instance needs to be contacted (via message).

• Based on user access patterns, a Dynamic Re-Mastering (DRM) can be triggered, changing the master for an object from one instance to another.

• Relocating the master to the instance where data is requested most reduces the messaging.

Mastering and Dynamic Re-Mastering

germany Oracle GI | HUB

Oracle RAC

argentina Oracle GI | HUB

Oracle RAC

brazil Oracle GI | HUB

Oracle RAC

germany Oracle GI | HUB

Oracle RAC

argentina Oracle GI | HUB

Oracle RAC

brazil Oracle GI | HUB

Oracle RAC

write

germany Oracle GI | HUB

Oracle RAC

argentina Oracle GI | HUB

Oracle RAC

brazil Oracle GI | HUB

Oracle RAC

write

germany Oracle GI | HUB

Oracle RAC

argentina Oracle GI | HUB

Oracle RAC

brazil Oracle GI | HUB

Oracle RAC

write

Page 16: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• DRM is used internally and externally.

• Externally means access patterns can influence the mastering of user data.

• Internally DRM is used for the management of UNDO data across instances for example.

• DRM-like activity is also performed upon instances leaving or joining the cluster, in course of which it is a.k.a. “recovery”

• Do not turn DRM off (_gc_policy_time=0)!

• Optimize _gc_policy_minimum to run / trigger DRM at a convenient moment under normal operations.

• Consider using smaller SGA sizes. • See MOS note 1619155.1 – “Best Practices and Recommendations for RAC databases using SGA larger than 300GB” as applicable

• “dbms_cacheutil” can be used to manually set and release affinity under well understood circumstances. Contact Oracle Support for details as needed.

17

Dynamic Re-Mastering (DRM) – Recommendations

germany

Oracle GI | HUB

Oracle RAC

argentina

Oracle GI | HUB

Oracle RAC

brazil

Oracle GI | HUB

Oracle RAC

write

Page 17: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 19

New in 12.1.0.2: Cache Fusion Accelerator

germany

Oracle GI

Oracle RAC

argentina

Oracle GI

Oracle RAC

brazil

Oracle GI

Oracle RAC

write

LMS process Accelerator

• The Cache Fusion Accelerator (CFA) is an

• OS kernel (Linux & Solaris only) module

• which can respond directly to certain lock requests via RDSv3.

• The lock state is saved in memory shared by the database and the kernel.

• CFA saves user/kernel context switches, frees up CPU cycles in LMS, and “speeds up” messages.

• CFA will be activated on Engineered Systems over time, including the Oracle Database Appliance.

• CFA contributes to a better, linear scaling.

• CFA is one of a long list of improvements.

Page 18: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

5

Cache Fusion Overview

(Dynamic Re-) Mastering

Handling Contentions

RAC Meets Multitenant

RAC Meets In-Memory

20

Page 19: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

write write

• Contention can occur in any multi-user system (even in SI databases)

• User data as well as “metadata” (e.g. an index) can be subject to contention

21

• You can still guide users to only one instance in an Oracle RAC environment.

• As soon as you scale out, contention can occur between instances (not only within an instance).

• From a contention perspective, a 2-node setup is basically the same as a 3 or more nodes setup.

• The difference is in the required message-based communication to obtain a block considering the mastering.

How To Handle Contention – Basics write

write write

write

write write

write write

write write

write write

Note: for scalability, only write/write contention needs to be considered.

Page 20: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• Frequent transactional changes to the same data blocks in all instances may result in “write hot spots”.

• Block with pending changes may be “pinged” by other instances.

22

• Pending redo must be written to log before the block can be transferred

• Latency for a deferred block transfer becomes dependent on delay for log IO

• Contention can affect related data as much as it can affect the actual “user data”.

• Right growing indexes and index contention are common.

• In 99% of OLTP performance issues, write hot spots occur on indexes.

How To Handle Contention – Considerations write

write write

write

write write

write write

Sequence

write write

write write

REDO

Page 21: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• Avoid frequent transactional changes to the same data blocks in all instances using partitioning and services.

• Logically partition data so that sub-data is handled in one instance only. Guide users via services accordingly.

23

• Place redo logs on fast storage if performance critical; e.g. SSDs

•Implemented in 11.2.2.4 of Exadata and Oracle Database Appliance by default (Smart Logs and SSDs, respectively)

•Separate disks for logs from other IO busy disks

• Use either for better cache locality:

• Global hash partitioned indexes

• Locally partitioned indexes

• Drop unused indexes

How To Handle Contention – General Solutions write

write write

write

Sequence

write write

write write

REDO

write write

write write

Page 22: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• UCP supports “Connection Affinity”:

• Transaction-Based Affinity

• Web Session Affinity

http://docs.oracle.com/database/121/JJUCP/rac.htm#JJUCP8197

24

• Connection Pools limit the number of connections to the database – example:

• Oracle Universal Connection Pool (UCP)

• Use “Database Resident Connection Pooling” when no. of active sessions much smaller then no. of open session.

• Fast Application Notification (FAN) – enabled connection pools receive Load Balancing information based on the Workload Repository and on a per-service basis.

How To Handle Contention – Use Connection Pools write

write write

write write

write Connection Pool Connection Pool Connection Pool

busy idle

Page 23: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

5

Cache Fusion Overview

(Dynamic Re-) Mastering

Handling Contentions

RAC Meets Multitenant

RAC Meets In-Memory

25

Page 24: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

•Pluggable Databases (PDBs) represent themselves as services in an Oracle RAC Multitenant Database, ensuring extremely fast failover.

• PDBs can be used to conveniently align users to instances with all the benefits, providing higher consolidation benefit and agility.

26

• Oracle RAC Multitenant Databases help to consolidate.

• Future improvements will ensure greater efficiency.

• Multithreaded Redo Log writer already implemented.

Oracle Multitenant and Oracle RAC – a Symbiosis

germany

Oracle GI | HUB

Oracle RAC

argentina

Oracle GI | HUB

Oracle RAC

cons_1 cons_2

REDO

Page 25: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

•Pluggable Databases (PDBs) represent themselves as services in an Oracle RAC Multitenant Database, ensuring extremely fast failover.

• PDBs can be used to conveniently align users to instances with all the benefits, providing higher consolidation benefit and agility.

27

Oracle Multitenant and Oracle RAC – a Symbiosis

germany

Oracle GI | HUB

Oracle RAC

argentina

Oracle GI | HUB

Oracle RAC

cons_1 cons_2

REDO

• Oracle RAC Multitenant Databases help to consolidate.

• Future improvements will ensure greater efficiency.

• Multithreaded Redo Log writer already implemented.

• Per-PDB/service optimized GCS operations coming in future.

More information, see: Oracle Multitenant meets Oracle RAC - http://www.slideshare.net/MarkusMichalewicz/oracle-

multitenant-meets-oracle-rac-ioug-2014-version

Page 26: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Program Agenda

1

2

3

4

5

Cache Fusion Overview

(Dynamic Re-) Mastering

Handling Contentions

RAC Meets Multitenant

RAC Meets In-Memory

28

Page 27: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 29

Oracle In-Memory and Oracle RAC

Breakthrough: Dual Format

Database

Generate Reports Instantly

Column Store

Replaces Analytic Indexes

In-Memory Column Store

Full HA & Integration

with Industry

Standards

Page 28: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 30

Oracle In-Memory and Oracle RAC – A Dream-Team

In-Memory Column Store

Addresses

Addresses

Breakthrough: Dual Format

Database

Column Store

Replaces Analytic Indexes

Page 29: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

“Full support for RAC scale-out means Oracle Database In-Memory can be used on our largest Data Warehouse, enabling more near real-time analytics.”

– Sudhi Vijayakumar, Senior Oracle DBA Yahoo Inc.

31

Page 30: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

Oracle Database In-Memory: Unique Fault Tolerance

• Similar to storage mirroring

• Duplicate in-memory columns on another node

• Enabled per table/partition

• Application transparent

• Downtime eliminated by using duplicate after failure

32

Only Available on Engineered Systems

Page 31: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

“Downtime is extremely costly for our business. Oracle’s In-Memory architecture takes the right approach to balancing real-time speed with continuous availability.”

– Jens-Christian Pokolm Analyst IT-DB Architecture & Engineering Postbank Systems AG

33

Page 32: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 35

• In-Memory Speed and Capacity of Low Cost Disk

• PCI Flash works mostly locally.

• Shared solutions require RAC external synchronization.

• Coming soon on ODA: a fully integrated shared flash cache.

Optimized Used of Memory in Every Layer

Normal Buffer Cache

In-Memory Format

Normal Buffer Cache

In-Memory Format

germany

Oracle GI | HUB

Oracle RAC

argentina

Oracle GI | HUB

Oracle RAC

brazil

Oracle GI | HUB

Oracle RAC

? ?

Coming soon…

• In-Memory compresses data, optimizing the use of memory.

• Memory allocated for In-Memory is subject to simplified lock mgmt.

• Compressed • Lock mgmt.

simplified

Page 33: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

• Database and Middleware streams available

• Videos

– Feature different groups and presenters

– Cover a broad range of topics and products

• FREE OF CHARGE Oracle RAC PM / Dev contributions:

– Oracle Flex Cluster: Optimized Resource Management for the Cloud - Ian Cookson

– Oracle Grid Infrastructure 12c Bundled Agents – Shankar Iyer

– ACFS Product Overview and Use Cases - Ara Shakian

– The Oracle Real Application Clusters (RAC) Family of Solutions - A User Guide – Markus Michalewicz

– Next Generation Oracle Automatic Storage Management - Jim Williams

– Implementing DBaaS with Oracle RAC 12c and Quality of Service Management - Mark Scardina

– Oracle RAC Practical Performance Management and Tuning – Markus Michalewicz

Oracle Learning Streams: http://education.oracle.com/streams/

Page 34: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 37

Page 35: Understanding Oracle RAC (12.1.0.2) Oracle RAC Internals - The Cache Fusion Edition Author Markus Michalewicz Subject Cache Fusion, its working and how to optimize its use. Keywords