Installing Oracle Database on LDOMSPhilippe Fierens

About Myself

• Freelance Oracle DBA • I am not a Solaris SysAdmin • Working with Oracle since 2001, Oracle iAS 1.0.2

• Currently working for EurID, • Registry for the .eu TLD • Creator of the fastest open source DNS server on the planet

My Belgian Dark Side:

Agenda

• Project overview : replace an M9000• HW choices• What are LDOMS• LDOM vs ZONES• How we choose to implement it• Problems and solutions

Replace an M9000

• Why• Power hog : cooling + power consumption• Maintenance cost• Not transportable (ok it has wheels but still 1300kg ;-)• Licensing

0,75 core factor ( Ultra Sparc VI ) VS

0,5 core factor ( T4,T5,M5,M6, …)

Our M9000 setupZones to limit number of CPU’s (license)

Requirements

• Biz decision : We stay on SPARC• M – class hardware needed? • Hot-pluggable features needed ?• We have RAC + Data Guard after all

• Can we live with less PCI slots ? • It needs to be “transportable”• Performance ?

Performance

• new T-series ?

• M values oracle provides you eg “T4-2 has a relative performance increase of 1.35 compared to the current M9000”

Is this throughput ? Response time ? what load ?

BUT I PREFER TO SEESEE

Comparing performance

• How to measure that ?• NO Real Application Test license • NO “paid” Load Generator

• What will_not_change • Storage • Network

SLOB to the rescue

• Kevin Clossons Silly Little Oracle Benchmark SLOB V1 and to measure the LIOPS

PERFORMS real world Oracle Database IO

See on

http://kevinclosson.wordpress.com/2012/02/06/introducing-slob-the-silly-little-oracle-benchmark/

What does slob do ?

FOR i IN 1..5000 LOOP        v_r := dbms_random.value(257, 10000) ;        SELECT COUNT(c2) into x FROM cf1 where custid >  v_r - 256 AND  custid < v_r;END LOOP;

SELECT COUNT(C2) FROM CF1 WHERE CUSTID > :B1 - 256 AND CUSTID < :B1

Plan hash value: 684126879

---------------------------------------------------------------------------------------| Id  | Operation                     | Name  | Rows  | Bytes | Cost (%CPU)| Time     |---------------------------------------------------------------------------------------|   0 | SELECT STATEMENT              |       |       |       |   259 (100)|          ||   1 |  SORT AGGREGATE               |       |     1 |   132 |            |          ||*  2 |   FILTER                      |       |       |       |            |          ||   3 |    TABLE ACCESS BY INDEX ROWID| CF1   |   256 | 33792 |   259   (0)| 00:00:04 ||*  4 |     INDEX RANGE SCAN          | I_CF1 |   256 |       |     2   (0)| 00:00:01 |---------------------------------------------------------------------------------------

WAIT #18446744071490302944: nam='db file sequential read' ela= 6144 file#=4 block#=651384 blocks=1 obj#=13604 tim=1275446655630

WAIT #18446744071490302944: nam='db file sequential read' ela= 2677 file#=4 block#=653529 blocks=1 obj#=13604 tim=1275446658528

Data Collection

• Collected and Compared SLOB stats • (send me mail if you want complete chart)

• Tested • T4-2 ( 16 cores 8 threads /core) • T5-2 when it came out (32 cores 8 threads / core)• M5000

Some Slob Results1 2 4 8 16 32 64 96 128

0

500

1000

1500

2000

2500

3000

3500

4000

M5K 8 core

T4-1

T4-2 8core ISM

T4-2 8core DISM

T5 8 cores

T5 16 cores

Number of Sessions

Tim

e in

Sec

onds

More Slob Results

SLOB LIOPS totalsessions M5000 8 core T4-8core T5-2-8core T5 16 cores

1 1033,057851 1157,407407 1506,024096 1519,756839

2 1992,031873 2207,505519 2816,901408 2793,296089

4 3868,471954 3861,003861 5347,593583 5319,148936

8 6611,570248 7561,436673 10256,41026 10282,77635

16 8403,361345 11283,49788 15533,98058 20000

32 8565,310493 19207,68307 25723,47267 29739,77695

64 8662,696264 26733,50042 35126,23491 49459,04173

96 32653,06122 61538,46154

128 32636,40999 66736,18352

• We did the tests on M5000 (approx same cpu as M9000)• SLOB LIOPS => (session# * 500.000) / elapsed time

AWR numbers from T5-2

Host Name Platform CPUs Cores Sockets Memory(GB)---------------- -------------------------------- ---- ----- ------- ----------t5play Solaris[tm] OE (64-bit) 128 16 1 512.00

Elapsed: 17.32 (mins) DB Time: 2,179.96 (mins)

Cache Sizes Begin End~~~~~~~~~~~ ---------- ----------

Buffer Cache: 61,440M 61,440M Std Block Size: 8K Shared Pool Size: 8,192M 8,192M Log Buffer: 217,416K

Load Profile Per Second Per Transaction Per Exec Per Call~~~~~~~~~~~~ --------------- --------------- ---------- ---------- DB Time(s): 125.8 11,890.7 0.00 47.81 DB CPU(s): 125.5 11,856.5 0.00 47.67

Redo size: 3,425.6 323,680.0

Logical reads: 15,852,361.5 1,497,873,782.5 Block changes: 8.4 788.8 Physical reads: 0.0 1.4 Physical writes: 1.4 130.6 User calls: 2.6 248.7 Parses: 2.3 213.8 Hard parses: 0.4 37.6W/A MB processed: 0.1 5.0 Logons: 0.1 13.0 Executes: 61,579.7 5,818,606.7 Rollbacks: 0.0 0.0

Transactions: 0.0Avg

wait % DBEvent Waits Time(s) (ms) time Wait Class------------------------------ ------------ ----------- ------ ------ ----------

DB CPU 130,422 99.7library cache: mutex X 144,867 114 1 .1 Concurrencresmgr:cpu quantum 44,990 71 2 .1 Schedulerlibrary cache lock 892 19 21 .0 Concurrenccursor: pin S wait on X 1,689 18 11 .0 Concurrenc

Host CPU (CPUs: 128 Cores: 16 Sockets: 1)

Result

Performance comparison conclusion

• T5-2 gives AMPLE processing power • Fraction of cost• Cap CPU possible ( ZONES or LDOM)

Questions so far?

Virtualizing in Solaris

• Zones aka containers

• LDOM aka OVM for SPARC

* Both technologies can be used to hard partition

Zones

• Global zone • “owns” the hardware and controls the resources

• Non Global zones• Configured and run in global zone • Exclusive access and Shared network access possible, • NO Exclusive access to HBA • Share same kernel as global zone (except branded zones)• Isolated

• Supported to CAP CPU (licensing)

LDOM

• Hypervisor • HW abstraction• presentation virtual devices• Integrated in the HW

• LDOM guests are separated machines, have direct access to CPU and Memory

• Requires CPU that supports this ( sun4v… T x, M5,M6 …)

• Very flexible

Flexibility

• Virtual switches• Virtual networks• Stub devices

• Virtual Disks• Since the IO is virtualized there is a penalty• See later

• Virtual terminal consoles

Roles

• Different types/roles• Control domain : • primary domain • used to connect to HV and manage resources

• I/O domain :• Direct access to phys I/O devices• PCI root complex • PCI card • Regular non virtualized drivers• Limited number of I/O domains depending on platform T5-2 vs

T5-4

Roles (continued)

• Service domain :Provides virtual I/O to guests

Virtual network

Virtual Disk

• Guest domainconsumer of virtual I/O

Requirements and Constraints

• Production isolation• Live migration some guests• Prod + ACC ==> RAC • 4 quad 1 gbit nics per machine • 4 dual 8Gbit HBA• No 10 gbit infrastructure (only “cross-over” cable for live

migration)

Chosen Architecture

• 1 control / IO / service domain :Not a lot of pci slots but we can have “unlimited” virtual nics *

• 1 root domain2 network port for PUBLIC interface

6 network port for PRIVATE interconnect

* Depends on LDC

Root complexes T5-2

Internal HDDs• /pci@300/pci@1/pci@0/pci@2/scsi@0/iport@[1-4]/disk@[0-3] (Controller 0; LSI SAS2308 #1)• /pci@3c0/pci@1/pci@0/pci@2/scsi@0/iport@[1&2]/disk@[4&5] (Controller 1; LSI SAS2308 #2)• CDROM - /pci@3c0/pci@1/pci@0/pci@2/scsi@0/iport@8/cdrom@p3• NET 0 - /pci@300/pci@1/pci@0/pci@1/network@0• NET 1 - /pci@300/pci@1/pci@0/pci@1/network@0,1• NET 2 - /pci@3c0/pci@1/pci@0/pci@1/network@0• NET 3 - /pci@3c0/pci@1/pci@0/pci@1/network@0,1• PCIe 1 - /pci@300/pci@1/pci@0/pci@4• PCIe 2 - /pci@380/pci@1/pci@0/pci@5• PCIe 3 - /pci@380/pci@1/pci@0/pci@6• PCIe 4 - /pci@380/pci@1/pci@0/pci@7• PCIe 5 - /pci@340/pci@1/pci@0/pci@4• PCIe 6 - /pci@340/pci@1/pci@0/pci@5• PCIe 7 - /pci@340/pci@1/pci@0/pci@6• PCIe 8 - /pci@3c0/pci@1/pci@0/pci@7• USB 0 (rear top)

• See note SPARC Platforms: Matrix of Recognized Device Paths (Doc ID 1005907.1)

Setup

Things which we ran into

• RAC and Jumbo Frames• How to provide network to guests• Integration with Symantec netbackup • Impact of Virtual IO on performance

RAC and Jumbo frames

• Make sure MTU size is big enough :• SOLARIS mtu = payload• Juniper switch mtu = payload + overhead

• When not matched, GI root.sh on second node fails ;-(

• Ping is your friend ( the NW admin as well) • ping -s -c 2 -i 192.168.10.1 host-priv1 9000

Network config service domain:

• Traditionally aggr on Service Domain IPMP in the guests.• DLMP• “Combines” IPMP and LACP• No configuration required on Switch • Multiple NIC are combined together• When one nic fails the guests using that NIC repointed to other NIC

DLMP how to

• On Service domain :• Create DLMP interface• Create VNIC to access the machine• Create vswitch on DLMP interface• Create NICs for guests on vswitch

Netbackup Constraints

• Guest no HBA access• service domain = media server• Master server talks to client• We don’t want REAL network traffic• We want to use FCAL to go to VTL

• Backup via stub device to media server

Netbackup Solution

• Create stub device vswitch• Vnic service domain• Vnic Guest

• Master communicates with public address of guest.• on service domain guest is guest.bck.domain (192.168.xx)• On master in /etc/host • Guest.bck.domain = 172.x.x.x

Performance Virtual io

• Tested with orion :• Results :• Random io always faster on Service / ROOT domain • Sequential IO

• Faster on guest then on service domain when >=1MB

• Slower for 8k block on guest

• See my blog for more details in the near futur ;)

SLOB P-IOPS1 2 4 8 12 16 32 48 64

0

5000

10000

15000

20000

25000

30000

35000

root domain

guest

readers

Phy

sica

l IO

per

Sec

ond

Small Recap

• T series can be alternative to M series• Logical Domains : Flexible• Performance penalty with Virtual IO• MEASURE it

Thank you

• Peter De Clerk ( Oracle Belgium)• Kevin Closson (father of SLOB :-) • Karoly Vegh ( Oracle Austria) via twitter• EurID for letting me present this• The three other pragma’s; Iloon, Joel and Jan• Thanks for reviewing @drune / @OsamaOracle

• You can contact me via

be.linkedin.com/in/pfierens/@PFIERENS philippe@fierensconsulting.eu

Pfierens.blogspot.com