Owner’s Guide: Installation

Oracle®

SuperCluster M6-32

Owner’s Guide: Installation

Part No.: E41536-05September 2015

Copyright © 2013, 2015, Oracle and/or its affiliates. All rights reserved.This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected byintellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate,broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering,disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to usin writing.If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then thefollowing notice is applicable:U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware,and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal AcquisitionRegulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, includingany operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and licenserestrictions applicable to the programs. No other rights are granted to the U.S. Government.This software or hardware is developed for general use in a variety of information management applications. It is not developed or intended for use in anyinherently dangerous applications, including applications that may create a risk of personal injury. If you use this software or hardware in dangerousapplications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure its safe use. OracleCorporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications.Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks orregistered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks ofAdvanced Micro Devices. UNIX is a registered trademark of The Open Group.This software or hardware and documentation may provide access to or information about content, products, and services from third parties. OracleCorporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, andservices unless otherwise set forth in an applicable agreement between you and Oracle. Oracle Corporation and its affiliates will not be responsible for anyloss, costs, or damages incurred due to your access to or use of third-party content, products, or services, except as set forth in an applicable agreementbetween you and Oracle.Documentation AccessibilityFor information about Oracle’s commitment to accessibility, visit the Oracle Accessibility Program website athttp://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.Access to Oracle SupportOracle customers that have purchased support have access to electronic support through My Oracle Support. For information, visithttp://www.oracle.com/pls/topic/lookup?ctx=acc&id=info or visit http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs if you are hearing impaired.

Copyright © 2013, 2015, Oracle et/ou ses affiliés. Tous droits réservés.Ce logiciel et la documentation qui l’accompagne sont protégés par les lois sur la propriété intellectuelle. Ils sont concédés sous licence et soumis à desrestrictions d’ utilisation et de divulgation. Sauf stipulation expresse de votre contrat de licence ou de la loi, vous ne pouvez pas copier, reproduire,traduire, diffuser, modifier, breveter, transmettre, distribuer, exposer, exécuter, publier ou afficher le logiciel, même partiellement, sous quelque forme etpar quelque procédé que ce soit. Par ailleurs, il est interdit de procéder à toute ingénierie inverse du logiciel, de le désassembler ou de le décompiler,excepté à des fins d’interopérabilité avec des logiciels tiers ou tel que prescrit par la loi.Les informations fournies dans ce document sont susceptibles de modification sans préavis. Par ailleurs, Oracle Corporation ne garantit pas qu’ellessoient exemptes d’erreurs et vous invite, le cas échéant, à lui en faire part par écrit.Si ce logiciel, ou la documentation qui l’accompagne, est concédé sous licence au Gouvernement des Etats-Unis, ou à toute entité qui délivre la licence dece logiciel ou l’utilise pour le compte du Gouvernement des Etats-Unis, la notice suivante s’applique:U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware,and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal AcquisitionRegulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, includingany operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and licenserestrictions applicable to the programs. No other rights are granted to the U.S. Government.Ce logiciel ou matériel a été développé pour un usage général dans le cadre d’applications de gestion des informations. Ce logiciel ou matériel n’est pasconçu ni n’est destiné à être utilisé dans des applications à risque, notamment dans des applications pouvant causer des dommages corporels. Si vousutilisez ce logiciel ou matériel dans le cadre d’applications dangereuses, il est de votre responsabilité de prendre toutes les mesures de secours, desauvegarde, de redondance et autres mesures nécessaires à son utilisation dans des conditions optimales de sécurité. Oracle Corporation et ses affiliésdéclinent toute responsabilité quant aux dommages causés par l’utilisation de ce logiciel ou matériel pour ce type d’applications.Oracle et Java sont des marques déposées d’Oracle Corporation et/ou de ses affiliés. Tout autre nom mentionné peut correspondre à des marquesappartenant à d’autres propriétaires qu’Oracle.Intel et Intel Xeon sont des marques ou des marques déposées d’Intel Corporation. Toutes les marques SPARC sont utilisées sous licence et sont desmarques ou des marques déposées de SPARC International, Inc. AMD, Opteron, le logo AMD et le logo AMD Opteron sont des marques ou des marquesdéposées d’Advanced Micro Devices. UNIX est une marque déposée d’The Open Group.Ce logiciel ou matériel et la documentation qui l’accompagne peuvent fournir des informations ou des liens donnant accès à des contenus, des produits etdes services émanant de tiers. Oracle Corporation et ses affiliés déclinent toute responsabilité ou garantie expresse quant aux contenus, produits ouservices émanant de tiers, sauf mention contraire stipulée dans un contrat entre vous et Oracle. En aucun cas, Oracle Corporation et ses affiliés nesauraient être tenus pour responsables des pertes subies, des coûts occasionnés ou des dommages causés par l’accès à des contenus, produits ou servicestiers, ou à leur utilisation, sauf mention contraire stipulée dans un contrat entre vous et Oracle.Accessibilité de la documentation

PleaseRecycle

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs


Pour plus d’informations sur l’engagement d’Oracle pour l’accessibilité à la documentation, visitez le site Web Oracle Accessibility Program, à l’adressehttp://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc.Accès au support électroniqueLes clients Oracle qui ont souscrit un contrat de support ont accès au support électronique via My Oracle Support. Pour plus d’informations, visitez le sitehttp://www.oracle.com/pls/topic/lookup?ctx=acc&id=info ou le site http://www.oracle.com/pls/topic/lookup?ctx=acc&id=trs si vous êtes malentendant.

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=docacc

http://www.oracle.com/pls/topic/lookup?ctx=acc&id=info



Contents

Using This Documentation viii

Installing SuperCluster M6-32 1

Hardware Installation Overview 1

Weight Specifications 3

Hardware Installation Task Overview 3

Hardware Installation Documents 8

Preparing the Site(Storage Rack and Expansion Racks) 11

▼ Prepare the Site for the Racks 12

Storage Rack Components 12

Physical Specifications 13

Reviewing Power Requirements 14

PDU Power Specifications 15

Facility Power Requirements 17

Circuit Breaker Requirements 17

Grounding Guidelines 18

PDU Thresholds 18

Preparing for Cooling 19

Heat Dissipation Specifications 20

Airflow Requirements 21

Perforated Floor Tiles 22

iv

Environmental Specifications 23

Preparing the Network 25

Network Topology 26

Network Infrastructure Requirements 27

Compute Server Default Host Names and IP Addresses 28

▼ Install Cable Drops 29

▼ Prepare DNS 30

Cabling SuperCluster M6-32 31

Compute Server Network Components 31

Storage Rack Network Components 34

▼ Cable the ZFS Storage Appliance 36

ZFS Appliance Power Cord Connection Reference 37

ZFS Storage Appliance Cabling Reference 38

▼ Cable IB Switches 39

Leaf Switch 1 Cabling Reference 41

Leaf Switch 2 Cabling Reference 43

IB Switch-to-Switch Cabling Reference 45

▼ Cable the Ethernet Management Switch 46

Ethernet Management Switch Cabling Reference 47

▼ Connect SuperCluster M6-32 to the Facility Networks 51

Connecting Expansion Racks 53

Expansion Rack Overview 54

Expansion Rack Components 55

Expansion Rack Layouts 56

▼ Installing Expansion Racks 59

One Expansion Rack Cabling 60

Two Expansion Racks Cabling 62

v Oracle SuperCluster M6-32 Owner’s Guide: Installation • September 2015

Three Expansion Racks Cabling 65

Four Expansion Racks Cabling 68

Five Expansion Racks Cabling 71

Six Expansion Racks Cabling 76

Seven Expansion Racks Cabling 81

Expansion Rack Default IP Addresses 87

Understanding Internal Cabling (Expansion Rack) 88

Oracle ILOM Cabling 89

Administrative Gigabit Ethernet Port Cabling 91

Single-Phase PDU Cabling 93

Three-Phase PDU Cabling (Full Rack) 95

IB Network Cabling 97

Glossary 103

Index 113

Contents vi

vii Oracle SuperCluster M6-32 Owner’s Guide: Installation • September 2015

Using This Documentation

■ Overview – Provides site planning specifications for the Oracle SuperClusterM6-32 storage rack, and describes how to install and cable SuperCluster M6-32components

Note – Because Oracle SuperCluster M6-32 is an engineered systems, comprisingvarious Oracle products that are documented, this document only providesinstallation information that is not described in other documents.

■ Audience – Technicians, system administrators, and authorized service providers

■ Required knowledge – Advanced experience installing computer hardware

These topics are covered in this section:

■ “Product Documentation Library” on page viii

■ “Feedback” on page ix

Product Documentation LibraryDocumentation for this product can be viewed through a browser by viewing thefollowing directory on the first compute server installed in SuperCluster M6-32:

/opt/oracle/node/doc/E41531_01

viii

FeedbackProvide feedback about this documentation at:

http://www.oracle.com/goto/docfeedback

ix Oracle SuperCluster M6-32 Owner’s Guide: Installation • September 2015

http://www.oracle.com/goto/docfeedback

Installing SuperCluster M6-32

These topics provide an overview and a list of resources for installing SuperClusterM6-32 hardware:

■ “Hardware Installation Overview” on page 1

■ “Weight Specifications” on page 3

■ “Hardware Installation Task Overview” on page 3

■ “Hardware Installation Documents” on page 8

Related Information

■ “Preparing the Site (Storage Rack and Expansion Racks)” on page 11

■ “Preparing the Network” on page 25

■ “Cabling SuperCluster M6-32” on page 31

■ “Connecting Expansion Racks” on page 53

Hardware Installation OverviewInstalling SuperCluster M6-32 hardware involves installing individual products(shown in this illustration), then cabling them together through network switches(see “Cabling SuperCluster M6-32” on page 31).

For more detailed configuration information, refer to the Oracle SuperCluster M6-32Owner’s Guide: Overview.

1

No. Major Component Description

1 SPARC M6-32 server (compute server) Provides compute, storage, and I/O resources in severalorderable configurations.

2 Oracle SuperCluster M6-32 storage rack(storage rack)

The storage rack includes 3 IB switches, 1 Ethernetmanagement switch, and once installed an Oracle ExadataStorage Servers X4-2, Oracle ZS3-ES Storage Appliance(ZFS storage appliance),Up to 17 additional Oracle Exadata Storage ExpansionRacks (expansion racks) can be added.

3 Optional second compute server If included, the configuration results in componentsspread across both chassis.

2 Oracle SuperCluster M6-32 Owner’s Guide: Installation • September 2015

Related Information




■ “Compute Server Network Components” on page 31

■ “Storage Rack Network Components” on page 34

Weight SpecificationsThis table lists the weight specifications for the major components. For additionalspecifications for the SPARC M6-32 server, refer to the SPARC M6-32 documentation(see “Hardware Installation Documents” on page 8). For expansion rackspecifications, see “Preparing the Site (Storage Rack and Expansion Racks)” onpage 11.

Related Information





Hardware Installation Task OverviewThis table provides a summary of the installation process that is performed by Oracleservice personnel.

Component Approximate Weight

SuperCluster M6-32 chassis 3700 lbs (1678 kg)

Expansion Rack 1335 lbs (606 kg)

Total weight (one SuperCluster M6-32 chassis) 5035 lb. (2284 kg)

Total weight (two SuperCluster M6-32 chassis) 8735 lbs (3962 kg)


Because Oracle SuperCluster M6-32 comprises various Oracle products that aredocumented, this document only provides installation information that is notdescribed in other documents.

Use this table in conjunction with the installation documents listed in “HardwareInstallation Documents” on page 8.

Steps Links

1. Prepare the site.

1. Ensure that the site meets all the requirements for each SuperClusterM6-32 major component. Specifications for these requirements arelisted in each product installation guide:- Physical space, receiving area, and access route- Facility power, circuit breakers, and receptacles- Cooling

2. Complete the customer site checklists and configuration worksheets.3. Review the network requirements.

• “Hardware InstallationDocuments” on page 8

• “Network InfrastructureRequirements” onpage 27

2. Prepare for installation.

1. Review the safety precautions, guidelines, site checklists, and siterequirements.

2. Gather the tools and equipment needed for installation.3. Ensure that power intended for SuperCluster M6-32 is disconnected at

the facility circuit breakers.4. Route network drops to the installation area.5. Prepare DNS for SuperCluster M6-32.


• “Install Cable Drops” onpage 29

• “Prepare DNS” onpage 30

3. Unpack and place SuperCluster racks in their intended locations.

1. Unpack the racks as described in the unpacking guides that areattached to the shipping cartons.

2. Place the racks in their allocated space. If two compute servers areincluded, place the storage rack between the two compute servers.Note – Due to cable length restrictions, the storage rack must beinstalled adjacent to the SPARC M6-32 servers. If you have two SPARCM6-32 servers, the storage rack must be installed between the twoSPARC M6-32 servers

3. If additional expansion racks are included (up to 17), position themadjacent to the other SuperCluster racks.

4. Stabilize and level all racks.


4. Apply power to the compute servers (minimum 1, maximum 2)


1. Connect each compute server to the facility power source, routing thepower cords either through the top or bottom of the cabinet,connecting and securing each cord to the rack.

2. Connect the SP serial cables to both SPs.3. Switch on the facility circuit breakers.4. Wait several minutes for the SPs to boot.5. Verify that server standby power is on and that there are no fault

indicators illuminated.6. Inspect and verify the hardware.Note - Do not connect SuperCluster M6-32 to any external networks atthis time.


5. Apply power to the storage and expansion racks.

1. Connect the storage rack to the facility power source, routing thepower cord either through the top or bottom of the cabinet.

2. Switch on the facility circuit breakers.3. Switch on the storage rack PDU breakers.4. Wait several minutes for all SPs to boot.5. Verify that standby power is on and that there are no fault indicators

illuminated.6. Either press the Power On buttons, or turn on the storage servers

through Oracle ILOM.7. Verify that power is applied to the Ethernet management switch.8. Verify that power is applied to the IB switches.9. Inspect and verify the storage and expansion rack storage hardware.


6. Apply power to any optional expansion racks.

1. Connect the expansion racks to the facility power source, routing thepower cord either through the top or bottom of the cabinet.

2. Switch on the facility circuit breakers.3. Switch on the expansion rack PDU breakers.4. Wait several minutes for POST device checks to finish and for all SPs

to boot.5. Verify that standby power is on and that there are no fault indicators

illuminated.6. Either press the Power On buttons, or turn on the storage servers

through Oracle ILOM.7. Verify that power is applied to the Ethernet management switch.8. Verify that power is applied to the IB switches.9. Inspect and verify the expansion rack storage hardware.10.Repeat these steps for additional optional expansion racks.


7. Install the ZFS storage appliance into the storage rack.


1. Unpack the ZFS storage appliance.2. Install the mounting brackets on both ZFS storage controllers.3. Install one set of the slide rails in the storage rack at U25 and the other

set at U26.4. Slide both ZFS storage controllers into the slide rails.5. Attach CMAs to the back of the slide rails.6. Install rails for the disk shelf (DE-24C) in the storage rack at U27.7. Install the disk shelf onto the rails and slide the enclosure into the

rack.8. Cable the controllers to the shelf and route the cables through the

CMA.9. Connect the power cords to the controllers and shelf. Route the cords

through the CMA and plug them into specific outlets in the storagerack PDUs.

10. Wait several minutes for devices to initialize and for all SPs to boot.11. Verify that server standby power is on and that there are no fault

indicators illuminated.12. If the ZFS storage controllers do not start up, press the switches

located on the front of the two ZFS storage controllers.13. Wait 3 to 5 minutes for the ZFS storage appliance to initiate services.14. Inspect and verify the ZFS storage appliance hardware.


• “Storage Rack NetworkComponents” on page 34

• “Cable the ZFS StorageAppliance” on page 36

8. (If included) Install 1 spine switch in each quarter and eighth populatedexpansion racks.Note - The spine switch is already factory installed in full and half expansionracks, so the subsequent steps only need to be performed if quarter and eighthracks are included.


1. Unpack the IB switch.2. Install the rails and cable management extenders in the expansion rack

at U1.3. Install the mounting brackets on the IB switch.4. Slide the IB switch into the slide rails.5. Connect the power cord to the IB switch then to a rack PDU outlet.6. Wait several minutes for the IB switch to initialize.7. Verify that power is on and that no fault indicators are illuminated.

9. Cable SuperCluster M6-32.


Related Information




The standard IB cables that ship with SuperCluster M6-32 must berouted over the top of the cabinets. The standard cables are not longenough to be routed down and under a raised floor.1. Cable the IB switch in U1.2. Cable the IB switch in U20.3. Cable the Ethernet management switch in U21.4. Cable the IB switch in U22.5. Route and dress the cables with velcro ties so that components can be

serviced.Note - Do not connect SuperCluster M6-32 to any external networks atthis time.

• “Cabling SuperClusterM6-32” on page 31

10. Authorized Oracle service personnel configure SuperCluster softwareaccording to site requirements.

Contact your Oraclesupport representative fordetails.

11. Connect SuperCluster M6-32 to external networks.

1. Ensure that the default factory installed IP addresses do not conflictwith the site IP addresses.

2. Connect the SuperCluster M6-32 Ethernet management switch to thefacility management network.

3. Connect the SuperCluster M6-32 10GbE EMS ports to the facilitynetwork.

• “Compute Server DefaultHost Names and IPAddresses” on page 28

• “Connect SuperClusterM6-32 to the FacilityNetworks” on page 51

12. Log in to SuperCluster M6-32.

There are a variety of methods you can use to log in to SuperClusterM6-32. The method you choose depends on the state of SuperClusterM6-32. Refer to the SPARC M5-32 and SPARC M6-32 Administration Guidefor details.



Hardware Installation DocumentsUse the documents in this table for detailed installation instructions. For a summaryof the installation process, see “Hardware Installation Task Overview” on page 3.

Information Document Location

Safety andcompliance

Important Safety Information forOracle’s Hardware Systems

Hard copy document shipped with SuperClusterM6-32

Oracle SuperCluster M6-32 Safety andCompliance Guide


Security SuperCluster System Security Guide /opt/oracle/node/doc/E41531_01

Late-breakinginformation

Oracle SuperCluster M6-32 ProductNotes


Site planningspecifications,installationpreparation,installationprocedures, andapplying power

Refer to base productdocumentation:

• Compute server –SPARC M5-32 and SPARC M6-32Servers Installation Guide

http://www.oracle.com/goto/M6-32/docs

• Storage and expansion racks –This document,Sun Rack II User’s Guide, and

Exadata Storage Server SoftwareUser’s Guide

“Preparing the Site (Storage Rack and ExpansionRacks)” on page 11

http://docs.oracle.com/cd/E19657-01

/opt/oracle/cell/doc

• ZFS storage appliance –ZFS Storage ZS3-ES ApplianceInstallation Guide

http://www.oracle.com/goto/ZS3-ES/docs

• IB switch –Sun Datacenter InfiniBand Switch 36HTML Collection for FirmwareVersion 2.1

http://docs.oracle.com/cd/E36265_01

AdditionalSuperCluster M6-32requirements

SuperCluster System Site Checklists,Oracle SuperCluster M6-32Configuration Worksheets




http://docs.oracle.com/cd/E19657-01

http://www.oracle.com/goto/ZS3-ES/docs

http://docs.oracle.com/cd/E36265_01

Related Information




Preparing thenetwork

This document “Preparing the Network” on page 25

Cabling the racks This document “Cabling SuperCluster M6-32” on page 31

Logging in SPARC M5-32 and SPARC M6-32Servers Administration Guide


Information Document Location



Preparing the Site(Storage Rack and Expansion Racks)

These topics describe how to prepare your site for the installation of the storage rack.The tasks and specifications also apply to the optional expansion racks.

Note – This chapter only provides site preparation information for the storage rackand expansion racks. For information on how to prepare the site for the computeserver, refer to the SPARC M5-32 and SPARC M6-32 Servers Installation Guide, asdescribed in “Hardware Installation Documents” on page 8.

■ “Prepare the Site for the Racks” on page 12

■ “Storage Rack Components” on page 12

■ “Physical Specifications” on page 13

■ “Reviewing Power Requirements” on page 14

■ “Preparing for Cooling” on page 19

Related Information

■ “Installing SuperCluster M6-32” on page 1




11

▼ Prepare the Site for the Racks1. Ensure that there is enough space to accommodate each rack.

One storage rack is included with SuperCluster M6-32, and 17 additional optionalexpansion racks can be added.

See “Storage Rack Components” on page 12 and “Physical Specifications” onpage 13.

2. Provide required power for each rack.

See “Reviewing Power Requirements” on page 14.

3. Provide proper cooling for each rack.

See “Preparing for Cooling” on page 19.

Related Information■ “Storage Rack Components” on page 12





Storage Rack ComponentsOne storage rack is included with SuperCluster M6-32 and contains the componentlisted in this table.

X4-2 Storage Racks X5-2 Storage Racks

9 Exadata Storage Servers X4-2 9 Exadata Storage Servers X5-2

Depending on the model, the storageservers provide these types of drives:• High-performance (HP) e 10,000 RPM

high-performance SAS drives• High-capacity (HC) 72,000 RPM

high-capacity SAS drives

Depending on the model, the storage serversprovide these types of storage devices:• 8 Extreme Flash (EF) drives• 12 High-capacity (HC) high-capacity SAS

drives and flash cards

3 36-port IB switches 3 36-port IB switches


You can add up to 17 additional optional expansion racks (in full-, half-, quarter-,and eighth-rack configurations) if additional storage is needed. To see a list ofexpansion rack components, see “Expansion Rack Components” on page 55.

Related Information





Physical Specifications

1 Ethernet management switch 1 Ethernet management switch

1 ZFS storage appliance 1 ZFS storage appliance

Redundant PDUs Redundant PDUs

TABLE: Storage and Expansion Rack Physical Specifications

Parameter English Metric

Height 78.66 in. 1998 mm

Width with side panels 23.62 in. 600 mm

Depth with front and rear doors 47.24 in. 1200 mm

Depth without doors 43.78 in. 1112 mm

Minimum space between top of cabinet and ceiling 36 in. 914 mm

Weight (full rack, with doors) 1824 lbs 827 kg

Weight (half rack, with doors) 1138 lbs 516 kg

Weight (quarter rack, with doors) 771 lbs 350 kg

X4-2 Storage Racks X5-2 Storage Racks

Preparing the Site (Storage Rack and Expansion Racks) 13

Related Information






Reviewing Power RequirementsThese topics list the storage and expansion rack power requirements.

■ “PDU Power Specifications” on page 15

■ “Facility Power Requirements” on page 17

■ “PDU Thresholds” on page 18

Related Information






TABLE: Rack Shipping Package Dimensions and Weights

Parameter English Metric

Height 85 in. 2159 mm

Width 48 in. 1219 mm

Depth 62 in. 1575 mm

Shipping weight (full rack) 2207 lbs 1001.1 kg

Shipping weight (half rack) 1454 lbs 659.6 kg

Shipping weight (quarter rack) 1048 lbs 475.3 kg


PDU Power SpecificationsWhen ordering SuperCluster M6-32, you must provide two specifications for thestorage and expansion racks:

■ Low or high voltage

■ Single- or three-phase power

Use one of these tables to identify the PDU power specifications for your type ofstorage and expansion rack.

TABLE: Low-Voltage, Single-Phase PDUs

Specification Requirements for each PDU

Voltage 200 to 240 VAC

Frequency 50/60 Hz

Current 24A maximum per input

Power rating 15 kVA

Output current 72A (3 inputs x 24A)

Outlets 42 x C13; 6 x C19

Outlet groups 6

Group protection(UL489 2-pole circuit breaker)

20A

Datacenter receptacle 15kVA, with three 30A/250V 2-pole/3-wire NEMAL6-30P plugs

TABLE: Low-Voltage, Three-Phase PDUs


Voltage 190 - 220 VAC 3ph

Frequency 50/60 Hz

Current 40A maximum per phase

Power rating 14.4 kVA

Output current 69.3A (3 x 23.1A)



Outlet groups 6

Group protection(UL489 2-pole circuitbreaker)

20A

Datacenter receptacle 15kVA IEC 60309 60A 4-pin 250VAC three phase IP67

TABLE: High-Voltage, Single-Phase PDUs

Specification Requirement for each PDU

Voltage 220-240 VAC

Frequency 50/60 Hz


Power rating 22 kVA

Output current 96A (3 x 32 A)


Outlet groups 6


20A

Datacenter receptacle 32A, with 3-pin 250 VAC IP44 IEC 60309 plugs

TABLE: High-Voltage, Three-Phase PDUs


Voltage 220/380 - 240/415 VAC 3 ph

Frequency 50/60 Hz


Power rating 14.4 kVA

Output current 62.7A (3 x 20.9 A)


TABLE: Low-Voltage, Three-Phase PDUs (Continued)



Related Information


■ “PDU Thresholds” on page 18

Facility Power RequirementsElectrical work and installations must comply with applicable local, state, or nationalelectrical codes. Contact your facilities manager or qualified electrician to determinewhat type of power is supplied to the building.

To prevent catastrophic failures, design the input power sources to ensure thatadequate power is provided to the PDUs.

In the United States and Canada, ensure that the overall system AC input currentload does not exceed 80 percent of the branch circuit AC current rating.

When planning for power distribution requirements, balance the power loadbetween available AC supply branch circuits.

The installation site AC power receptacle must be within 6.6 feet (2 m) of the rack.

Circuit Breaker RequirementsProvide dedicated AC breaker panels for all power circuits that supply power to thestorage and expansion racks.

In addition to circuit breakers, provide a stable power source, such as a UPS, toreduce the possibility of component failures. If computer equipment is subjected torepeated power interruptions and fluctuations, then it is susceptible to a higher rateof component failure.

Outlet groups 6


20A

Datacenter receptacle 15 KVA, three phase, 5-pin, IEC 60309 32A, 5-pin230/400V, three phase IP44

TABLE: High-Voltage, Three-Phase PDUs (Continued)



Grounding GuidelinesElectrical circuits must be grounded to the earth.

The storage and expansion racks are shipped with grounding-type power cords.Always connect the cords to grounded power outlets. Because different groundingmethods are used, depending on location, check the grounding type. Also refer todocumentation, such as IEC documents, for the correct grounding method. Ensurethat the facility administrator or qualified electrical engineer verifies the groundingmethod for the building, and performs the grounding work.

Related Information


PDU ThresholdsThis section provides the default PDU current thresholds for warnings and alarmsfor several SuperCluster M6-32 configurations.

You can also view the values by accessing the PDU metering unit as described in theSun Rack II Power Distribution Units User’s Guide. To access this guide, see “HardwareInstallation Documents” on page 8.

Use one of these tables based on the type of PDU in the racks.

TABLE: 15kVA Single-Phase PDUs

Low Voltage High Voltage

PDU A PDU B Warning (Amps) Alarm (Amps) Warning (Amps) Alarm (Amps)

M1-3 M1-1 10 13 9 12

M1-2 M1-2 8 11 7 10

M1-1 M1-3 20 24 18 23


Related Information



Preparing for CoolingThese topics describe how to prepare the site with adequate cooling.

■ “Heat Dissipation Specifications” on page 20

■ “Airflow Requirements” on page 21

■ “Perforated Floor Tiles” on page 22

■ “Environmental Specifications” on page 23

Related Information






TABLE: 15kVA 3-Phase PDUs

Low Voltage High Voltage

PDU A PDU B Warning (Amps) Alarm (Amps) Warning (Amps) Alarm (Amps)

M1-3 M1-3 22 28 9 12

M1-2 M1-2 24 31 15 19

M1-1 M1-1 19 24 11 14


Heat Dissipation Specifications

Related Information




TABLE: Racks with Exadata X5-2 Storage Servers

Configuration BTU/Hour kj/Hour

EF Quarter rack Maximum 12,362 13,042

Typical 8,654 9,129

HC Quarter rack Maximum 11,516 12,149

Typical 8,061 8,505

Individual EF storageserver

Maximum 2,037 2,149

Typical 1,426 1,504

Individual HC storageserver

Maximum 1,825 1,926

Typical 1,278 1,348

TABLE: Racks with Exadata X4-2 Storage Servers

Storage Type Full Rack Half Rack Quarter Rack

Highperformance

Maximum 34,100 BTU/hour(36,000 kJ/hour)

19,100 BTU/hour(20,150 kJ/hour)


Typical 23,900 BTU/hour(25,200 kJ/hour)



High capacity Maximum 30,400 BTU/hour(32,100 kJ/hour)



Typical 21,200 BTU/hour(22,400 kJ/hour)




Airflow RequirementsIn order to cool SuperCluster M6-32 properly, ensure that adequate airflow travelsthrough the cabinets.

Caution – Do not restrict the movement of cool air from the air conditioner to thecabinet, or the movement of hot air out of the rear of the cabinet.

Observe these requirements:

■ Allow a minimum clearance of 36 inches (914 mm) at the front of the rack, and 36inches (914 mm) at the rear of the rack for ventilation. There is no airflowrequirement for the left and right sides, or the top of the rack.

■ If the rack is not completely filled with components, cover the empty sections withfiller panels.

FIGURE: Direction of Airflow Is Front to Back


Related Information




Perforated Floor TilesIf you install SuperCluster M6-32 on a raised floor, use perforated tiles in front of therack to supply cold air to the system. Each tile should support an airflow ofapproximately 400 CFM.

Perforated tiles can be arranged in any order in front of the rack, as long as cold airfrom the tiles can flow into the rack.

This table lists the recommended number of floor tiles.

Related Information




TABLE: Airflow (Quantities Are Approximate)

Rack Maximum (CFM) Typical (CFM)

Full rack 1,600 1,100

Half rack 900 650

Quarter rack 440 310

Rack Number of Tiles

Full rack 4

Half rack 3

Quarter rack 2


Environmental Specifications

Related Information




Condition Operating Requirement Nonoperating Requirement Comments

Temperature 5 to 32°C (41 to 89.6°F) -40 to 70°C (-40 to 158°F). For optimal rack cooling, datacenter temperatures from 21 to23°C (70 to 74°F).

Relative humidity 10 to 90% relativehumidity, noncondensing

Up to 93% relativehumidity.

For optimal data center rackcooling, 45 to 50%,noncondensing.

Altitude 3048 m (10000 ft.)maximum

12000 m (40000 ft.). Ambient temperature is reducedby 1 degree Celsius per 300 mabove 900 m altitude above sealevel.


Preparing the Network

These topics describe how to prepare the network.

■ “Network Topology” on page 26

■ “Network Infrastructure Requirements” on page 27

■ “Compute Server Default Host Names and IP Addresses” on page 28

■ “Install Cable Drops” on page 29

■ “Prepare DNS” on page 30

Related Information





25

Network Topology

Each network must be on a distinct and separate subnet from the others.

■ Client access network – Connects SuperCluster M6-32 to your existing clientnetwork and provides client access to SuperCluster M6-32. Database applicationsaccess the database through this network using SCAN and Oracle RAC VIPaddresses.

■ Management network – Connects your existing management network toSuperCluster M6-32 and is used for administrative access. This network consists ofthe servers, SPs, and IB switches connected through the Ethernet managementswitch. There is one uplink from the Ethernet management switch in the rack toyour existing management network.

Each storage server provides two network interfaces for management. The 1GbEhost management interface provides management access to the OS, and the NETMGT port provides access to Oracle ILOM.

■ Serial access to Oracle ILOM – Connects the compute server SP SER MGT portsto your terminal server (or equivalent) for direct access to the Oracle ILOM CLI.

■ Private InfiniBand network – Connects the storage servers and ZFS storagecontrollers to the compute servers through the IB switches. This nonroutingnetwork is internal to SuperCluster M6-32 and does not connect to your existingnetwork.


Related Information

■ Oracle SuperCluster M6-32 Owner’s Guide: Overview





Network Infrastructure RequirementsThe installation site must provide these network infrastructure items:

■ Client access network

■ 10GbE switch for connectivity from the compute server EMS ports to the clientaccess network

■ Cat6A (or better) cables to connect SuperCluster to the 10GbE switch. See “InstallCable Drops” on page 29.

Note – At the time of installation, if you do not have a 10GbE client access networkinfrastructure set up at the site, you must provide a 10GbE network switch thatSuperCluster M6-32 can connect to, even if the network speed drops from 10Gb to1Gb on the other side of the switch.

■ Adequate space to install the all of the cabinets adjacent to each other.

Due to cable length restrictions, the storage rack must be installed adjacent to theSPARC M6-32 servers. If you have two SPARC M6-32 servers, the storage rackmust be installed between the two SPARC M6-32 servers.

■ Adequate space above the cabinets to accommodate cabinet to cabinet cabling.

The standard cables that ship with SuperCluster M6-32 must be routed over thetop of the cabinets. The standard cables are not long enough to be routed downand under a raised floor.

Note – If your installation site cannot accommodate these restrictions, you can orderlonger IB cables (10m or longer) and replace the Ethernet cables with industrystandard cables in appropriate lengths. See “Cabling SuperCluster M6-32” onpage 31. Contact your sales representative for ordering details.


Related Information

■ Configuration worksheets





Compute Server Default Host Namesand IP AddressesThis table lists the SuperCluster M6-32 default host names and IP addresses. Thenames and addresses might be different if they were changed to meet siterequirements. Refer to your configuration worksheets for details.

Related Information

■ Configuration worksheets





Interface Host Name IP Address Gateway Address Netmask

Host GBE0 (IOU 0) ssccn1 192.168.1.12 192.168.1.254 255.255.255.0

SPs (static) 192.168.1.254 255.255.255.0

/SP/network/Active_SP sscch1-sp 192.168.1.121

/SP/network/SP0 sscch1-sp0 192.168.1.124

/SP/network/SP1 sscch1-sp1 192.168.1.125

/SP/network/HOST0 sscch1-spp0 192.168.1.126





▼ Install Cable Drops● Run cables from the facility network infrastructure to the installation site.

This table lists the minimum number of cable drops that you need.

Note – The number of IP addresses that you need for each network variesdepending on the type of configuration you choose. For more information on thenumber of IP addresses required for your configuration, refer to the appropriateconfiguration worksheet.

Related Information■ Configuration worksheets

■ “Connect SuperCluster M6-32 to the Facility Networks” on page 51





Network Cable Type (1]GbE or 10GbE) Minimum Number of Drops

Client access Cat6A or better 2 for each logical domain thatconnects to the computeserver’s EMS 10GbE ports

Ethernet Management Cat6A or better 1 that connects to theEthernet management switchin the storage rack


▼ Prepare DNSYou must prepare DNS before installing SuperCluster M6-32. Installation and theinitial configuration cannot proceed until these tasks are finished.

Note – DNS is not configured on SuperCluster M6-32 until after initial configuration.

1. Provide the necessary information in these documents:

■ SuperCluster System Site Checklist

■ Oracle SuperCluster M6-32 Configuration Worksheet

2. Use the host names and IP addresses specified in the completed configurationworksheets document to create and register DNS addresses for SuperClusterM6-32.

You must register all public addresses, SCAN addresses, and VIP addresses inDNS prior to installation.

Note – The configuration worksheets document defines the SCAN as a single namewith three IP addresses on the client access network.

3. Configure all addresses registered in DNS for both forward resolution andreverse resolution.

Reverse resolution must be forward confirmed (forward-confirmed reverse DNS),such that both the forward and reverse DNS entries match each other.

The SCAN name to the three SCAN addresses must be configured in DNS forround-trip resolution.

Related Information■ “Hardware Installation Documents” on page 8

■ Oracle Grid Infrastructure Installation Guide for Linux for additional informationabout SCAN addresses

■ Your DNS vendor documentation for additional information about configuringround-trip name resolution






Cabling SuperCluster M6-32

These topics describe how to cable SuperCluster M6-32.

Related Information





Compute Server Network ComponentsUse this illustration and table to identify the locations of the network components inthe compute server.

Description Links

Identify where the network componentsare located.

“Compute Server Network Components” on page 31“Storage Rack Network Components” on page 34

Cable the ZFS storage appliance. “Cable the ZFS Storage Appliance” on page 36“ZFS Storage Appliance Cabling Reference” on page 38

Cable the IB switches. “Cable IB Switches” on page 39“Leaf Switch 1 Cabling Reference” on page 41“Leaf Switch 2 Cabling Reference” on page 43“IB Switch-to-Switch Cabling Reference” on page 45

Cable the network components. “Cable the Ethernet Management Switch” on page 46“Ethernet Management Switch Cabling Reference” on page 47

Connect SuperCluster M6-32 to thefacility networks.

“Connect SuperCluster M6-32 to the Facility Networks” on page 51

31

No. Network Component or Port Chassis Location Connector and Cable Type Links

1 2 SPs, each providing:• 1 10/100/10000BASE-T

port (NET MGT)• 1 serial port (SER MGT)

Front lower bay, SP 0(left) and SP 1 (right)

NET MGT – RJ-45, useCat.5 or better cables.SER MGT – RJ-45, useserial cables.

“Cable the EthernetManagement Switch” onpage 46


Related Information


■ “Cable the Ethernet Management Switch” on page 46

■ “Cable IB Switches” on page 39


2 8 or 16* IB HCA PCIe cards(4 cards per IOU, 2 ports percard)

Installed in these slots inall functional IOUs:• PCIe 2• PCIe 6• PCIe 10• PCIe 14

16 to 32standard IBconnectors supportingcables with QSFPconnectors.

“Cable IB Switches” onpage 39

3 EMSs (4 per functionalIOU), each providing 2100/1000/100000BASE-Tports

Installed in these slots inall functional IOUs:• EMS 1• EMS 2• EMS 3• EMS 4

RJ-45, use Cat6A orbetter cables.

“Connect SuperClusterM6-32 to the FacilityNetworks” on page 51

4 2 or 4* Sun Quad Port GbEPCIe 2.0 Low ProfileAdapters, UTP (1 per IOU)

Installed in these slots inall functional IOUs:• PCIe 3

RJ-45, use Cat.5 or bettercables.


* Quantity is based on the number of IOUs ordered.

No. Network Component or Port Chassis Location Connector and Cable Type Links


Storage Rack Network ComponentsUse this illustration and table to identify the locations of the network components inthe storage rack.


Related Information


■ “Cable the ZFS Storage Appliance” on page 36




No. Network ComponentRackLocation

Connector and CableType Links

1 1 ZFS storage appliance with 1 disk shelf and2 ZFS storage controllers, each controllerproviding:• 4 1GbE/10GbE ports (NET0, NET1, NET2,

NET3)• 2 IB ports• 1 SP 10/100BASE-T port (NET MGT) for

Oracle ILOM connectivity• 1 SP serial port (SER MGT)

U25 - U30 For 10GbE, useCAT6A or better.For other RJ-45connections, useCAT5 or bettercables.For IB ports, usestandard IB cableswith QSFPconnectors.

“Cable the ZFS StorageAppliance” on page 36

2 3 IB switches U1 (spine)U20 (leaf 1)U22 (leaf 2)

Use standard IBcables with QSFPconnectors.


3 1 48-port 10/100/1000 Ethernet managementswitch

U21 For RJ-45, use Cat5or better cables.


4 9 storage servers, each providing:• 4 1GbE/10GbE ports (NET0, NET1, NET2,

NET3)• 1 dual-port 4X QDR (40 Gb/s) IB HCA• 1 SP Ethernet port for Oracle ILOM

connectivity

U2U4U6U8U10U12U14U16U18

For 10GbE, useCAT6A or better.For other RJ-45connections, useCAT5 or bettercables.For IB ports, usestandard IB cableswith QSFPconnectors.



▼ Cable the ZFS Storage ApplianceAfter installing the ZFS storage controllers, disk shelf, and CMA, follow these stepsto cable the ZFS storage appliance components to each other.

1. Label both ends of the ZFS storage appliance cables.

2. Connect the cables to the ZFS storage appliance components as specified.

For specifications, see “ZFS Storage Appliance Cabling Reference” on page 38.

3. Connect the power cords to the ZFS storage appliance components.

4. Route the cables and power cords through the ZFS storage appliance CMA.

Refer to the ZFS Storage ZS3-ES Appliance Installation Guide. See “HardwareInstallation Documents” on page 8.

5. Connect the ZFS storage appliance power cords to the storage rack PDUs.

See “ZFS Appliance Power Cord Connection Reference” on page 37.

6. Verify that server standby power is on, and that there are no fault indicatorsilluminated.

7. If the ZFS storage controllers do not start up, press the switches located on thefront of the two ZFS storage controllers.

8. Wait several minutes for the ZFS storage appliance to initiate services.

9. Inspect and verify the ZFS storage appliance hardware.

If you detect any problems, correct them before continuing with the installation.Refer to the ZFS Storage ZS3-ES Appliance Service Manual. See “HardwareInstallation Documents” on page 8.

10. Go to the next installation task.

See “Cable the Ethernet Management Switch” on page 46.

Related Information■ “ZFS Storage Appliance Cabling Reference” on page 38





ZFS Appliance Power Cord ConnectionReferenceUse one of these tables according to the storage rack power type.

TABLE: 3-Phase Storage Racks

Location ComponentFromPS No.

ToPDU Receptacle Cord Length

27 Disk shelf PCM2 PDU A, Input 0 - Group 5 2 m

PCM1 PDU B, Input 6 - Group 2 2 m

26 ZFS storagecontroller

PS 0 PDU A, Input 2 - Group 4 2 m

PS 1 PDU B, Input 1 - Group 1 2 m




TABLE: Single-Phase Storage Racks

Location ComponentFromPS No.

ToPDU Receptacle Cord Length

27 Disk shelf PCM2 PDU A, Input 0 - Group 5 2 m

PCM1 PDU B, Input 0 - Group 0 2 m








ZFS Storage Appliance CablingReferenceUse this illustration and table to connect the ZFS storage controllers and disk shelf.

Note – For cable connections from the ZFS storage appliance to the Ethernetmanagement switch, see “Ethernet Management Switch Cabling Reference” onpage 47. For IB switch connections, see “Leaf Switch 1 Cabling Reference” on page 41and “Leaf Switch 2 Cabling Reference” on page 43.


Related Information

■ “Cable the ZFS Storage Appliance” on page 36



■ “Ethernet Management Switch Cabling Reference” on page 47

■ “Leaf Switch 1 Cabling Reference” on page 41


▼ Cable IB SwitchesCabling instructions assume that the racks are adjacent to each other. If they are not,longer cables are required for the rack-to-rack connections. See “NetworkInfrastructure Requirements” on page 27.

When cabling the IB Switches, start with the lower switches and work up. Start withthe lowest leaf switch, then cable the Ethernet management switch, then the top leafswitch.

From To

ComponentComponentLocation Port Component

ComponentLocation Port Cable

ZFS storagecontroller

U25 PCIe 2, P0 Storagecontroller

U26 PCIe 2, P2 Green


U26 PCIe 2, P1 Black


U26 PCIe 2, P0 Yellow

U25 PCIe 3, P1 Disk shelf U27 IOM 0, P0 Black


ZFS storagecontroller




Note – The cables from the spine switch in U1 to the two leaf switches areconfigured at the factory as listed in “IB Switch-to-Switch Cabling Reference” onpage 45.

1. Gather and label both ends of the supplied IB cables.

2. In groups of four, connect the IB cables to the compute servers and route themto their destination in the storage rack.

See “Leaf Switch 1 Cabling Reference” on page 41 and “Leaf Switch 2 CablingReference” on page 43.

For cables that go from one cabinet to another, you can route the cables throughthe top or bottom of the cabinets. Routing options are described in the SPARCM5-32 and SPARC M6-32 Installation Guide. See “Hardware InstallationDocuments” on page 8.

3. For cables that connect to the ZFS storage appliance, route the cables into theZFS storage appliance CMA.

Route the cables in a bundle into the CMA to ensure that all the cables fit in theCMA.


4. Use velcro cable ties to secure cable bundles so that components can beserviced.

Locate excess cable length either on top of the cabinets or under the raised floor.

5. Consider your next installation task.

■ See “Cable the Ethernet Management Switch” on page 46.

■ See “Connect SuperCluster M6-32 to the Facility Networks” on page 51.

Related Information■ “Leaf Switch 1 Cabling Reference” on page 41


■ “IB Switch-to-Switch Cabling Reference” on page 45





Leaf Switch 1 Cabling ReferenceLeaf switch 1 is the IB switch installed in U20.

When the table lists Compute server 0 or 1, choose the server that has an IB HCA PCIecard in the specified IOU.

When the table lists IOU n or n, choose the IOU that has an IB HCA PCIe card.

From To

X =FactoryInstalled

U20Leaf Switch 1Port Component Location Port Cable

0A Compute server 0 IOU 0 PCIe 2, P1 5m

0B Compute server 0 IOU 0 PCIe 6, P1 5m







4A Compute server 0 or 1 IOU 2 or 0 PCIe 2, P2 5m

4B Compute server 0 or 1 IOU 2 or 0 PCIe 6, P2 5m



Related Information








X 8A IB leaf switch 2 U22 8A 2m

X 8B IB spine switch U1 1B 3m

X 9A IB leaf switch 2 U22 9B 2m

X 9B IB leaf switch 2 U22 9A 2m





12A Unused

12B ZFS storage controller 1 U25 PCIe 0, P1 2m

X 13A Storage server 9 U18 PCIe 3, P1 3m



X 14B Storage server 7 U14 PCIe 3, P1 3m







From To

X =FactoryInstalled





Leaf Switch 2 Cabling ReferenceLeaf switch 2 is the IB switch installed in U22.

When the table lists Compute server 0 or 1, choose the server that has an IB HCA PCIecard in the specified IOU.

When the table lists IOU n or n, choose the IOU that has an IB HCA PCIe card.

From To

X =FactoryInstalled



















X 8A IB leaf switch 1 U20 8A 2m

X 8B IB spine switch U1 0B 3m







12A Unused












From To

X =FactoryInstalled



Related Information





IB Switch-to-Switch Cabling ReferenceThe IB switches are cabled together at the factory as shown in this figure and table.

X = FactoryInstalled

From To

Component Location Port Component Location Port

X Spine U1 0B Leaf switch 2 U22 8B

X Spine U1 1B Leaf switch 1 U20 8B

X Leaf switch 1 U20 8A Leaf switch 2 U22 8A

X Leaf switch 1 U20 9A Leaf switch 2 U22 9B

X Leaf switch 1 U20 9B Leaf switch 2 U22 9A



Related Information





▼ Cable the Ethernet ManagementSwitchCabling instructions assume that the racks are adjacent to each other. If they are not,longer cables are required for the rack-to-rack connections. See “NetworkInfrastructure Requirements” on page 27.

1. Label both ends of all of the Ethernet cables.

2. Connect the Ethernet cables to the Ethernet management switch and to theEthernet ports on SuperCluster M6-32 components as specified.

For specifications, see “Ethernet Management Switch Cabling Reference” onpage 47.

For cables that go from one cabinet to another, you can route the cables throughthe top or bottom of the cabinets. Routing options are described in the SPARCM5-32 and SPARC M6-32 Installation Guide. See “Hardware InstallationDocuments” on page 8.

3. For cables that connect to the ZFS storage appliance, route the cables into theZFS storage appliance CMA.

Route the cables in a bundle into the CMA to ensure that all the cables fit in theCMA.





X = FactoryInstalled

From To

Component Location Port Component Location Port




5. Consider your next installation task.

■ See “Cable IB Switches” on page 39.

■ See “Connect SuperCluster M6-32 to the Facility Networks” on page 51.

Related Information■ “Ethernet Management Switch Cabling Reference” on page 47




Ethernet Management Switch CablingReferenceThe Ethernet management switch is located in the storage rack in U21.

When the table lists Compute server 0 or 1, choose the server that has a GbE NIC in thespecified IOU.

No. Description No. Description

1 Indicators and reset switch 4 Ports 33-48, 10/100/1000BASE-TEthernet

2 Ports 1 to 16, 10/100/1000BASE-TEthernet

5 CON (upper), MGT (lower)

3 Ports 17 to 32, 10/100/1000BASE-TEthernet

6 Ports 45-48, 10GbE


From To

X =FactoryInstalled

EthernetManagementSwitch (U21)Port No. Cabinet Component Location Port Cable

1 Compute server 0, IOU 2orCompute server 1, IOU 0

GbE PCIe 3 0 20 ft blue



3 Compute server 0, IOU 1 GbE PCIe 3 3 20 ft blue













12 Compute server 1 SP 1 NET MGT 20 ft red




X 16 Storage rack PDU A NET MGT 1m gray




19 Storage rack ZFS storagecontroller 2

U26 NET 1 10 ft black

X 20 Storage rack PDU B NET MGT 1m gray








X 25 Storage rack Storageserver



U18 NET MGT 10 ft red



















From To

X =FactoryInstalled



Related Information






















U2 Net MGT 10 ft red

X 45 Storage rack IB leaf switch2

U22 NET MGT0

10 ft black

X 46 Storage rack IB leaf switch1

U20 NET MGT0

10 ft black

X 47 Storage rack IB spineswitch

U1 NET MGT0

10 ft black

48 Client access LANNote - Do not connectuntil you ensure that thereare no IP address conflicts.

10” blue

From To

X =FactoryInstalled




▼ Connect SuperCluster M6-32 to theFacility Networks

Caution – Authorized service personnel must perform SuperCluster softwareconfiguration tasks before SuperCluster M6-32 is connected to facility networks.

1. Before connecting SuperCluster M6-32 to any external networks, verify that thefactory-provided IP addresses do not conflict with the site network addresses.

Consult the network administrator and see “Compute Server Default Host Namesand IP Addresses” on page 28.

2. Ensure that the facility network is prepared to connect to SuperCluster M6-32.

See “Preparing the Network” on page 25.

3. Label the cables used to connect to the facility Ethernet networks.

See “Install Cable Drops” on page 29.

4. Route and connect one facility management network cable to the Ethernetmanagement switch port 24.

The Ethernet management switch is in the storage rack in U21. See “Storage RackNetwork Components” on page 34.

5. Route and connect all client access network cables to the compute server EMSEthernet ports.

See “Compute Server Network Components” on page 31.



7. Log into SuperCluster M6-32 and verify network connectivity to the site.

Refer to the SPARC M5-32 and SPARC M6-32 Administration Guide. See“Hardware Installation Documents” on page 8.

Related Information■ “Network Infrastructure Requirements” on page 27



Connecting Expansion Racks

These topics describe how to add optional expansion racks to Oracle SuperClusterM6-32.

Related Information





Description Links

Learn about adding expansion racks. “Expansion Rack Overview” on page 54

Identify full-, half-, and quarter-expansionrack components.

“Expansion Rack Components” on page 55

View the location of expansion rackcomponents.

“Expansion Rack Layouts” on page 56

Install expansion racks. “Installing Expansion Racks” on page 59

Review the cabling information fordifferent quantities of expansion racks.

“One Expansion Rack Cabling” on page 60“Two Expansion Racks Cabling” on page 62“Three Expansion Racks Cabling” on page 65“Four Expansion Racks Cabling” on page 68“Five Expansion Racks Cabling” on page 71“Six Expansion Racks Cabling” on page 76“Seven Expansion Racks Cabling” on page 81

Review expansion rack default IPaddresses.

“Expansion Rack Default IP Addresses” onpage 87

Review expansion rack internal cableconnections.

“Understanding Internal Cabling (ExpansionRack)” on page 88

53

Expansion Rack OverviewExpansion racks provide additional storage for SuperCluster M6-32. The additionalstorage can be used for backups, historical data, and unstructured data.

Expansion racks are available as a full rack, half rack, or quarter rack. See “ExpansionRack Components” on page 55.

When you add expansion racks to SuperCluster M6-32, you must disconnect some ofthe factory installed IB cables and then interconnect the IB switches from rack torack. See “Installing Expansion Racks” on page 59.

Then you connect each leaf switch to the spine switches in all racks.

This figure shows how the IP switches are connected across two racks.

Each leaf switch in rack 1 connects to the following switches:

■ Four connections to its internal spine switch

■ Four connections to the spine switch in rack 2

The spine switch in rack 1 connects to the following switches:

■ Eight connections to both internal leaf switches

■ Eight connections to both leaf switches in rack 2


Related Information


■ “Expansion Rack Components” on page 55

■ “Installing Expansion Racks” on page 59

■ “Expansion Rack Default IP Addresses” on page 87

■ “Understanding Internal Cabling (Expansion Rack)” on page 88

Expansion Rack Components

TABLE: X5-2 Expansion Racks

Rack

• 4 - 19 Exadata Storage Serverswith: Extreme Flash drives orhigh-capacity SAS disks*

• 2 or 3 Sun Datacenter InfiniBandSwitch 36 switches†

• High speed flash• 2 redundant 15 kVA PDUs

(single-phase or three-phase,high-voltage or low-voltage)

• 1 48-port Cisco Catalyst 4948Ethernet switch

* You can increase the number of storageservers up to 19

† When connecting a quarter rack to Su-perCluster M6-32 you must install athird IB switch into the expansion rack.


Related Information


■ “Expansion Rack Overview” on page 54




Expansion Rack LayoutsUse this illustration and table to identify the locations of the main components in theexpansion rack.

This illustration shows a fully populated expansion rack. Expansion racks mighthave fewer components.

TABLE: X4-2 Expansion Racks

Full Rack Half Rack Quarter Rack

• 18 Exadata Storage Servers with:- High-performance SAS disks orhigh-capacity SAS disks

• 3 Sun Datacenter InfiniBandSwitch 36 switches

• High speed flash• 2 redundant 15 kVA PDU




• 3 Sun Datacenter InfiniBandSwitch 36 switches





• 2 Sun Datacenter InfiniBandSwitch 36 switches*




* When connecting a quarter rack to SuperCluster M6-32 you must install a third IB switch into the expansion rack.


Related Information







No. Network ComponentRackLocation

Connector and CableType Links

1 (Optional) 1 to 15 additional storageservers with these network components:• 4 1GbE/10GbE ports (NET 0, NET 1,

NET 2, NET 3)• 1 dual-port 4X QDR (40 Gb/s) IB

HCA• 1 SP Ethernet port for Oracle ILOM

connectivity

U10 - U19U23 - U42

10GbE usesCAT6A or better.Other RJ-45connections canuse CAT5 or bettercables.IB ports usestandard IB cableswith QSFPconnectors.

2 3 IB switches U1 (spine)U20 (leaf 1)U22 (leaf 2)

Standard IB cableswith QSFPconnectors.

“Installing ExpansionRacks” on page 59

3 1 48-port 10/100/1000 Ethernetmanagement switch

U21 RJ-45 uses CAT5 orbetter cables.

4 4 storage servers, with these networkcomponents:• 4 1GbE/10GbE ports (NET 0, NET 1,

NET 2, NET 3)• 1 dual-port 4X QDR (40 Gb/s) IB

HCA• 1 SP Ethernet port for Oracle ILOM

connectivity

U2 - U9 10GbE usesCAT6A or better.Other RJ-45connections canuse CAT5 or bettercables.IB ports usestandard IB cableswith QSFPconnectors.


▼ Installing Expansion Racks

Note – For information about preparing the site for expansion racks see “Preparingthe Site (Storage Rack and Expansion Racks)” on page 11.

1. Position the expansion racks adjacent to SuperCluster M6-32.

See “Hardware Installation Task Overview” on page 3.

2. If you are installing quarter racks, you must install an IB switch (spine switch)in each quarter rack in U1.

Refer to the IB switch documentation for installation instructions. See “HardwareInstallation Documents” on page 8.

3. Connect the expansion rack power cords to facility power and apply power.

This action enables the expansion racks to initialize while you perform theremaining installation steps. Refer to the Sun Rack II User’s Guide. See “HardwareInstallation Documents” on page 8.

4. In the storage rack and in each expansion rack, remove factory installed leaf andspine cables so that ports are available to connect the racks together.

Remove the cables listed in this table.

5. Cable the expansion racks to the storage rack and to each other.

Use one of these sections based on the number of expansion racks you are adding:

From To

Slot Port Slot Port Cable Length

U1 1B U20 8B 3 m

U1 0B U22 8B 3 m

U20 9B U22 9A 2 m

U20 10B U22 10A 2 m

U20 11B U22 11A 2 m

U20 8A U22 8A 2 m

U20 9A U22 9B 2 m

U20 10A U22 10B 2 m

U20 11A U22 11B 2 m


■ “One Expansion Rack Cabling” on page 60

■ “Two Expansion Racks Cabling” on page 62

■ “Three Expansion Racks Cabling” on page 65

■ “Four Expansion Racks Cabling” on page 68

■ “Five Expansion Racks Cabling” on page 71

■ “Six Expansion Racks Cabling” on page 76

■ “Seven Expansion Racks Cabling” on page 81

6. Continue to the next SuperCluster M6-32 installation step.

See “Hardware Installation Task Overview” on page 3.

Related Information■ “Preparing the Site (Storage Rack and Expansion Racks)” on page 11





One Expansion Rack CablingThese abbreviations are used in the tables:

■ SR – SuperCluster M6-32 storage rack

■ Rn – Rack n (R1 is the first expansion rack, R2 is the second, and so on.)

■ U1 – Spine switch in U1

■ U20 – Leaf switch 1 in U20


■ Pn – Port n


TABLE: IB Switch Connections for the Storage Rack When Adding One Expansion Rack

Leaf Switch Connection Cable Length

SR-U22 within SR SR-U22-P8A to SR-U1-P3ASR-U22-P8B to SR-U1-P4ASR-U22-P9A to SR-U1-P5ASR-U22-P9B to SR-U1-P6A

3 m

SR-U22 to R1 SR-U22-P10A to R1-U1-P7ASR-U22-P10B to R1-U1-P8ASR-U22-P11A to R1-U1-P9ASR-U22-P11B to R1-U1-P10A

5 m

SR-U20 within SR SR-U20-P8A to SR-U1-P3BSR-U20-P8B to SR-U1-P4BSR-U20-P9A to SR-U1-P5BSR-U20-P9B to SR-U1-P6B

3 m

SR-U20 to R1 SR-U20-P10A to R1-U1-P7BSR-U20-P10B to R1-U1-P8BSR-U20-P11A to R1-U1-P9BSR-U20-P11B to R1-U1-P10B

5 m

TABLE: IB Switch Connections for the First Expansion Rack of One


R1-U22 within R1 R1-U22-P8A to R1-U1-P3AR1-U22-P8B to R1-U1-P4AR1-U22-P9A to R1-U1-P5AR1-U22-P9B to R1-U1-P6A

3 m


Related Information



Two Expansion Racks CablingThese abbreviations are used in the tables:






■ Pn – Port n

R1-U22 to SR R1-U22-P10A to SR-U1-P7AR1-U22-P10B to SR-U1-P8AR1-U22-P11A to SR-U1-P9AR1-U22-P11B to SR-U1-P10A

5 m

R1-U20 within R1 R1-U20-P8A to R1-U1-P3BR1-U20-P8B to R1-U1-P4BR1-U20-P9A to R1-U1-P5BR1-U20-P9B to R1-U1-P6B

5 m

R1-U20 to SR R1-U20-P10A to SR-U1-P7BR1-U20-P10B to SR-U1-P8BR1-U20-P11A to SR-U1-P9BR1-U20-P11B to SR-U1-P10B

5 m

TABLE: IB Switch Connections for the First Expansion Rack of One (Continued)



TABLE: IB Switch Connections for the Storage Rack When Adding Two Expansion Racks


SR-U22 within SR SR-U22-P8A to SR-U1-P3ASR-U22-P8B to SR-U1-P4ASR-U22-P9A to SR-U1-P5A

3 m

SR-U22 to R1 SR-U22-P9B to R1-U1-P6ASR-U22-P10A to R1-U1-P7ASR-U22-P10B to R1-U1-P8A

5 m

SR-U22 to R2 SR-U22-P11A to R2-U1-P9ASR-U22-P11B to R2-U1-P10A

5 m

SR-U20 within SR SR-U20-P8A to SR-U1-P3BSR-U20-P8B to SR-U1-P4BSR-U20-P9A to SR-U1-P5B

5 m

SR-U20 to R1 SR-U20-P9B to R1-U1-P6BSR-U20-P10A to R1-U1-P7BSR-U20-P10B to R1-U1-P8B

5 m

SR-U20 to R2 SR-U20-P11A to R2-U1-P9BSR-U20-P11B to R2-U1-P10B

5 m

TABLE: IB Switch Connections for the First Expansion Rack of Two


R1-U22 within R1 R1-U22-P8A to R1-U1-P3AR1-U22-P8B to R1-U1-P4AR1-U22-P9A to R1-U1-P5A

3 m

R1-U22 to SR R1-U22-P11A to SR-U1-P9AR1-U22-P11B to SR-U1-P10A

5 m

R1-U22 to R2 R1-U22-P9B to R2-U1-P6AR1-U22-P10A to R2-U1-P7AR1-U22-P10B to R2-U1-P8A

5 m


Related Information



R1-U20 within R1 R1-U20-P8A to R1-U1-P3BR1-U20-P8B to R1-U1-P4BR1-U20-P9A to R1-U1-P5B

5 m

R1-U20 to SR R1-U20-P11A to SR-U1-P9BR1-U20-P11B to SR-U1-P10B

5 m

R1-U20 to R2 R1-U20-P9B to R2-U1-P6BR1-U20-P10A to R2-U1-P7BR1-U20-P10B to R2-U1-P8B

5 m

TABLE: IB Switch Connections for the Second Expansion Rack of Two


R2-U22 within R2 R2-U22-P8A to R2-U1-P3AR2-U22-P8B to R2-U1-P4AR2-U22-P9A to R2-U1-P5A

3 m

R2-U22 to SR R2-U22-P9B to SR-U1-P6AR2-U22-P10A to SR-U1-P7AR2-U22-P10B to SR-U1-P8A

5 m

R2-U22 to R1 R2-U22-P11A to R1-U1-P9AR2-U22-P11B to R1-U1-P10A

5 m

R2-U20 within R2 R2-U20-P8A to R2-U1-P3BR2-U20-P8B to R2-U1-P4BR2-U20-P9A to R2-U1-P5B

5 m

R2-U20 to SR R2-U20-P9B to SR-U1-P6BR2-U20-P10A to SR-U1-P7BR2-U20-P10B to SR-U1-P8B

5 m

R2-U20 to R1 R2-U20-P11A to R1-U1-P9BR2-U20-P11B to R1-U1-P10B

5 m

TABLE: IB Switch Connections for the First Expansion Rack of Two



Three Expansion Racks CablingThese abbreviations are used in the tables:






■ Pn – Port n

TABLE: IB Switch Connections for the Storage Rack When Adding Three Expansion Racks


SR-U22 within SR SR-U22-P8A to SR-U1-P3ASR-U22-P8B to SR-U1-P4A

3 m


5 m


5 m


10 m

SR-U20 within SR SR-U20-P8A to SR-U1-P3BSR-U20-P8B to SR-U1-P4B

5 m


5 m


5 m


10 m


TABLE: IB Switch Connections for the First Expansion Rack of Three


R1-U22 within R1 R1-U22-P8A to R1-U1-P3AR1-U22-P8B to R1-U1-P4A

3 m


5 m


5 m


5 m

R1-U20 within R1 R1-U20-P8A to R1-U1-P3BR1-U20-P8B to R1-U1-P4B

5 m


5 m


5 m


5 m

TABLE: IB Switch Connections for the Second Expansion Rack of Three



3 m


5 m


5 m


5 m


5 m


Related Information




5 m


5 m


5 m

TABLE: IB Switch Connections for the Third Expansion Rack of Three



3 m


10 m


5 m


5 m


5 m


10 m


5 m


5 m

TABLE: IB Switch Connections for the Second Expansion Rack of Three



Four Expansion Racks CablingThese abbreviations are used in the tables:






■ Pn – Port n

TABLE: IB Switch Connections for the Storage Rack When Adding Four Expansion Racks


SR U22 within SR SR-U22-P8A to SR-U1-P3ASR-U22-P8B to SR-U1-P4A

3 m

SR U22 to R1 SR-U22-P9A to R1-U1-P5ASR-U22-P9B to R1-U1-P6A

5 m


5 m

SR U22 to R3 SR-U22-P11A to R3-U1-P9A 10 m

SR U22 to R4 SR-U22-P11B to R4-U1-P10A 10 m

SR U20 within SR SR-U20-P8A to SR-U1-P3BSR-U20-P8B to SR-U1-P4B

3 m

SR U20 to R1 SR-U20-P9A to R1-U1-P5BSR-U20-P9B to R1-U1-P6B

3 m

SR U20 to R2 SR-U20-P10A to R2-U1-P7BSR-U20-P10B to R2-U1-P8B

5 m

SR U20 to R3 SR-U20-P11A to R3-U1-P9B 10 m

SR U20 to R4 SR-U20-P11B to R4-U1-P10B 10 m


TABLE: IB Switch Connections for the First Expansion Rack of Four


R1 U22 within R1 R1-U22-P8A to R1-U1-P3AR1-U22-P8B to R1-U1-P4A

3 m

R1 U22 to SR R1-U22-P11B to SR-U1-P10A 5 m

R1 U22 to R2 R1-U22-P9A to R2-U1-P5AR1-U22-P9B to R2-U1-P6A

5 m


5 m

R1 U22 to R4 R1-U22-P11A to R4-U1-P9A 10 m

R1 U20 within R1 R1-U20-P8A to R1-U1-P3BR1-U20-P8B to R1-U1-P4B

3 m

R1 U20 to SR R1-U20-P11B to SR-U1-P10B 5 m

R1 U20 to R2 R1-U20-P9A to R2-U1-P5BR1-U20-P9B to R2-U1-P6B

5 m


5 m

R1 U20 to R4 R1-U20-P11A to R4-U1-P9B 10 m

TABLE: IB Switch Connections for the Second Expansion Rack of Four



3 m

R2 U22 to SR R2-U22-P11A to SR-U1-P9A 5 m

R2 U22 to R1 R2-U22-P11B to R1-U1-P10A 5 m


5 m


5 m


3 m

R2 U20 to SR R2-U20-P11A to SR-U1-P9B 5 m


R2 U20 to R1 R2-U20-P11B to R1-U1-P10B 5 m


5 m


5 m

TABLE: IB Switch Connections for the Third Expansion Rack of Four



3 m

R3 U22 to SR R3-U22-P10A to SR-U1-P7AR3-U22-P10B to SR-U1-P8A

10 m




5 m


3 m

R3 U20 to SR R3-U20-P10A to SR-U1-P7BR3-U20-P10B to SR-U1-P8B

10 m




5 m

TABLE: IB Switch Connections for the Second Expansion Rack of Four (Continued)



Related Information



Five Expansion Racks CablingThese abbreviations are used in the tables:






TABLE: IB Switch Connections for the Fourth Expansion Rack of Four



3 m


10 m


10 m




3 m


10 m


10 m




■ Pn – Port n

TABLE: IB Switch Connections for the Storage Rack When Adding Five Expansion Racks



3 m


5 m






3 m

SR U20 to R1R1 SR-U20-P9A to R1-U1-P5BSR-U20-P9B to R1-U1-P6B

5 m





TABLE: IB Switch Connections for the First Expansion Rack of Five



3 m



5 m



R1 U22 to R5 R1-U22-P11Ato R5-U1-P9A 10 m


3 m




5 m



R1 U20 to R5 R1-U20-P11Ato R5-U1-P9B 10 m

TABLE: IB Switch Connections for the Second Expansion Rack of Five



3 m




5 m




3 m




5 m



TABLE: IB Switch Connections for the First Expansion Rack of Five (Continued)



TABLE: IB Switch Connections for the Third Expansion Rack of Five



3 m





5 m



3 m





5 m


TABLE: IB Switch Connections for the Fourth Expansion Rack of Five



3 m






5 m


3 m



Related Information







5 m

TABLE: IB Switch Connections for the Fifth Expansion Rack of Five



3 m


10 m






3 m



10 m




TABLE: IB Switch Connections for the Fourth Expansion Rack of Five (Continued)



Six Expansion Racks CablingThese abbreviations are used in the tables:






■ Pn – Port n

TABLE: IB Switch Connections for the Storage Rack When Adding Six Expansion Racks



3 m








3 m








TABLE: IB Switch Connections for the First Expansion Rack of Six



3 m








3 m





R1 U20 to R5 R1-U20-P10Bto R5-U1-P8B 10 m


TABLE: IB Switch Connections for the Second Expansion Rack of Six



3 m








3 m








TABLE: IB Switch Connections for the Third Expansion Rack of Six



3 m








3 m







TABLE: IB Switch Connections for the Second Expansion Rack of Six (Continued)



TABLE: IB Switch Connections for the Fourth Expansion Rack of Six



3 m








3 m







TABLE: IB Switch Connections for the Fifth Expansion Rack of Six



3 m








3 m


Related Information









TABLE: IB Switch Connections for the Sixth Rack of Six



3 m








3 m







TABLE: IB Switch Connections for the Fifth Expansion Rack of Six (Continued)



Seven Expansion Racks CablingThese abbreviations are used in the tables:






■ Pn – Port n

TABLE: IB Switch Connections for the Storage Rack When Adding Seven ExpansionRacks


SR U22 within SR SR-U22-P8A to SR-U1-P3A 3 m








SR U20 within SR SR-U20-P8A to SR-U1-P3B 3 m









TABLE: IB Switch Connections for the First Expansion Rack of Seven


R1 U22 within R1 R1-U22-P8A to R1-U1-P3A 3 m








R1 U20 within R1 R1-U20-P8A to R1-U1-P3B 3 m








TABLE: IB Switch Connections for the Second Expansion Rack of Seven



















TABLE: IB Switch Connections for the Third Expansion Rack of Seven


















TABLE: IB Switch Connections for the Second Expansion Rack of Seven (Continued)



TABLE: IB Switch Connections for the Fourth Expansion Rack of Seven


















TABLE: IB Switch Connections for the Fifth Expansion Rack of Seven



















TABLE: IB Switch Connections for the Sixth Expansion Rack of Seven


















TABLE: IB Switch Connections for the Fifth Expansion Rack of Seven (Continued)



Related Information



TABLE: IB Switch Connections for the Seventh Expansion Rack of Seven



















Expansion Rack Default IP Addresses

Component NET0 IP AddressesOracle ILOM IPAddresses

IB Active Bonded IPAddresses

Exadata Storage Server 18 192.168.1.68 192.168.1.168 192.168.10.86

192.168.10.85


192.168.10.83


192.168.10.81


192.168.10.79


192.168.10.77


192.168.10.75


192.168.10.73


192.168.10.71


192.168.10.69


192.168.10.67


192.168.10.65


192.168.10.63


192.168.10.61


192.168.10.59


192.168.10.57


Related Information






Understanding Internal Cabling(Expansion Rack)Use these topics to identify expansion rack internal factory installed cableconnections.

■ “Oracle ILOM Cabling” on page 89

■ “Administrative Gigabit Ethernet Port Cabling” on page 91

■ “Single-Phase PDU Cabling” on page 93

■ “Three-Phase PDU Cabling (Full Rack)” on page 95

■ “IB Network Cabling” on page 97


192.168.10.55


192.168.10.53


192.168.10.51

IB switch 3 192.168.1.223 NA NA

IB switch 2 192.168.1.222 NA NA

IB switch 1 192.168.1.221 NA NA

Ethernet switch 192.168.1.220 NA NA

PDU-A 192.168.1.212 NA NA

PDU-B 192.168.1.213 NA NA

Component NET0 IP AddressesOracle ILOM IPAddresses

IB Active Bonded IPAddresses


Related Information






Oracle ILOM Cabling

TABLE: Oracle ILOM Cabling for the Expansion Rack (Full Rack)

From Rack Unit Type of Equipment Gigabit Ethernet Port

U41 Exadata Storage Server 2



















Related Information


■ “Administrative Gigabit Ethernet Port Cabling” on page 91

TABLE: Oracle ILOM Cabling for the Expansion Rack (Half Rack)











TABLE: Oracle ILOM Cabling for the Expansion Rack (Quarter Rack)







Administrative Gigabit Ethernet Port Cabling

TABLE: Gigabit Ethernet Cabling for the Expansion Rack (Full Rack)











U24 IB switch 45

U20 IB switch 46










U1 IB switch 47

PDU-A PDU 15

PDU-B PDU 19


Related Information


TABLE: Gigabit Ethernet Cabling for the Expansion Rack (Half Rack)


U24 IB switch 45

U20 IB switch 46










U1 IB switch 47

PDU-A PDU 15

PDU-B PDU 19

TABLE: Gigabit Ethernet Cabling for the Expansion Rack (Quarter Rack)


U24 IB switch 45

U20 IB switch 46





PDU-A PDU 15

PDU-B PDU 19


■ “Single-Phase PDU Cabling” on page 93

Single-Phase PDU Cabling

TABLE: Single-Phase PDU Cabling for the Expansion Rack (Full Rack)

Rack Unit PDU-A/PS-00 PDU-B/PS-01 Cable Length

U41 G5-6 G0-0 2 m

U39 G5-3 G0-3 2 m

U37 G5-0 G0-6 2 m

U35 G4-6 G1-0 2 m

U33 G4-4 G1-2 2 m

U31 G4-2 G1-4 2 m

U29 G3-6 G2-0 2 m

U27 G3-5 G2-1 2 m

U25 G3-3 G2-3 2 m

U24 G3-1 G2-5 2 m

U23 N/A G3-0 included

U22 G2-5 G3-1 1 m

U21 G3-0 G2-6 2 m

U20 G2-4 G3-2 2 m

U18 G2-2 G3-4 2 m

U16 G1-6 G4-0 2 m

U14 G2-0 G3-6 2 m

U12 G1-4 G4-2 2 m

U10 G1-2 G4-4 2 m

U8 G1-0 G4-6 2 m

U6 G0-6 G5-0 2 m

U4 G0-4 G5-2 2 m

U2 G0-2 G5-4 2 m

U1 G0-0 G5-6 2 m


TABLE: Single-Phase PDU Cabling for the Expansion Rack (Half Rack)


U24 G3-1 G2-5 2 m


U22 G2-5 G3-1 1 m

U21 G3-0 G2-6 2 m

U20 G2-4 G3-2 2 m

U18 G2-2 G3-4 2 m

U16 G1-6 G4-0 2 m

U14 G2-0 G3-6 2 m

U12 G1-4 G4-2 2 m

U10 G1-2 G4-4 2 m

U8 G1-0 G4-6 2 m

U6 G0-6 G5-0 2 m

U4 G0-4 G5-2 2 m

U2 G0-2 G5-4 2 m

TABLE: Single-Phase PDU Cabling for the Expansion Rack (Quarter Rack)


U24 G3-1 G2-5 2 m


U22 G2-5 G3-1 1 m

U21 G3-0 G2-6 2 m

U20 G2-4 G3-2 2 m

U8 G1-0 G4-6 2 m

U6 G0-6 G5-0 2 m

U4 G0-4 G5-2 2 m

U2 G0-2 G5-4 2 m


Related Information


■ “Three-Phase PDU Cabling (Full Rack)” on page 95

Three-Phase PDU Cabling (Full Rack)

TABLE: Three-Phase PDU Cabling for the Expansion Rack (Full Rack)


U41 G5-6 G2-0 2 m

U39 G5-3 G2-3 2 m

U37 G5-0 G2-6 2 m

U35 G4-6 G1-0 2 m

U33 G4-4 G1-2 2 m

U31 G4-2 G1-4 2 m

U29 G3-6 G0-0 2 m

U27 G3-5 G0-1 2 m

U25 G3-3 G0-3 2 m

U24 G3-1 G0-5 2 m


U22 G2-5 G5-1 1 m

U21 G3-0 G0-6 2 m

U20 G2-4 G5-2 2 m

U18 G2-2 G5-4 2 m

U16 G1-6 G4-0 2 m

U14 G2-0 G5-6 2 m

U12 G1-4 G4-2 2 m

U10 G1-2 G4-4 2 m

U8 G1-0 G4-6 2 m

U6 G0-6 G3-0 2 m

U4 G0-4 G3-2 2 m

U2 G0-2 G3-4 2 m

U1 G0-0 G3-6 2 m


TABLE: Three-Phase PDU Cabling for the Expansion Rack (Half Rack)


U24 G3-1 G0-5 2 m


U22 G2-5 G5-1 1 m

U21 G3-0 G0-6 2 m

U20 G2-4 G5-2 2 m

U18 G2-2 G5-4 2 m

U16 G1-6 G4-0 2 m

U14 G2-0 G5-6 2 m

U12 G1-4 G4-2 2 m

U10 G1-2 G4-4 2 m

U8 G1-0 G4-6 2 m

U6 G0-6 G3-0 2 m

U4 G0-4 G3-2 2 m

U2 G0-2 G3-4 2 m

U1 G0-0 G3-6 2 m

TABLE: Three-Phase PDU Cabling for the Expansion Rack (Quarter Rack)


U24 G3-1 G0-5 2 m


U22 G2-5 G5-1 1 m

U21 G3-0 G0-6 2 m

U20 G2-4 G5-2 2 m

U8 G1-0 G4-6 2 m

U6 G0-6 G3-0 2 m

U4 G0-4 G3-2 2 m

U2 G0-2 G3-4 2 m


Related Information


■ “IB Network Cabling” on page 97

IB Network Cabling

TABLE: IB Network Cabling for the Expansion Rack (Full Rack)

From IB SwitchRack Unit Port To Rack Unit Type of Equipment Port Cable Description

U24 0A U41 Exadata Storage Server PCIe 3, P1 3 m QDR IB cable

U24 0B U39 Exadata Storage Server PCIe 3, P1 3 m QDR IB cable




































U20 9B U24 Sun Datacenter InfiniBandSwitch 36 switch

9A 2 m QDR IB cable


10A 2 m QDR IB cable



U20 8A U24 Sun Datacenter InfiniBandSwitch 36 switch

8A 2 m QDR IB cable


9B 2 m QDR IB cable


10B 2 m QDR IB cable




8B 3 m QDR IB cable


8B 3 m QDR IB cable

TABLE: IB Network Cabling for the Expansion Rack (Full Rack) (Continued)



TABLE: IB Network Cabling for the Expansion Rack (Half Rack)





















9A 2 m QDR IB cable






8A 2 m QDR IB cable


9B 2 m QDR IB cable







8B 3 m QDR IB cable


8B 3 m QDR IB cable

TABLE: IB Network Cabling for the Expansion Rack (Quarter Rack)











9A 2 m QDR IB cable






8A 2 m QDR IB cable


9B 2 m QDR IB cable





TABLE: IB Network Cabling for the Expansion Rack (Half Rack) (Continued)



Related Information



Glossary

AApplication Domain A domain that runs Oracle Solaris and client applications.

ASMM Automatic shared memory management.

ASR Auto Service Request. A feature of Oracle or Sun hardware thatautomatically opens service requests when specific hardware faults occur.ASR is integrated with MOS and requires a support agreement. See alsoMOS.

Bbase configuration

PDomainA SuperCluster M6-32 configuration comprised of two or four PDomains,where one DCU is associated with each PDomain. A base configurationPDomain can reside on a single compute server or can be split across twocompute servers. See also compute server, DCU, extended configurationPDomain, and PDomain.

CCFM Cubic feet per minute.

Cisco Catalyst Ethernetswitch

Provides the SuperCluster M6-32 management network. Referred to in thisdocumentation using the shortened name “Ethernet management switch.”See also Ethernet management switch.

103

CMP Chip multiprocessing. Each CMU contains 2 CMP processors. The computeserver can contain a maximum of 32 CMPs.

CMU CPU memory unit. Each CMU in the compute server contains two CMPsand two sets of DIMM slots.

COD Capacity on Demand.

compute server Shortened name for the SPARC M6-32 server, a major component ofSuperCluster M6-32. See also SPARC M6-32 server.

DDatabase Domain The domain that contains the SuperCluster M6-32 database.

DB Oracle Database.

DCM Domain configuration management. The reconfiguration of boards inPDomains for Enterprise-class systems. See also PDomain.

DCU Domain configurable unit. The smallest building block for PDomains. EachDCU in the compute server contains two or four CMUs and one IOU. Seealso PDomain.

dedicated domain A SuperCluster LDom category that includes the domains configured atinstallation time as either a Database Domain or an Application Domain(running the Oracle Solaris 10 or Oracle Solaris 11 OS). Dedicated domainshave direct access to the 10GbE NICs and IB HCAs (and Fibre Channelcards, if present). See also Database Domain and Application Domain.

DHCP Dynamic Host Configuration Protocol. Software that automatically assignsIP addresses to clients on a TCP/IP network. See also TCP.

DIMM Dual in-line memory module.

DISM Dynamic intimate shared memory.

EEECS Oracle Exalogic Elastic Cloud software.

EMS Express module SAS. Each EMS contains two 10GBASE-T networkconnections and provides access to four hard drives on the compute server.

EPO switch Emergency power-off switch.


ESD Electrostatic discharge.

Ethernet managementswitch

Shortened name for the Cisco Catalyst Ethernet switch. See also CiscoCatalyst Ethernet switch.

expansion rack Shortened name for optional Oracle Exadata Storage Expansion Racks (up to17) that can be added to SuperCluster M6-32. See also Oracle Exadata StorageExpansion Rack.

extended configurationPDomain

A SuperCluster M6-32 configuration comprised of two PDomains, wheretwo DCUs are associated with each PDomain. An extended configurationPDomain can reside on a single compute server or can be split across twocompute servers. See also, base configuration PDomain, compute server, DCU,and PDomain.

FFAN Fast application notification event.

FCoE Fibre Channel over Ethernet.

FM Fan module.

FMA Fault management architecture. A feature of Oracle Solaris servers thatincludes error handlers, structured error telemetry, automated diagnosticsoftware, response agents, and messaging.

FRU Field-replaceable unit.

fully-populated DCUconfiguration

A configuration where each DCU in the compute servers contains fourCMUs. See also DCU and half-populated DCU configuration.

GGB Gigabyte. 1 gigabyte = 1024 megabytes.

GbE Gigabit Ethernet.

GNS Grid Naming Service.

Glossary 105

Hhalf-populated DCU

configurationA configuration where each DCU in the compute servers contains twoCMUs. See also DCU and fully-populated DCU configuration.

HCA Host channel adapter.

HDD Hard disk drive. In Oracle Solaris OS output, HDD can refer to hard diskdrives or SSDs.

IIB InfiniBand.

IB switch Shortened name for the Sun Datacenter InfiniBand Switch 36. See also leafswitch, spine switch, and Sun Datacenter InfiniBand Switch 36.

ILOM See Oracle ILOM.

I/O Domain If you have Root Domains, you create I/O Domains with your choice ofresources at the time of your choosing. The I/O Domain Creation toolenables you to assign resources to I/O Domains from the CPU and memoryrepositories, and from virtual functions hosted by Root Domains. When youcreate an I/O Domain, you assign it as a Database Domain or ApplicationDomain running the Oracle Solaris 11 OS. See also Root Domain.

IOU I/O unit. The compute server contains up to 4 IOUs, one for each DCU. EachIOU supports up to 16 PCIe slots, 8 10GBASE-T ports on 4 EMS modules,and 8 drives.

IPMI Intelligent Platform Management Interface.

IPMP IP network multipathing.

iSCSI Internet Small Computer System Interface.

KKVMS Keyboard video mouse storage.


Lleaf switch Two of the IB switches are configured as leaf switches, the third is

configured as a spine switch. See also IB switch.

LDom Logical domain. A virtual machine comprising a discrete logical grouping ofresources that has its own operating system and identity within a singlecomputer system. LDoms are created using Oracle VM Server for SPARCsoftware. See also Oracle VM Server for SPARC.

MMIB Management information base.

MOS My Oracle Support.

NNET MGT The network management port on an SP. See also SP.

NIC Network interface card.

NUMA Nonuniform memory access.

OOBP OpenBoot PROM. Firmware on SPARC servers that enables the server to

load platform-independent drivers directly from devices, and provides aninterface through which you can boot the compute server and run low-leveldiagnostics.

Oracle ASM Oracle Automatic Storage Management. A volume manager and a filesystem that supports Oracle databases.

OCM Oracle Configuration Manager.

ONS Oracle Notification Service.

Glossary 107

Oracle Exadata StorageExpansion Rack

Optional expansion racks (in full, half, or quarter configurations) that can beadded to SuperCluster M6-32 systems that require additional storage.Referred to in this documentation using the shortened name “expansionrack.” See also expansion rack.

Oracle ILOM Oracle Integrated Lights Out Manager. Software on the SP that enables youto manage a server independently from the operating system. See also SP.

Oracle Solaris OS Oracle Solaris operating system.

Oracle SuperCluster Refers to all Oracle SuperCluster models.

Oracle SuperClusterM6-32

Full name of this SuperCluster model. Referred to in this documentationusing the shortened name “SuperCluster M6-32.” See also SuperClusterM6-32.

Oracle SuperClusterM6-32 storage rack

Full name of first storage rack that contains the storage servers, ZFS storageappliance, IB switches, and Ethernet management switch. Referred to in thisdocumentation using the shortened name “storage rack.” See also storagerack.

Oracle VM Server forSPARC

SPARC server virtualization and partitioning technology. See also LDom.

Oracle VTS Oracle Validation Test Suite. An application, preinstalled with Oracle Solaris,that exercises the system, provides hardware validation, and identifiespossible faulty components.

Oracle XA Oracle’s implementation of the X/Open distributed transaction processingXA interface that is included in Oracle DB software.

Oracle ZFS ZS3-ESstorage appliance

Located in the storage rack, it provides SuperCluster M6-32 with sharedstorage capabilities. Referred to in this documentation using the shortenedname “ZFS storage appliance.” See also ZFS storage appliance.

OS Operating system.

Pparked resources CPU and memory resources that are set aside in the CPU and memory

repositories. You assign parked resources to I/O Domains with the I/ODomain Creation tool.

PCIe Peripheral Component Interconnect Express.

PDomain Physical domain. Each PDomain on the compute server is an independentlyconfigurable and bootable entity with full hardware domain isolation forfault isolation and security purposes. See also compute server, DCU, and SSB.


PDomain-SPP The lead SPP of a PDomain. The PDomain-SPP on the compute servermanages tasks and provides rKVMS service for that PDomain. See alsoPDomain.

PDU Power distribution unit.

PF Physical function. Functions provided by physical I/O devices, such as theIB HCAs, 10GbE NICs, and any Fibre Channel cards installed in the PCIeslots. Logical devices, or virtual functions (VFs), are created from PFs, witheach PF hosting 32 VFs.

POST Power-on self-test. A diagnostic that runs when the compute server ispowered on.

PS Power supply.

PSDB Power system distribution board.

PSH Predictive self healing. An Oracle Solaris OS technology that continuouslymonitors the health of the compute server and works with Oracle ILOM totake a faulty component offline if needed.

QQMU Quarterly maintenance update.

QSFP Quad small form-factor, pluggable. A transceiver specification for 10GbEtechnology.

RRAC Real Application Cluster.

RCLB Runtime connection load balancing.

rKVMS Remote keyboard video mouse and storage.

root complex CMP circuitry that provides the base to a PCIe I/O fabric. Each PCIe I/Ofabric consists of the PCIe switches, PCIe slots, and leaf devices associatedwith the root complex.

Glossary 109

Root Domain A logical domain that is configured at installation time. Root Domains arerequired if you plan to configure I/O Domains. Root Domains host PFs fromwhich I/O Domains derive VFs. The majority of Root Domain CPU andmemory resources are parked for later use by I/O Domains.

SSAS Serial attached SCSI.

SATA Serial advance technology attachment.

scalability The ability to increase (or scale up) processing power in a compute server bycombining the server’s physical configurable hardware (see also DCU) intoone or more logical groups (see also PDomain).

SER MGT The serial management port on an SP. See also SP.

SCAN Single Client Access Name. A feature used in RAC environments thatprovides a single name for clients to access any Oracle Database running ina cluster. See also RAC.

SDP Session Description Protocol.

SFP and SFP+ Small form-factor pluggable standard. SFP+ is a specification for atransceiver for 10GbE technology.

SGA System global area.

SMF Service Management Facility.

SNEEP Serial number in EEPROM.

SNMP Simple Management Network Protocol.

SP Service processor. A processor, separate from the host, that monitors andmanages the host no matter what state the host is in. The SP runs OracleILOM, which provides remote lights out management. In SuperClusterM6-32, SPs are located on the compute servers, storage servers, ZFS storageappliance controllers, and IB switches. See also Oracle ILOM.

SPARC M6-32 server A major component of SuperCluster M6-32 that provides the main computeresources. Referred to in this documentation using the shortened name“compute server.” See also compute server.

spine switch One of the SuperCluster M6-32 IB switches that is configured as a spineswitch. See also IB switch and leaf switch.


SPP Service processor proxy. One SPP in the compute server is assigned tomanage each PDomain. SPPs monitor environmental sensors and managethe CMUs, memory controllers, and DIMMs within the DCU. See alsoPDomain-SPP.

SR-IOV Domain Single-Root I/O Virtualization Domain -- A SuperCluster logical domaincategory that includes Root Domains and I/O Domains. This category ofdomains support single-root I/O virtualization. See also I/O Domain andRoot Domain.

SSB Scalability switch board in the compute server.

SSD Solid state drive.

STB Oracle Services Tool Bundle.

storage rack Shortened name for the Oracle SuperCluster M6-32 storage rack thatcontains the storage servers. See also Oracle SuperCluster M6-32 storage rack.

storage server Storage servers in SuperCluster M6-32.

Sun DatacenterInfiniBand Switch 36

Interconnects SuperCluster M6-32 components on a private network.Referred to in this documentation using the shortened name “IB switch.” Seealso IB switch, leaf switch, and spine switch.

SuperCluster M6-32 Shortened name for Oracle SuperCluster M6-32. See also Oracle SuperClusterM6-32.

TTCP Transmission Control Protocol.

TNS Transparent Network Substrate.

TPM Trusted platform module.

UUPS Uninterruptible power supply.

Glossary 111

VVAC Voltage alternating current.

VF Virtual function. Logical I/O devices that are created from PFs, with each PFhosting 32 VFs.

VIP Virtual IP.

VLAN Virtual local area network.

VNET Virtual network.

WWWN World Wide Name.

XXA See Oracle XA.

ZZFS A file system with added volume management capabilities. ZFS is the

default file system in Oracle Solaris 11.

ZFS storage appliance Shortened name for Oracle ZFS Storage ZS3-ES storage appliance. See alsoOracle ZFS ZS3-ES storage appliance.

ZFS storage controller Servers in the Oracle ZFS ZS3-ES storage appliance that manage the storageappliance. See also ZFS storage appliance.


Index

Aairflow requirements, storage rack, 21

Ccable drops, installing, 29cable types, 29, 31cabling

Ethernet management switch, 46expansion racks, 53, 59IB switches, 39storage appliance, 36SuperCluster M6-32, 31, 51

cabling referenceEthernet management switch, 47expansion racks, 60, 62, 65, 68, 71, 76, 81, 88IB switch-to-switch, 45leaf switch 1, 41leaf switch 2, 43storage appliance, 38

circuit breaker requirements, storage rack, 17client access network

topology, 26components

compute server, 31expansion rack, 55storage rack, 12

compute servers, 1host names and IP addresses, 28network components, 31

connecting expansion racks, 53cooling preparation, storage rack, 19

DDNS, preparing, 30documents for hardware installation, 8

EEMS network ports, 31Ethernet management switch

cabling, 46cabling reference, 47location, 34

expansion rackscabling tables for five, 71cabling tables for four, 68cabling tables for one, 60cabling tables for seven, 81cabling tables for six, 76cabling tables for three, 65cabling tables for two, 62components, 55connecting, 53cooling specifications, 19default IP addresses, 87environmental specifications, 23installing, 59internal cabling, 88layouts, 56overview, 54physical specifications, 13power specifications, 15preparing the site, 11

Ffacility

networks, 51power requirements, storage rack, 17

front expansion rack layouts, 56

GGbE ports, 31grounding guidelines, storage rack, 17

113

Hhardware installation

documents, 8overview, 1task overview, 3

host names, default, 28

IIB HCA, 31IB network, 26IB switches

cabling, 39cabling reference (leaf switch 1), 41cabling reference (leaf switch 2), 43location, 34switch-to-switch cabling, 45

installation task overview, 3installing

cable drops, 29expansion racks, 59SuperCluster M6-32, 1

internal cabling, expansion racks, 88IP addresses

compute server, default, 28default expansion rack, 87

Mmanagement network

description, 26

Nnetwork

infrastructure requirements, 27preparing, 25topology, 26

network componentscompute server, 31storage rack, 34

OOracle Exadata Storage HC Expansion Rack

components, 54overview

expansion racks, 54hardware installation, 1installation task, 3

PPDU power specifications, 15perforated floor tiles, 22physical specifications, storage rack, 13power specifications, storage rack, 15preparing

DNS, 30for installation, 3network, 25site, 3site (storage rack), 11

product notes, viii

Sspecifications

storage rack physical, 13storage rack power, 15

SPscompute server, 31

storage appliance, 1cabling, 36cabling reference, 38location, 34

storage rack, 1airflow requirements, 21components, 12cooling requirements, 19facility power requirements, 17network components, 34perforated flow tiles, 22physical specifications, 13power specifications, 15preparing the site, 11, 12

storage serversinstalling, 3location, 34

SuperCluster M6-32cabling to facility networks, 51installing, 1

T

topology, network, 26


Owner’s Guide: Installation

Documents

Transcript of Owner’s Guide: Installation