1

The Renaissance of

SPARC Björn Rost / Philippe Fierens

Stories from the field

2

Not so safe harbour ;-)

3

About Björn

• Founder, manager and consultant

• President RAC SIG

• ACE Director

About Phil

Philippe Fierens Independent

Oracle DBA (not solaris admin) Architect

and Oracle ACE Associate

and most of all DAD & husband ;-)

AND …

5

whoami

BBL

6

AgendaHistoric overview

SPARC features

Comparing performance : stories from the field

Licensing in a multi core multi socket world

Sparc Super Cluster Exadata ++

Solaris : well ahead of comet

The near (R)evolution

7

How it all began

1982

Sun M

icros

ystem

s Inc

Foun

ded

1985

Scala

ble P

roce

ssor A

RChitec

ture

sees

the l

ight

1987

32 bi

t SPA

RC V7 s

ystem

1994

64 bi

t SPA

RC V9 a

rchite

cture

1989

SPARC In

terna

tiona

l http

://s

parc.

org

2010

Oracle

Acq

uire

s Sun

Solar

is 11

1997

64 C

PU E

nter

prise

1000

Syste

m

2000

Solar

is m

ost u

sed

unix

in th

e wor

ld

2004 2005

Dual c

ore U

ltra I

VT1

relea

se N

iagar

a

Solar

is 10

2007

M se

ries l

aunc

h Ultr

a VI

8

2010 2011 2012 2013

9

M7M7#Processor# •  32#SPARC#S4#Cores#

–  #Dynamic#Threading#1#to#8#threads#/#core#

•  New#Cache#Organiza8ons#–  Shared#L2#Data#and#Instr#Cache#–  64#MB#Shared#and#Par@@oned#L3#Cache#

•  DDR4#DRAM#–  Upto#2TB#Physical#Mem#/#Processor#–  2x#F3x#beHer#Mem#Bandwidth#over#Last#gen#

•  Applica8on#Accelera8on#–  Real@me#Applica@on#Data#Integrity#–  Concurrent#Memory#Migra@on##–  VA#masking#–  DB#Query#Offload#Engine#

•  SMP#1#to#32#Processors#•  Technology#20nm#

2015

more about this later

10

SPARC FEATURES

11

RAS

Reliability

Availability

Serviceability

12

RAS• Redundant hardware in combination with software (FMA)

Fault Management Architecturepredictive self-healing, offline resources

with increased correctable errors

• Reducing components

• Reduce connectors to ASICS boards

• Instruction level retry and ECC• Hot-swapping / plug-ability

• PCI cards• hard disks• power supplies

13

SPARC Server Availability M6-32 / M5-32: Highest Availability in the SPARC Server Portfolio

RAS

T4#4$

T5#8$T5#4$T5#2$

T4#2$T4#1$ M6#32$$

M5#32$

VM Secure Live Migration Redundant / Hot-swap Fans, Power, Disks

ECC, Predictive Self Healing

Physical Partitions / Redundant Clock / Redundant & Hot-Plug SP Hot Plug PCIe Lane Sparing (L3$/Memory/Interconnect) + Redundant Interconnect

14

64 bit support (1994)•64 bit support in HW and Solaris

15

1991 1992 1994 1995 2001

MIP

S R4

000

DEC

PRI

SM 64

bit

Inte

l ann

ounc

es It

aniu

m IA

64

ULT

RA S

PARC

1997

IBM

PO

WER

64bi

tIn

tel r

elea

ses I

tani

um

large memory support

•64 bit allows large address space•Large pages support by default•SPARC 2GB large pages

% pagesize -a8192 <-- 8K65536 <-- 64K4194304 <-- 4M268435456 <-- 256M2147483648 <-- 2G

16

memory per CPU

x4170 M2 (nehalem)

X3-2 (sandy bridge)

X4-2 (sandy bridge)

3-channel Intel E5

T4 (S3)

T5 (S3)

M6 (S3)

M7 (S4)

0 512 1024 1536 2048

2048GB

1024GB

512GB

512GB

384GB

256GB

256GB

72GB

memory per CPU

17

Solaris 10 (2005)

18

• Supports SPARC and x86

• Scalable kernel

• ZFS

• Dtrace

• Zones lightweight virtualization

• >12 years of premier support

• +3 years of extended support

OVM for SPARC (2007)

19

• Logical Domains

• Hypervisor built in hardware

• Zero overhead

• Strict isolation of CPU, mem, IO

• Flexible vswitch “stuff”

• PCIe dio for HBAs

• Improved vdisk performance (sol11 SRU19)

• Supported for hard partitioning

OVM types/roles• Control domain :

primary domain used to connect to HV and manage resources

• I/O domain :Direct access to phys I/O devices

PCI root complex PCI card Regular non virtualized driversLimited number of I/O domains depending on platform T5-2

vs T5-4• Service domain :

Provides virtual I/O to guestsVirtual networkVirtual Disk

• Guest domain => consumer of virtual I/O 20

SPARC capping

21

SPARC offers lots of options to CAP CPU

❖ PDOMS

❖ LDOMS

❖ Zones

PDOMS

22

• Full hardware and software fault isolation• Divide large server into smaller one• Physical segregation

• CPU• memory• I/O resources

ZonesGlobal zone

“owns” the hardware and controls the resources

Non Global zonesConfigured and run in global zone Exclusive access and Shared network access possible, NO Exclusive access to HBA Share same kernel as global zone Isolated ( != kernel zones)

Supported to CAP CPU (licensing)

23

domain isolation

24

domain isolation

128GB

64GB

128GB

64GB

M9000 with 4 domains

PRODRAC

STAGINGRAC

ACCEPTANCERAC

PRODRAC

STAGINGRAC

ACCEPTANCERAC

ZONES ZONES

25

subserver licensing

68.000

1.221.760

SW license EE,RAC,PART,DIAG, TUNE + 3years supportHW 2*T4-1, 128GB RAM, 3 years support

26

❖ Rule: Oracle SW costs will always outweigh GW

❖ Standard Edition: get as many cores per CPU as possible

❖ Enterprise Edition: get the smallest number of cores with most power per core

dark ages of SPARC

• Single thread Ultra systems EOL after 05

• T1/T2 not really general purpose

• Slow single thread performance

• Mx000 the first successor

27

CMT explained

28

psrinfo0       on-line   since 12/07/2014 23:34:441       on-line   since 12/07/2014 23:34:46

snipped …

60      on-line   since 12/07/2014 23:34:4661      on-line   since 12/07/2014 23:34:4662      on-line   since 12/07/2014 23:34:4663      on-line   since 12/07/2014 23:34:46

perf benchmarkstim

e

0s

125s

250s

375s

500s

threads per core

0,5 1 2 4

x5570 @2.93GHzT4 @2.85GHz29

results - latency

30

• Very simple test w looped SQL

• Intel wins at very low utilization

• 90s vs 119s (30%)

• this is intel turbo-boost effect

• @ 1 thread/core same response time

• SPARC performs better at over-saturation

more benchmarks

http://kevinclosson.wordpress.com/2012/02/06/introducing-slob-the-silly-little-oracle-benchmark/31

• SLOB

• LIO test mode

• there are “modes” for PIO, writes, ...

• One suite of tools for multiple platforms

• Very simple (and open) logic

• Easy to set up, easy to compare

SLOB resultspeak SLOBops/s per core

0

1250

2500

3750

5000

x5570 E5-2640 E5-2690 SPARC64 T4 T5

32

SLOB resultstime to complete 500k SLOB loops

time

0s

400s

800s

1200s

1600s

threads per core

0,25 0,5 1 2 4 8

E5-2640 T4 T533

Solaris 11 (2010)

34

❖ Network stack re-written❖ ipadm/dladm❖ virtual networking

❖ Enhanced support for zones❖ kernel zones❖ immutable zones …❖ Vnics

❖ New IPS packaging system❖ GNU tools by default (feels like linux)❖ Puppet integration❖ Openstack integration❖ ZFS enhancements

❖ deduplication

golden ages of sparc

• T4

• T5

• M7

35

2010 2011 2012 2013

36

M7M7#Processor# •  32#SPARC#S4#Cores#

–  #Dynamic#Threading#1#to#8#threads#/#core#

•  New#Cache#Organiza8ons#–  Shared#L2#Data#and#Instr#Cache#–  64#MB#Shared#and#Par@@oned#L3#Cache#

•  DDR4#DRAM#–  Upto#2TB#Physical#Mem#/#Processor#–  2x#F3x#beHer#Mem#Bandwidth#over#Last#gen#

•  Applica8on#Accelera8on#–  Real@me#Applica@on#Data#Integrity#–  Concurrent#Memory#Migra@on##–  VA#masking#–  DB#Query#Offload#Engine#

•  SMP#1#to#32#Processors#•  Technology#20nm#

2015

37

M7 in a nutshell

Applica'on  Data  Integrity  Virtual  Address  (VA)  Masking  Database  Query  Accelerator  Engine  

In  memory  accelera'on  Value  Range  Comparisons  

Fine  Grain  Memory  Migra'on  On-­‐Chip  Decompression

2TB memory per chip32 cores on 1 chip

SOFTWARE IN SILICON

38

Test drive the M7

39

https://swisdev.oracle.com

SSC line-up

M6-32T5-8T4-440

SSC aka Exadata ++

Same exadata storage cells as in Exadata

Flexibility

2 Types :

A Technical Overview of the Oracle SPARC SuperCluster T4-4

Page 13

Application Domain configuration.

Figure 9. Domains mapped to a half rack system.

Notice how the CPU and memory is configured separately from the Root Complex. When SPARC SuperCluster is installed, allocation granularity is 25% (8 CPU cores and 256GB of memory) per domain. This means the number and types of domains, number of Root Complexes, cores and memory must be decided upon at installation time. The customer can reassign CPU and memory after the installation using a standalone tool, setcoremem. The granularity is 4 CPU cores and 32GB of memory.

Configurations

SPARC SuperCluster is a complete solution that has been designed, tested and integrated to run the Oracle software stack, third party ISV software, as well as any customer developed software. Oracle has tested, tuned, and optimized the best possible hardware and software combinations so customers can take advantage of this integration resulting in a shorter deployment time and better utilization.

Figure 10 shows the components of SPARC SuperCluster T4-4. The numbered items in the figure are identified in the table that immediately follows the figure.

DB Domains -—> IB connect exadata storage cells

App Domains —-> classical SAN based storage

41

Linux on Sparc?"We think Sparc will become clearly the best chip for running Oracle software. At that point we'd be nuts not to move Oracle Enterprise Linux there. We're a ways away, but I think that's definitely going to happen," Larry Ellison in 2010

http://www.pcworld.com/article/212564/article.html 42

Linux on Sparc?

42:linux-uek3-3.8 brost$ git log -SAthena commit aad464a5ef7e5ea745b816852029d830073c6d94 Author: Allen Pais <allen.pais@oracle.com> Date: Wed Mar 6 15:17:59 2013 +0530

correctly recognize SPARC64-X chips The following patch adds support for correctly recognizing SPARC-X chips. cpu : Unknown SUN4V CPU fpu : Unknown SUN4V FPU pmu : Unknown SUN4V PMU Signed-off-by: Katayama Yoshihiro <kata1@jp.fujitsu.com> Signed-off-by: Allen Pais <allen.pais@oracle.com>

42:linux-uek3-3.8 brost$ git log -SSPARC commit a330ea9963a7bce076b92372b35606223997a5b1 Author: Martin K. Petersen <martin.petersen@oracle.com> Date: Wed Feb 26 14:28:25 2014 -0500

ixgbe: Look up MAC address on SPARC systems On SPARC systems first attempt to look up the MAC address in OpenPROM. If that fails resort to using the IDPROM value. Orabug: 18182472 Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com> Acked-by: Jerry Snitselaar <jerry.snitselaar@oracle.com> Signed-off-by: Ethan Zhao <ethan.zhao@oracle.com>

43

SPARC not just SUN/Oracle

44

❖ Fujitsu M10 “Athena”

❖ OpenSparc

❖ study CMT design on FPGA boards!

❖ embedded systems

❖ automotive/industrial

❖ LEON for Space Agencies

45

SPARC    IS

Further Reading

46

M7 Hot Chips

ldoms and Oracle presentation p fierens