G2M Research Multi-Vendor Webinar Scale-Out Flash Storage...

RESEARCH

G2M Research Multi-Vendor Webinar Scale-Out Flash Storage (SOFS) – The Next-Generation Storage Solution?May 7, 2019

SPONSORED BY:

2 © 2019 G2M Communications, Inc.. All rights reserved.RESEARCH

Webinar Agenda

9:00-9:05 Ground Rules and Webinar Topic Introduction(G2M Research)

9:06-9:29 Sponsoring Vendor presentations on topic (8 minute each) 9:30-9:37 Key Question 1 (2-minute question; 2 minutes response per

vendor) 9:38-9:39 Audience Survey 1 (2 minutes) 9:40-9:47 Key Question 2 (2-minute question; 3 minutes response per

vendor) 9:48-9:49 Audience Survey 2 (2 minutes) 9:50-9:57 Key Question 3 (2-minute question; 3 minutes response per

vendor)9:58-9:59 Audience Survey 3 (2 minutes)10:00-10:07 Key Question 4 (2-minute question; 3 minutes response per

vendor)10:08-10:18 Audience Q&A (11 minutes) 10:19-10:20 Wrap-Up

RESEARCH

G2M Research Introductionand Ground RulesMike HeumannManaging Partner, G2M Research


Josh GoldenharVP Products(www.excelero.com)

Panelists

Host/Emcee: Mike HeumannManaging PartnerG2M Researchwww.g2minc.com

Joel DedrickVP/GM, Networked Storage SW(www.taec.toshiba.com)

Kam EshghiVP of Strategy/Business Development(www.lightbitslabs.com)

http://www.excelero.com/

http://www.g2minc.com/

http://www.taec.toshiba.com/

http://www.lightbitslabs.com/


What is a Scale-Out Flash Storage (SOFS) Solution?

Scale-Out Flash Storage (SOFS) solutions consist of the combination of an SOFS software package and servers (typically) containing NVMe™ storage devices.

SOFS evolved from Software-Defined Storage (SDS), but scales out to hundreds or even thousands of nodes.

SOFS also borrows heavily from the DAS-centric storage architectures utilized by hyperscale companies (AWS, Google, Azure, Facebook, and the BATs).

Utilizing NVMe over Fabrics as a backbone, SOFS solutions can provide remote file system access across large number of servers and storage appliances with SSDs, but with local-like performance and latency.


How Do SOFS Solutions Differ from All-Flash Arrays?

Ability to Combine Multiple Heterogeneous Servers and SSDs into a Single Storage Pool

Scale of Deployment

Multiple System Components That Need to be Integrated for SOFS vs “One Throat to Choke”

Complexity of Support


Storage Approach: Important Questions for Users

What are your use cases for your storage systems?

Is (maybe thousands) of “DIY” servers or storage appliances + SOFS SW the right choice?

How do you buy and support storageappliances plus SOFS SW deployments?

RESEARCH

ToshibaJoel DedrickVice President/General Manager,Networked Storage Softwarewww.taec.toshiba.com

http://www.taec.toshiba.com/

Toshiba Memory America, Inc.

Scale-Out Flash StorageTrickle-down IT and the journey to the cloud.

Joel Dedrick – VP & GM, Networked Storage Software

9

Scale-Out Flash Storage

Why?• What problem does this solve? Who really needs it?

Why Now?• What’s wrong with {iSCSI, iSER, Fibre Channel, FCOE, FCIP, …}• What’s different this time?

Why Toshiba Memory?• Don’t you guys make laptops?

10

Cloud Infrastructure 1.0

Uniform -- Everything’s a “node”• “Shared-nothing”

(Almost) everything is a “service”• Arbitrary scale, but high latency

Exception: Fast block storage (i.e., flash)• Mostly direct-attached SSDs - fast, but ephemeral

12

Problem 1 (Today)

13

Whatever size SSD you choose will be wrong.

Trickle-Down IT

15

Important advances now come mostly from Google, Azure, …

and diffuse downhill from there

Tier-2, Cloud-Native:• Create unique or high leverage pieces, but can’t do it all.

Enterprise:• Adopt when OEMs offer turnkey, supported versions

IT S

pend

Tier 1Hyperscale

Tier 2CloudNative

Enterprise

SoHo

2-5yrs*

5-10yrs*

10-20yrs* Customer rank

* Toshiba Memory estimates

Over Fabrics

Cloud Infrastructure 2.0 – Fast Block Storage as a Service

16

Disaggregate & Share• Every node gets exactly the needed

capacity

Virtualize & Customize• Per-application quotas, QoS guarantees,

access rights, …

Enabled by NVMe-oF™, CLOS, 100GE

• Originally driven by T1 Hyperscale• Now available for the rest of usNVMeoF targetNVMe-oF

Why Now?

Legacy vs. CLOS (“Flat”) Network Architectures (conceptual)

18

Hierarchical• Heavily oversubscribed• One path• Locality matters

CLOS• Little oversubscription• Many paths• Locality irrelevant

Lots more switch ports, optical transceivers, cables, power…

“Flat” Networks in Hyperscale are A Done Deal

19

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

“Firehose 1.1”1Gb nodes

20k @ 2:1 OS

“Watchtower”Nx1Gb nodes40k @ 2Gb,Nonblocking

“Saturn”Nx10Gb nodes10k @ 1x10Gb,

Nonblocking

“Jupiter”Nx10/ Nx40Gb nodes

32.5k @ 40Gb,Nonblocking

Last “4-post”

“Data Center Fabric”Nx10Gb nodes74k @ 1x10Gb,

4:1 OS, scalable to NB

Last “4-post”

“Flat Network Storage”

Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google’s Datacenter NetworkSIGCOMM ’15 August 17-21, 2015, London, United Kingdom.

Introducing Data Center Fabric, the next generation Facebook data center networkhttps://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/

Facebook Fabric Networking Deconstructedhttp://firstclassfunc.com/facebook-fabric-networking

https://azure.microsoft.com/en-us/blog/windows-azures-flat-network-storage-and-2012-scalability-targets/

Why Spend the Money?Optimal Workload Blending

Same workload with 30%* fewer machines• Prerequisite: Any job on any node

In turn, that requires• Everything equally “network close”• Networked flash at the speed of local SSDs

20* Large-scale cluster management at Google with BorgEuroSys’15, April 21–24, 2015

Large-scale cluster management at Google with BorgEurosys ‘15

* Range 0.. 60%

What’s Wrong With…{insert your favorite storage protocol}?

There are lots of block storage interfaces/transportsSATA, SAS, FibreChannel, iSCSI, iSER, FCoE, SSA, PCIe®, …

…but only two command languages!

NVMe is a Once-in-a-Lifetime EventNVMe-oF™ is its native transport

21

SCSI (1982), Invented for hard disks*

NVMe™ (2012)First and only command set specifically for flash

Cloud Datacenter IOPS Share by Command Set* (conceptual)

22

1980 1990 2000 2010 2020 2030

SCSINVMe™

SCSI Std.

NVMe Std.

“HDD-like” SSDs

NVMe SSDNVMe over Fabrics Std.

Fibre Channel Std.SCSI HDD * “Share of IOPS” is unrelated to market share for any

product or product segment, so this graph should not be construed to infer anything about rates of adoption.This graph is conceptual only.

Why Toshiba Memory?

About Toshiba Memory Corp.

Privately held• Spun out from Toshiba

Inventor of NAND flash• ~1/3 of world output *

Products• Flash• SSDs• Storage Node Software

24

Yokkaichi annual wafer output

(stacked)

Eiffel Tower

Burj Khalifa

BiCS 3D Flash

* Along with our partners

NVMe and NVMe-oF are trademarks of NVM Express, Inc.

PCIe is a registered trademark of PCI-SIG.

All company names, product names and service names may be trademarks of their respective companies.

RESEARCH

ExceleroJosh GoldenharVP of Productswww.excelero.com

http://www.excelero.com/

Excelero NVMeshLowest-latency, most scalable Distributed Block Storage

Lowest OverheadLocal Flash Latency across the Network

100% Software-definedUse any Hardware, no special accelerator needed

Network Protocol ChoiceRDMA, TCP – choose what’s best for you

Block StorageUse any File System

Software-defined Block Storage

29

• NVIDIA DGX enables Instadeep to run AI workloads for customers across industries such as Medical, Energy, Manufacturing …

• The DGX is used by multiple scientists who run workloads for many different customers.

• Local DGX Storage is too limited: the DGX only has 4TB of local storage while customers’ workloads require larger data sets.

• Traditional external storage requires moving data to/from the DGX, which interrupts the workflow and is time consuming.

• DGX systems connected over Mellanox IB fabrics to a Boston NVMe server powered by Excelero NVMesh provides access to hundreds of TB’s of external high-performance storage.

• DGX systems fully utilized to make perfect use of their resources and speed up InstaDeep customers deep learning workload

Customer Success

AI as a Service – data-center-scale pool of NVMestorage increases ROI for GPU’s

Use Case

Challenges

Solution

Feeding unlimited streams of data to GPU’s with local performance

30

• Any-K production & post-production

• Streaming, editing and reel presentation to simultaneously active workstations

• 4K & 8K video is changing requirements for storage performance and capacity

• Workstations need high BW & low latency performance for streaming, editing and reel presentation

• Customers need future-proof solutions that will support 10k

• With just 6 standard servers powered by 3D NAND, NVMesh enabled 4K streaming @ 60fps to 50+ workstations concurrently

• The solution gives twice the performance of competing solutions at 50% of the cost and requires a much lower data center footprint

Customer Success

Any-K Storage

Use Case

Challenges

Solution

4K streaming @ 60fps to 50+ workstations concurrently

31

• Large-scale modeling, simulation, analysis and visualization

• Visualizes supercomputer simulation data on 100s of compute nodes

• Finish check pointing faster and start running the job.

• Achieve performance of 250GB/Sec at the lowest price

• NVMesh by Excelero enables SciNet to create a petabyte-scale unified pool of high-performance flash distributed retaining the speeds and latencies of directly-attached media

Customer Success

Pooling NVMe Within GPFS NSDs enables new Science use cases

Use Case

Challenges

Solution

80 pooled NVMe devices give 250GB/s of throughput and 20M random 4k IOPS

32

• Cluster performance is a key priority for large Oracle environments, especially for customers with tables that size up to billions rows and serve thousands of users driving analytics against that data

• Traditional AFA storage solutions give latencies of slightly less than 1 millisecond; customers often get thousands of queries a day and some of these can take over an hour to run.

• NVMesh offers latencies of single digit microseconds. This significantly speeds up database scans and queries

• Running Oracle database on NVMesh enables customers to reduce the number of licenses needed for their clusters.

Customer Success

High throughput & low latency for Oracle databases

Use Case

Challenge

Solution

62% reduction in Oracle licensing costs doing the same amount of processing

Excelero NVMesh on 2U, 4-node Server Chassis - Performance33

• Volumes span 4 servers for higher aggregate performance

• RAID 10 and MeshProtect 60 (6+2) volumes can survive the failure of any node in the chassis

• Any single volume, or volumes in aggregate can achieve up to 80GB/s of bandwidth

• Up to 19M 4K IOPs

• No special hardware for protected 4K writes in as little as 30µs

• Because Clients distribute MeshProtect, no special hardware needed

2U, 24 Drives80GB/s Bandwidth

19M 4K IOPs

RESEARCH

Lightbits LabsKam EshghiVP of Strategy/Business Developmentwww.lightbitlabs.com

http://www.lightbitlabs.com/

Hyperscale Requirements

35

GrowthMore users

More performanceMore capacityMore services

EfficiencyHigher utilization

Easy to scaleStandard serversSoftware-defined

Yesterday’s Cloud Infrastructure

36

Pros● High performance● Easy to deploy at small scale

Cons● Constrained ratio of Compute-to-Storage

○ Mismatched App Needs● Stranded capacity/performance

○ Inefficient, low utilization● Large data movement & complicated storage

scheduler ○ Difficult to scale

• Maximize utilization• Scale storage &

compute independently• Easy to maintain &

upgrade • Composable

infrastructure

37

Today’s Cloud InfrastructureFrom direct-attached to disaggregation

Lightbits in a Nutshell

• Maximize resource utilization and improve operational efficiency

• Scale with simple & efficient TCP/IP, with end-to-end NVMe

• Accelerate data services in software & hardware

• Reduce cost by improving flash endurance• Build a cloud-optimized solution based

on standard servers

No changes to network infrastructure

Low, consistent latency

Hardware-accelerated data services

Seamlessly move your infrastructure from direct-attached SSDs to a remote low-latency pool of NVMe SSDs

38

Global FTL

NVMe/TCP

Scale-Out Flash Storage with LightbitsOptional hardware acceleration for SSD management and data services

High performance, low latency Global Flash Translation Layer with data services

High performance, low latency

NVMe/TCP target Standard TCP/IP

Network(no RDMA required)

Standard NVMe/TCP client

driver

ApplicationCassandra,

MongoDB ...

OS (Linux) with NVMe/TCP

Client 1


MongoDB ...


Client 2


MongoDB ...


Client N

Data Center Network Infrastructure

SSD

Storage server 1

SSD

Storage server M

NVMe/TCP target Global FTL with Rich Data Services

NVMe/TCP target Global FTL with Rich Data Services

39

LightOS Software Defined Storage SW layer that virtualizes pool of SSDs

● Optimized NVMe/TCP target● Thin Provisioning - scale as-you-grow● Erasure Coding - line rate Data Protection● Compression - increases write performance &

endurance● QoS - consistent latency and IOPS● Fast Write Acknowledge ● Use lower endurance SSDs (e.g., QLC)

Global FTL

NVMe/TCP

40

41

LightField Hardware AccelerationStorage acceleration for optimized performance and latency

• Data reduction • Data protection• NVMe/TCP acceleration• Global FTL acceleration

42

Our Vision

DAS+ High Performance- Underutilized infrastructure

+ DAS performance+ Easy to Deploy & Scale+ High Utilization+ Rich Storage Services+ TCO reductionRack-scale Storage

+ High Performance- Limited scale

Traditional SAN/HCI+ High utilization+ Advanced services- Low performance

43

Write Only(Compression 2:1)

Read Only(Compression 2:1)

30 136

58 161

167 266

Latency (Random 4k)with wirespeed compression & EC

Summary - Lightbits DifferentiationSeamlessly move your infrastructure from direct-attached SSDs to a

remote low-latency pool of NVMe SSDs

44

Bring Your Own Hardware

● Software-defined solution

● Runs on commodity storage server HW

● Standard NVMe SSDs

Don’t touch clients(Target side solution)

● No proprietary client software

● Standard NVMe/TCP client driver

No change in network(NVMe/TCP)

● Use vanilla TCP/IP network infrastructure -ubiquitous, simple & efficient

● Run on standard Ethernet NICs

Global FTL

● Thin Provisioning● Compression● RAID/EC● QoS services● Enhance Endurance

and Latency● QLC ready● Optional storage

acceleration card

RESEARCH

Panel Discussion


Panel Question #1

What do you see as the leading use cases today for SOFS solutions?– Toshiba– Lightbits Labs– Excelero


Audience Survey Question #1

Does SOFS look like a solution that makes sense for your organization? (check one; xx responses):

• Definitely; we have several of the SOFS use cases: 11%• Probably; we have one/a few of the SOFS use cases: 16%• Maybe; would likely evaluate SOFS as a solution: 26%• Probably not; we would likely continue to acquire

array-based storage solutions: 16%• Don’t know/haven’t studied SOFS as a solution: 32%


Panel Question #2

Should SOFS solutions support scale-out (more nodes), scale-up (more storage per node), or both to be effective?– Lightbits Labs– Excelero– Toshiba



Which of SOFS’s advantages/disadvantages are important for your use cases (check one answer for each question; xx responses):

Question Very Important

Somewhat Important

Not Important Don’t Know No Opinion

Advantage: Ability to deploy across heterogeneous hardware 65% 10% 5% 10% 10%

Advantage: Ability to scale to hundreds or thousands of nodes 45% 35% 0% % 10%

Advantage: Ability to share local server storage to remote nodes 35% 40% 0% % 15%

Disadvantage: Need to integrate solution 25% 35% 15% % 10%

Disadvantage: More complex support 30% 40% 15% % 10%


Panel Question #3

When designing scale-out storage solutions, it’s key to avoid bottlenecks anywhere in the architecture (both the network and storage). What steps should be taken to address congestion on Ethernet networks that now have to contend with the strain of storage traffic as well as network traffic?– Excelero– Toshiba– Lightbits Labs



How do you see SOFS fitting into the overall storage solutions environment? (select one; xx responses):

• It is a game-changer – we see it as the leading wayto deploy storage in the near-future: 27%

• It is very important – we see it eventually replacingclassical storage arrays over time: 40%

• It is a useful “tool in the toolbelt” that give ourorganization new storage options: 33%

• It is interesting, but is likely a “niche” technology: 0%• We don’t see it as a relevant technology: 0%


Panel Question #4

What new application deployment opportunities do SOFS solutions open up for users?– Lightbits Labs– Excelero– Toshiba


Audience Q&A

RESEARCH

Thank You For Attending

RESEARCH

G2M Research Multi-Vendor Webinar Scale-Out Flash Storage...

Documents

Transcript of G2M Research Multi-Vendor Webinar Scale-Out Flash Storage...