Post on 18-Jan-2017
88% (2015)
Unstructured dataworldwide ratio
Each year
Facts:
ESG Digital Archive Market Forecast IDGSource:
+62%
Scalability + PerformanceChallenge:
Object storageSolution: ?
Object storage ?
Object storage ?
Object storage !
Object storage
Advantages But not optimized for
Scalability 100s PBs, low TCO
Ease of access Simple API GET/PUT/DELETE
Resiliency Replication, erasure coding,geodistribution
Heavily updated data
No POSIX-like API
• VMs • SQL DBs
10+ yearsMarket: 1.2 B$ today 4.6 B$ > 2020
Maturity Fast growing
IDCSource:
Gartner, Critical Capabilities for Object Storage
Scalability Efficiency Interoperability Ease of management Performance Resiliency Security
Very high capacity Optimized TCO* Unstructured data Price per GB low
Rather low capacity Very high performanceVMs, SQL DBs Price per GB high
Object store Flash Arrays
Vision: 88% 12%
No future for traditional vendors?
*Total Cost of Ownership
Performance
Tightly integrated application connectors
Quality of Service
< Vision:
Very high capacity Optimized TCO Unstructured data Price per GB low
88%
Object store
Object storage is not only storage. It is a new way to build backend services for applications. As Docker has disrupted sysadmin, we want to disrupt storage management.
WHO
AntiSpam - AntiPhishing - eMail classification 235M MailBoxes in 76 countries
IaaS - PaaS - DaaSNetwork & Security for legacy, SaaS and Web Apps
Lille San Francisco Montréal Hong-Kong Tokyo
Innovative technologiesfor IT infrastructure
100 people
7 8 50fellow co-founders
years experience
million end users
16people
2015launch (June)
2006
Idea & 1st concept
2007
Designdev starts
2009
1st massiveproductionabove 1PB
2006
Idea & 1st concept
2007
Designdev starts
2009
1st massiveproduction above 1PB
2015
OpenIOfork
2012
Opensourced
2014
10+ PBmanaged
Lille
@OpenIO
OpenIO
github.com/open-io
San Francisco
FRANCE
USA
Tokyo
JAPAN
Member of Partners Customers
WE LOVE OPEN SOURCE
• AGPL V3 for core services
• LGPL V3 for libraries andclient APIs
LICENSES
github.com/open-io
WHAT
Grid for apps
ConscienceDirectory with indirections
1. 2. 4.
Grid of nodes
3.
Unique design
…
……
Namespace
Account
Container
Object
Multi tenancy
Flat structure
Track containers, not objects
Directory with indirections1.UNIQUE DESIGN
containers objects
Distributed Hash Tables Consistent Hashing
Single name node
• Good for trillion of objects
• Bad because of rebalancing data when scaling
• Good for few large files
• Bad for numerous small objects
Directory with indirections1.
• Good for trillion of objects
• Good, no need to rebalance data when scaling
Indirection table
Existing technologies OpenIO
UNIQUE DESIGN
Grid of nodes2.UNIQUE DESIGN
• Share nothing architecture
• Scale-out design with no contention
• New nodes are auto discovered
• No gateway in native object mode
• No SPOF, resilient to node failures
1. Collects metrics from the services of each node
2. Computes a score for each service
3. Distributes scores to every nodes and clients
4. On the fly best match making for each request
Conscience3.UNIQUE DESIGN
Realtime load balancing for optimal data placement
The score is computed with a configurable formula usually based on: capacity, io performance, CPU
Storage
CPU/RAMusage
Over the years
> Don’t waste servers Use their free CPU & RAM!
Grid for Apps4.UNIQUE DESIGN
Scality Ceph OpenIO
LICENSE Proprietary Open Source Open Source
ARCHITECTURE RING Crush maps 3-level distributed directory
CONTAINERS External DB RADOS GW only Native with strong consistency
SCALING rebalance rebalanceNo rebalance,immediate availability
LOAD BALANCING
Static, consistent hashing based
Static, consistent hashing based Real time, automatic
Competitors
Hardwareagnostic
0TB > 1000+PB
1. 2.
Never Rebalance
3.
Different
• Simple to use
• Low-latency and linear performance
• Never rebalance when scaling
• Immediate benefits in terms of capacity & performance
• Flexible increments
“True scalability”
0TB > 1000+PB1.DIFFERENT
Consistent hashing
Each node owns a part of the key space
New nodes get parts of the key space from the older ones
Never rebalance2.DIFFERENT
Recalculate key space
Scaling
OpenIO
New nodes automatically discovered and immediately available
Scaling
Immediate benefits in terms of performance from the new nodes
Never rebalance2.DIFFERENT
• Work with heterogeneous hardware
• No vendor lockin
Hardware agnostic3.DIFFERENT
• Hardware is used according to its capacity and performance
• Scores are also based on I/O figures and CPU, not only capacity
Conscience
Interoperability
1. 2. 3.
ConnectorsOpen Source Native APIs
>_Amazon S3® OpenStack
Swift® Optimized native APIs
• Python • Go • C • Java
Command line interface
Native APIs2.INTEROPERABILITY
Storage Email Video
Connectors3.INTEROPERABILITY
• File System (NFS)
• OpenStack SWIFT
• Amazon S3
• Cyrus 3.0
• Zimbra
• Mail Object
• Dovecot
• HTTP connector
• Adaptive streaming connector
• Event based transcoding
Hardware agnostic, True scalability 0 > 1000+PBs
Storage-tier, Conscience auto-optimization, Grid for Apps
Open Source, S3 and SWIFT APIs, Specific App connectors
Human free management, Simple to deploy
Low-latency design, Specific App connectors
Grid of nodes, Replication, Erasure Coding, Geodistribution
Multitenancy, silent corruption checks, WORM for compliance and encryption
Scalability Efficiency Interoperability Ease of management Performance Resiliency Security
Gartner, Critical Capabilities for Object Storage
HOW
Time >
PB >
Deployed Used ISP 40+ M mailboxes
Real life use caseREFERENCES
OIO Core
Standard Support 24/7
Premium Support 24/7
ProfessionalServices
MailConnectors
VideoConnectors
FileConnectors
WebUI
Open SourceSubscriptionService
New release scheme & new release 15.12
• One release for our projects
• Fuse & NFS file connector
• First LTS (long term support) release
• Multiple OpenIO core optimizations
• New python-based CLI
#1
File connector (Fuse & NFS)
• To ease adoption by the market
• Ability to « mount » an account as a filesystem
• Targets performance - close to our core solution
• Files will also appear as read-only objects under the traditional object APIs
#2
Philippe Nicolas
#3
As advisor
Welcome aboard! :-)
OpenIO to quickly become a major player