Oracle NoSQL
-
Upload
oracle-espana -
Category
Documents
-
view
206 -
download
3
description
Transcript of Oracle NoSQL
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL
J. Andrés AraújoPrincipal Sales ConsultantTechnology Sales ConsultingOctober 02, 2014
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 3
Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 5
Program Agenda
Introducing NoSQL
Technical Features Review
Summary
1
2
3
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 7
Program Agenda
Introducing NoSQL
Technical Features Review
Summary
1
2
3
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 13
The NoSQL Landscape
NoSQL
Columnar & Key/Value• Keyspaces, Tables & Records• Key-based access• Limited Transactions• Broad set of use cases
Document• Collections• Document-based access• JSON & XML• “Objects as documents” use cases
Graph• Interconnected graphs• Relatedness-based access• Properties and Graphs, RDF• Specific use cases
• Developer- centric APIs• Flexible schemas• Partitioned/sharded data
• Horizontally scalable• High Availability via Replication• Integrated with Hadoop
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle Database Offer
Embedded+Light weight
Web Scale-out & Real-time events+Schema-less+Relaxed consistency+Built-in availability
Web & Departmental +SQL and NoSQL+Basic security
Mission Critical OLTP and Data Warehousing+SQL, Transactions+High Availability and Security+Multitenant architecture
Oracle Enterprise ManagerOracle Enterprise Manager
Oracle Support
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 15
Choose the RIGHT storage option for the jobHadoop Distributed File System
(HDFS) Oracle NoSQL Database Oracle Database
File System Key-Value Database Relational Database
No inherent structure Simple data structure Complex data structures, rich SQL
High volume writes High volume random reads and writes High volume OLTP with 2-PC
Limited functionality, roll-your-own applications
Simple get/put high speed storage, flex configuration
Security, Backup/Restore, Data life cycle mgmt, XML, etc.
Batch Oriented Real-Time, web-scale specialized applications
General purpose SQL platform, multiple applications, ODBC, JDBC
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 17
Scalable, HA, Key-Value• Simple Key-Value Data Model• Horizontally Scalable• Highly Available • Simple administration• ACID Transactions at scale• Transparent load balancing• Elastic Configuration• Commercial grade software and
support
What is Oracle NoSQL Database
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Big Data Architecture
18
Where Does NoSQL Fit?
Data WarehouseData Reservoir +
Oracle Big Data Connectors
Oracle DataIntegrator
Oracle Advanced Analytics
OracleDatabase
Oracle Spatial & Graph
Oracle NoSQL Database
Cloudera Hadoop
Oracle R Distribution
Oracle Industry Models
OracleGoldenGate
Oracle DataIntegrator
Oracle Event Processing
Oracle Event Processing
Apache Flume
OracleGoldenGate
Oracle Advanced Analytics
Oracle Database
Oracle Spatial & Graph
Oracle Industry Models
Oracle DataIntegrator
Oracle NoSQL Database
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – Commodity Servers
NoSQL DB Driver
Processing ApplicationData
Connector
Servers
Big Data Acquisition
Real Time Access
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL Database – Engineered Systems
NoSQL DB Driver
Processing Application
HDFS, Hadoop, CDH
Map ReduceORCH - Stats
Map ReduceHive - Activities
Map ReducePig - Sessionize
Big Data Acquisition
Batch Processing
Big Data Appliance
Big Data Acquisition
Real Time Access
DataConnector
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 23
Program Agenda
Introducing NoSQL
Technical Features Review
Summary
1
2
3
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 24
Release3.0
Features• Key-value, JSON & RDF data• Large Object API• BASE & ACID Transactions• Data Center Support• Online Rolling Upgrade• Online Cluster Management• Table data model• Secondary Indices• Secondary Zones (Data Centers)• Security
Oracle NoSQL DatabaseScalable, Highly Available, Key-Value Database
Application
Storage NodesDatacenter B
Storage NodesDatacenter A
Application
NoSQL DB Driver
Application
NoSQL DB Driver
Application
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 25
Scalability ArchitectureApplications View
Elastic Shards (split, add, contract)
Store
Shard
M
Shard
M
R
Shard
M
R R
Application
NoSQL Driver
R R
R
Writes to elected node
Reads from any node in system
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Features:Flexible Data Model
• Simple data model – key-value pair (major+minor-key paradigm)• Simple operations – read/insert/update/delete, RMW support• Scope of transaction – records within a major key, single API call• Unordered scan of all data (non-transactional)
Key-Value Pairs
userid
addresssubscriptions
email idphone #expiration date
Major key:
Minor key:
Value:
Strings
Byte Array
Value Options: Key-Value JSON RDF Triples Tables/Rows
picture
.jpg
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Store JSON documents
{ "firstName": "John", "lastName": "Smith", “age": 25, “ address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": 10021 }, "phoneNumbers": [ { "type": "home", "number": "212 555-1234" }, { "type": "fax", "number": "646 555-4567" } ] }
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
What is Avro?
• Serialization protocol • Apache Open Source• Developed by Doug Cutting from Cloudera• C++ and Java language bindings• “Schemas” defined using JSON
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL – Why AVRO?
• Compact, highly efficient serialization• Synergy with Hadoop• Supports serialization from and to JSON strings• Bindings from serialized formats to language constructs• Easy to use mechanism for schema evolution• Schema definition tracked with schema ID of data writer• Schema versions can be opaque to readers
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Avro Schemas
• Defined using a JSON document• Supports records, enums, unions, maps, arrays, fixed• Supports field level aliasing• Supports a sort order for primitive and complex types• Supports schema evolution through reader/writer mechanism
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Features:Fexible Data Model
• Benefits– Lower barrier to adoption, shorter time to market– Simplified application modeling– Uses familiar table concepts
• Features– Layered on top of distributed key-value model– Compatible with Release 2.0 JSON schemas– Supports table evolution, retains flexible client access
• Sets foundation for future capabilities
NoSQL DB Table Model (New 3.0)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Features:Fexible Data Model
table create -name Users
add-field -name userid -type integer
add-field -name lname -type string
add-field -name fname -type string
add-field -name email -type string
primary-key -field userid
shard-key -field userid
exit
plan add-table -name Users -wait
Simple Table Example (New 3.0)
Can be specified as a JSON string
Must be proper subset of primary-keyBy default shard-key == primary-key
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Features:Fexible Data Model Simple Table Example (New 3.0)
userId lname fname email
Table
Shard Key
Users
ValuePrimary Key
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Features:Fexible Data Model
table create -name Users
add-field -name userid -type integer
add-field -name lname -type string
add-field -name fname -type string
add-field -name email -type string
primary-key -field userid
exit
plan add-table -name Users –wait
table create -name Users.Folders
add-field -name foldername -type string
add-field -name msgcount -type integer
add-field -name favorite –type boolean
-default 'F'
primary-key -field foldername
exit
plan add-table -name Users.Folders -wait
Nested Table Example (New 3.0)
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Features:Fexible Data Model Nested Table Example (New 3.0)
UserId lname fname email
TableUsers
Users.Folders
[Value]Primary Key
…
[Value]
UserId Foldername msgcount favorite
Primary KeyShard Key
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 37
Auto-Sharding• Provides linear scale-out of write
ops/sec• No need to develop sharding logic in
Application• Intelligent Driver – aware of hash
function and shard location• Hash function to map a key to a
partition• Each partition is routed to a single
shard
Features:Sharding
Shard 0
Rep Node
Master
Rep Node
Replica
Rep Node
Replica
Shard 11
Rep Node
Master
Rep Node
Replica
Rep Node
Replica
…
Partitions
Hash Fn
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 38
High Availability• Master-Slave replication
• Supports Heterogeneous platform hardware/OS/JVM
• Provides linear scale out for read ops/sec
Features:Replication
Shard 0
Rep Node Master
Rep Node Replica
Rep Node Replica
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 39
Automatic Failover• Automatic election of new Master
• Rejoining nodes automatically synchronize with the Master
• Isolated nodes can still service reads
• All nodes are symmetric
Features:Failover
Replication factor = 5
Rep Node Master
Rep Node Replica
Rep Node Replica
Rep Node Replica
Rep Node Replica
New Master
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 40
Automated Resource Planning• Storage nodes have indication of
“capacity”• System allocates replicas per storage
node• Intelligent Master/Replica load
balancing• Ensures distribution of replicas• Efficient use of system resources• Reduces operator-caused configuration
errors
Features:Smart Topology Management
Application
Smart Topology Driver
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 41
Smart topology• Increase Data Capacity
– Add more storage nodes– New shards automatically created
• Increase Data Throughput– More shards = better write throughput– More replicas/shard = better read throughput
Features:Elastic Cluster Expansion
NoSQL DB Driver
Application
Master
Replica
Replica
StorageNode StorageNode StorageNode
Shard-1
Master
Replica
Replica
Shard-2
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 42
Improved Performance• Supports heterogeneous storage
topology• Replicas move from over-utilized to
under-utilized storage nodes• Number of shards and replication
factor remain unchanged
Rebalance an Unbalanced StoreStorage Node 1 Storage Node 2 Storage Node 3
Represents a partition
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 43
Greater Flexibility• Configurable Durability per
operation• Configurable Consistency per
operation• ACID by default• Transaction scope is single API call• Records share same major key• Multiple operations supported
Features:Configurable CAP
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 44
Java, C, Jython, Jruby, etc.• Simple CRUD operations
– Minimize client/server round trips
• Iteration operations– Returns multiple results in a single API
call– Get full records or just the keys
• Multi and Sub-Key operations– Multiple operations and/or Multiple
records in a single transaction
Features:Easy to use API
JSON
putIfPresent
PUTputIfAbsent
multiGet
deleteAVROgetLOB
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 45
Parallel Streaming Interface• Efficient storage and retrieval of
large objects• Client side streaming interface for
low memory consumption• Server side splitting and
distribution of object chunks across nodes for better read/write latency
• Automatic partial LOB detection
Features:Large Objects Support
Large Object
NoS
QL
DB
Driv
er
Appl
icati
on Shard 2
Shard N
Shard 1
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 46
Centralized Management• Configuration via CLI commands• Web-based console option• Manages and Monitors
– Configuration changes– Load: Performance: Events:– Alerts: Failure, poor performance, etc.
• SNMP and JMX interfaces
Features:Simple Administration
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 47
Easy Prototype Development• Topology Planning• JSON Schema load• Put/Get/Delete data• Aggregates
– Average, Count, Sum
• File based data loading
Features:Data and Management CLI
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 48
Availability Zones• Flexible configuration• Metro-Local Quorum
– Low latency writes, HA
• Secondary Read-Only Zones– Analytic workloads– Oracle Reporting– Asynchronous replication
• Topology Aware Client Driver• Provides business continuity and
distributed workload management
Features:Data Center Support (New 3.0)
DC1 DC2 DC3
Metropolitan Zones
ReportsBatch Analytics
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 50
What’s the Big Deal• We did do it!• Admin commands available to
describe safe upgrade order• Scripted available hands-free
upgrade experience• Read/Write availability throughout
the upgrade process
Features:Online Rolling Upgrades
72 (24x3) 144 (48x3) 216 (72x3)0
2.5
5
7.5
10
12.5
15
17.5
Online Rolling Upgrade
Total Nodes (Shards x Rep. Factor)
Tim
e to
Upg
rade
(min
)
Ever tried to upgrade a 200 node system while it’s active?
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 51
Standard Protocol Support• SNMP (Enterprise Edition Only)
– MIBs in SNMPv2 format– Storage node SNMP agent – admin and rep
nodes– Performance metrics – Via polling– Service change events – Via traps or polling– Notifications (Traps) – Latency and throughput
limits
• Java JMX– Performance Metrics – Via Suscriptions– Service changes events – Via subscriptions– Notifications – Via subscriptions
Features:System Monitoring
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 52
No Data Loss!• Backup is done by creating a
snapshot of the store at each node
• Restore from the snapshot– Load utility to restore snapshot into
any topology– Direct Recovery to recover to the exact
topology of the snapshot
Backup & Restore
Oracle Confidential – Internal/Restricted/Highly Restricted 53Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Integration
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration - RDF
RDF description
GeraldAlex
Steve
NoSQL
knows knows
has skill
has skill
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration - Avro
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration - Hadoop
HDFS
HDFS Files
REDUCE
MAP
MAP
MAP
MAP
MAP
MAP
REDUCE
REDUCE
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration – Oracle RDBMS
Externaltable
SELECT name FROM unstruct_peopleSQL engine
nosql_streampreprocessor
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration – Oracle Coherence
NoSQL DB DriverTB
Mem
ory
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL DB integration – OEP
NoSQL DBDriver
NoSQL DBDriver
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 68
Program Agenda
Introducing NoSQL
Technical Features Review
Summary
1
2
3
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
PointFlexibility PointWeb-Scale
Transaction Processing
High velocity, volume, variety, low information density data
capture
Web browsing, Shopping Carts, CDR processing, Sensor data
capture
Web-Scale Personalization
Guaranteed low latency lookups for end-customers
Advertising, Product Recommendations, Online
Catalogs, Social Media, Profile Management, Personalization
Real-Time Event Processing
Real time events trigger rule that perform low latency lookups
Medical Monitoring, Factory Automation, Oil & Gas, Geo-
location
Oracle NoSQL Database Use Case Summary
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
NoSQL or Relational?
?
NoSQL DBMSSchema private to application
Collections of data objects
Basic Availability, Soft-state, Eventual consistency
Schema shared by multiple applications
Atomic, Consistent, Isolated, Durable
Inter-related collections of data objects
Relational DBMS
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Key Value Store Document / Relational Store
Data Collections Integrated Documents and Relational
ACID & Eventual Consistency Joins and Transactions across Documents
Simple Key-Value and Table Model SQL, Reporting, Analytics
Sharded Trusted Technology Foundation: RAC, Multitenant, Security, Availability, In-Memory
Oracle for “NoSQL”
Oracle Enterprise Manager and Support
JSON
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle Database 12.1.0.2Flexible Schema development
JSON
JSONVia RESTful service
Via Native APIsData persisted in database
In JSON
SQL
Data analyzed via SQL
Oracle Database 12c
JSON
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL DB Resources
• NoSQL DB Use Cases, White Papers, Data Sheets, Benchmarkshttp://www.oracle.com/technetwork/products/nosqldb/overview/index.html
• NoSQL DB Documentationhttp://www.oracle.com/technetwork/products/nosqldb/documentation/index.html
• NoSQL DB Downloadshttp://www.oracle.com/technetwork/products/nosqldb/downloads/index.html
• NoSQL DB OTN Forumhttp://forums.oracle.com/forums/forum.jspa?forumID=1388
NoSQL DB version 2.0 Featureshttp://bit.ly/UKn5Sc
• OU Training Classeshttp://bit.ly/V5qbmY
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle NoSQL DB Resources
• Great Application Development article on OTN: http://www.oracle.com/technetwork/articles/bigdata/oracle-nosqldb-appdev-1891870.html
75Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Q & A
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 76