The Kumofs Project and MessagePack-RPC
-
Upload
sadayuki-furuhashi -
Category
Technology
-
view
5.138 -
download
0
Transcript of The Kumofs Project and MessagePack-RPC
The Kumofs Project
and MessagePack-RPCSadayuki Furuhashi
About me
• University of Tsukuba, Japan
• Twitter: @frsyuki_ha
• Blog: http://frsyuki.wordpress.com/
What’s kumofs?
• Distributed key-value store
• “Tokyo Cabinet + Scalability”
• Extreme single node performance (chart)
• Linear scalability of read/write (chart)
• Dynamic rebalance + Consistency control
“key-value store”
• Set(key, value)
• Delete(key)
• value, unique = Get(key)
• CAS(key, unique, value)> if the key is not modified, then set the value.otherwise it fails.
CAS - Compare and Swap
db.atomic do |array| # Get current value. element = array.pop # Modify the value. array # Put the modified value (<--CAS)end # Retry if the value is updated while modifying.
Feature ComparisionData Model Atomic Ops dynamic
rebalance
kumofs
Voldemort
Cassandra
Tokyo Tyrant
Redis
key-value CAS Supported
key-value - Supported
partitioned sorted map table ops Supported
K-V, sorted mapor table
queue ops, etc. -
queue, etc. queue ops, etc. -
Feature ComparisionConsistency Control on
Dynamic Rebalancing
kumofs
Voldemort
Cassandra
Tokyo Tyrant
Redis
on write (original algorithm)
on read (Vector Clock, R+W>N)
on read (Vector Clock, R+W>N)
N/A
N/A
“Tokyo Cabinet + Scalability”
• Extreme single node performance• Dynamic Rebalancing with Consistency Control• CAS support• Linear scalability of read/write
Archtectureof
kumofs
Manager
Gateway
kumo-manager:Manages kumo-servers
Archtecture of kumofs
kumo-server:Replicates and stores data
kumo-gateway:Relays requests from applications to kumo-servers.
Server
Application
ServerManager
Gateway
Manager
Duplicated(HA)
Server
Server
Server
Server
Server
Application
GatewayApplication
Gateway
Replication
Tokyo Cabinet
Server A
Server B
Server C
Server D
Consistent Hashing
hash(ServerA.id)
hash(ServerB.id)
hash(ServerC.id)
hash(ServerD.id)
hash(key1)
Server A
Server B
Server C
Server D
Consistent Hashing
Server A
Server B
Server C
Server D
coordinated by Server B
Server A
Server B
Server C
Server D
coordinated by Server B
Server CServer D
Server A
Server A
Server B
Server C
Server D
set
Replicate
Replication mechanism
Server A
Server B
Server C
Server D
getHigh-availability
Server A
Server C
Server D
getHigh-availability
Server A
Server C
Server D
getHigh-availability
Demo
Technologies of kumofs
• Dynamic rebalancing with Consistency Control- Double-hash-space Algorithm
• Fully multithreaded event-driven I/O- Wavy I/O Archtecture
• Cross-language communication- MessagePack
Server A
Server B
Server C
Server D
coordinated by Server B
Server CServer D
Server A
Adding nodes dynamically
Server A
Server B
Server C
Server D
coordinated by Server B
Server C
Server A
Server E
Move data
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
Data is moving...
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
setget
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
setget
Adding nodes dynamically
Server A
Server B
Server C
Server D
Server E
setget
Adding nodes dynamically
getset
Consistent Hashing for Reading
Consistent Hashing for Writing
Dynamic rebalacing algorithm of kumofs(double-hash-space)
• Fully multithreaded event-driven I/O
Wainting forevents...
Waiting formutex lock
※ An event arrived ※ More event arrivedThreads
Proceeds event handlersin parallel
Proceeds event handler
The Wavy I/O Archtecture
The Wavy I/O Archtecture
Socket A
Data1
ProceedsData 1
※ Socket-binded I/O (thread-based I/O) ※ Wavy I/O Archtecture
ProceedsData 3
parallel
Proceeds in parallelserializedProceedsData 2
Data2
Data3
Socket B Socket A
Data1
Data2
Data3
Socket B
Cronss-language communication
• Cross-language object serialization library> Management tools := Ruby> kumo-servers := C++> “MessagePack”
• “MessagePack” + “Wavy” is now known as “MessagePack-RPC”
• What about HBASE-794? (New RPC)
What’s MessagePack?
• Efficient serialization library
• Rich data structures - compatible with JSON
• Dynamic typing, Self-describing
• Remote Procedure Call (RPC)
• Synchronous, Asynchronous and Callback style
• Concurrent calls with multiple servers
• Event-driven I/O
• Interface Definition Language (IDL) - compatible with Thrift
Efficient Serialization
1. Compact 2. Fast
・ Zero-copy (C++)
・ Stream deserialization
・ Binary-based format
・ Embed type information
Format of MessagePack
Fixed length types Variable length types
・ Raw bytes
・ Array
・ Map
・ Integer
・ Floating point
・ Boolean
・ Nil
type value type body...length
Type information
Format of MessagePack
JSON MessagePack
null
Integer
Array
String
Map
null c0
10 0a
[20] 91 14
”30” a2 ‘3’ ‘0’
{“40”:null} 81 a2 ‘4’ ‘0’ c0
Format of MessagePack
JSON MessagePack
null
Integer
Array
String
Map
null c0
10 0a
[20] 91 14
”30” a2 ‘3’ ‘0’
{“40”:null} 81 a2 ‘4’ ‘0’ c0
4 bytes 1 byte
2 bytes 1 byte
4 bytes 2 bytes
4 bytes 3 bytes
11 bytes 5 bytes
Type information0x000xc20xc30xca0xcb0xcc0xcd0xce0xcf0xdf...
nil false true float double uint8 uint16 uint32 uint64 int8 ...
Types
0xc00xe0
Type information
Embed value
Positive FixNum
Negative FixNum
FixMapFixArray
FixRaw
0x00
0xc0
0x800x900xa0
0xe0
0x000xc20xc30xca0xcb0xcc0xcd0xce0xcf0xdf...
nil false true float double uint8 uint16 uint32 uint64 int8 ...
Type information Types
Zero-copy serialization
Zero-copy deserialization
Performance
It measured the elapsed time of serializing and deserializing 200,000 target objects. The target object consists of the three integers and 512 bytes string.
Remote Procedure Call
MessagePack-RPC
Inter-process messaging library forclients, servers and cluster applications.
Inter-process messaging library forclients, servers and cluster applications.
Remote Procedure Call
MessagePack-RPC
Concept of Future
Multithreadedevent-driven I/O
Communicates with multiple servers concurrently
Synchronous call
require 'msgpack/rpc'
client = MessagePack::RPC::Client.new(host, port)
result = client.call(:method, arg1, arg2)
Asynchronous callrequire 'msgpack/rpc'
client = MessagePack::RPC::Client.new(host, port)
future1 = client.call_async(:methodA, arg1, arg2)future2 = client.call_async(:methodB, arg1, arg2)
result1 = future1.getresult2 = future2.get
Callbackrequire 'msgpack/rpc'
client = MessagePack::RPC::Client.new(host, port)
client.callback(:method, arg, arg2) do |future| result = future.getend
client.join
Concurrent calls with multiple servers
require 'msgpack/rpc'
loop = MessagePack::RPC::Loop.new
client1 = MessagePack::RPC::Client.new(host1, port1, loop)client2 = MessagePack::RPC::Client.new(host2, port2, loop)
future1 = client1.call_async(:methodA, arg1, arg2)future2 = client2.call_async(:methodB, arg1, arg2)
result1 = future1.getresult2 = future2.get
Connection Poolingrequire 'msgpack/rpc'
sp = MessagePack::RPC::SessionPool.new
session1 = sp.get_session(host1, port1)session2 = sp.get_session(host2, port2)
future1 = session1.call_async(:methodA, arg1, arg2)future2 = session2.call_async(:methodB, arg1, arg2)
result1 = future1.getresult2 = future2.get
Concurrent calls with multiple servers
Client
Session Loop Server
Client
ServerSession Loop
sharedevent loop
Client
Client
Concurrent calls with multiple servers
Server
Server
Loop
Session
Session
Connection Pooling
Session PoolServer
Server
pools these connectionsLoop
Session
Session
Server architecture
Dispatcher
ServerClient
Client
Loop
Event-driven I/O
• Performance of the Loop is important.
• Java version uses Netty (JBoss’ I/O framework)
• Multithreaded
• Utilizes Java’s NIO
• C++ version uses mpio (Kumofs’ I/O architecture)
• Multithreaded
• Utilizes epoll or kqueue
• Ruby version uses Rev (libev for Ruby)
• Utilizes epoll or kqueue
Languages
• Serialization
• C, C++, Java, Python, Ruby, PHP, Perl, JavaScript, D, Lua, Erlang, Haskell, ...
• RPC
• C++, Java, Python, Ruby, PHP, Perl, Haskell, Lua, ...
The MessagePack Project
http://msgpack.sourceforge.net/
The Kumofs Project
http://kumofs.sourceforge.jp/