Object-based SSD (OSSD): Our Practice and Experience · OSD and Object-based SSD Introduction Our...
Transcript of Object-based SSD (OSSD): Our Practice and Experience · OSD and Object-based SSD Introduction Our...
SAMSUNGFSWD
SAMSUNG
Object-based SSD (OSSD):
Our Practice and Experience
Jaesoo Lee
Flash Solution Team, Memory Division
Samsung Electronics Co.
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
2
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
3
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNGOutdated Storage Stack
� HDD as main storage for decades
� Structure is relatively simple
� Physical location can be easily derived
� Storage stack has remained static
( ) ( ) 1secsec −+⋅+⋅⋅= sNhNNcLBA torstorsheads
4
Storage stack has remained static
� Narrow block interfaces (ATA, SCSI)
� No information flow except block reads/writes
� File subsystems make HDD-specific assumptions
� Sequential read is much faster than random read
� No write amplification, wear-out, background activity, …
� What if the underlying device changes ?
[Excerpted from Block Management in SSD, Usenix 2009 ]
SAMSUNG� Host-side optimization is no more feasible
� Confidential, subject to changes, different among SSDs, …
Challenges in SSD Storage Stack
□ Striping method
□ Buffer cache management policy
□ Logical-to-physical mapping
□ Garbage collection policy
□ Wear-leveling policy
□ Bad block management
□ # of channels
□ # of ways
□ # of planes per chip
□ # of blocks per plane
□ # of pages per block
5
Processor
Core
□ # of pages per block
□ Block size
□ Page size
[figure by Sang Lyul Min @ SNU]
SAMSUNGSSD-Aware File System
� Preliminary results with in-house SSD-aware file system
� Up to 1700% improvement for random writes
� 13% degradation in sequential write performance
12
14
16
18
No
rma
lize
d T
hro
ug
hp
ut
SSD-Aware FS
ext3
6
0
2
4
6
8
10
12
4 8 16 32 64 128 256 512 1024 2048 4096 8192
No
rma
lize
d T
hro
ug
hp
ut
I/O Size (KB)
SAMSUNGOSD: The Basic Concept
� Optional command set defined for SCSI device
� Provide object-based interface instead of traditional block-
based interface
� In OSD, an object is a flexible-sized data container
� Unique object ID
� A set of attributes
7
� A set of attributes
SAMSUNGSSD as OSD
� OSD manages space for objects
� Informed cleaning (utilize delete info)
� Stripe aligned accesses
� Logical to physical mapping
� OSD supports object attributes
8
� OSD supports object attributes
� Wear-leveling using cold data
information
� Priority assigned to objects (i.e., QoS)
� OSD handles low-level operations
� Block management in SSD
[Source: Block Management in SSD, Usenix 2009 ]
SAMSUNGLinux Support for OSD
� Linux provide a ready-to-use OSD prototype
� open-osd: open source initiator and exofs filesystem
� OSC-OSD: iSCSI OSD target
� Open-iSCSI: iSCSI transportvfs
exofs ext3 udf
9
scsi sata
iscsiiscsi
initiatorscsi initiator
SCSI/SCSI/
SATA
SCSI core (scsi_mod.ko)
bsg sd sr stosd_uld (osd.ko)
osd_lld (libosd.ko)
SAMSUNGTarget Platform
� OSD is the most promising for mobile storages
� Storage and system vendors are decoupled inherently
� QoS provisioning is essential
� e.g., managed flash memory applications including
� eMMC, UFD, miniSD, SD, …
� Our primary target is a Linux-based mobile platform
10
� Our primary target is a Linux-based mobile platform
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
11
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNG� Prototyped in the Linux-based host
� MLD: object mapping, space management, …
� New type of SSD developed (called rawSSD)
OSSD Prototype
exofs
VFS□ Get rawSSD information
□ Erase all blocksfrom open-osd
12
Host (Linux)
OSSD ULD
OSSD MLD
OSSD LLD
raw SSD
Flash Cmd I/F
SCSI/SATA Flash Mng.
□ Erase all blocks
□ Erase a block
□ Read a flash page
□ Program a flash page
SATA Link
from open-osd
and Linux kernel
SAMSUNGMLD: Object Management Layer
� Overall architecture
(RB-Tree)
Index
(RB-Tree)
{PID, OID} [127:0]Allocator
GC
Wear-leveling
OSD Commands
13
Object descriptorWear-leveling
Meta data User data
Flash
media
OSSD LLD
READ / PROGRAM / ERASE
SAMSUNGMLD: Object Descriptor
� Containing metadata including extents and attributes
� Cached in memory (LRU)
Descriptor Header□ Object ID
□ PPN
□ Object information
14
Block Table
Attributes
1page
(16KB)
□ PPNs for object data
□ Extent: support 9MB
□ Indirect table: support 4GB
□ Attr. data of object
□ Usually contain i-node and length
□ Managed using original code
SAMSUNGMLD: Object Mapping
� Support page-wise mapping
� Based on page extents
� Association policies of an update block
� Fully associative
� Separation of index and user data
15
� No separation (called unified)
� Object associative (called per object)
� Garbage collection
� Victim selection
� Highest invalid pages
� Bitmap for invalid information
� Background GC supported
SAMSUNGLow Level Driver
� Provide block + Raw SSD specific
interfaces to upper layers
� Sanity check
� Page overwrite
� Out-of-order page write
OSSD LLD
Sanity Checker
Block interface
- submit_bio()
Raw SSD interface
- obtain_config()
- erase_block()
- format()
16
SCSI
subsystem
� Out-of-order page write
� Provide backend flexibility
� RawSSD
� Via SCSI-ATA Translation Layer (SATL)
� Loop
� RAM
Backend dispatcher
RAM LoopRaw
SSD
SAMSUNGExperimental Setup
� Host system
� Quad Q9650 3GHz
� 4GB RAM
� Linux kernel 2.6.33
� 4GB partition for OSSD
Write ThreadsRead Thread
…
17
Host
exofs
OSSD Driver
raw SSD
Flash Cmd I/F
SCSI/SATAFlash Mng.
VFS
SAMSUNGExperiments
� 1. Better performance for fragmented/multi-stream
writes
� Effect of incorporating space management in OSSD
� 2. QoS support (read prioritized service)
� Effect of having knowledge on the contexts of data blocks
18
� Effect of having knowledge on the contexts of data blocks
and a request
SAMSUNGExp1. Multi-thread File Write
� Scenario
� Write 800 MB 4 thread � delete 2 file � write 800 MB 2
thread
� Show the effect of fragmentation
Unified update
block
Index / Data Index / Per object
19
block
Write 800MB
4 thread
3m48.470s 3m52.263s 3m49.769s
After delete,
write 800MB
2 thread
6m34.535s 2m59.239s 1m59.375s
Erase 689 599 414
Valid copy 34887 23605 0
SAMSUNGExp2. Read Priority
� Scenario
� 200MB read over total 2.4G write (e.g., 300MB x 8)
20
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
21
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNGCapacity Outran Bandwidth
� Capacity of HDD grows exponentially
� Ratio of the capacity to the interface bandwidth also
grows exponentially
100000
1000000
100000
1000000
Ca
pa
city
/ I
nte
rfa
ce B
an
dw
idth
(se
c)
Ca
pa
city
(M
B,
log
sca
le)
Capacity
C/B Ratio
22
1
10
100
1000
10000
1
10
100
1000
10000
1990 1995 2000 2005 2010
Ca
pa
city
/ I
nte
rfa
ce B
an
dw
idth
(se
c)
Ca
pa
city
(M
B,
log
sca
le)
Year
Maxtor 7000
(IDE)
Quantum Fireball ST
(UltraATA/33)
IBM Deskstar 16GP
(UltraATA/33)
Seagate Barracuda IV
(UltraATA/100)
Seagate Barracuda
(SATA/300)
SAMSUNGApplication-Aware Storage
� With a small database proxy, the amount of data
transferred can be decreased significantly
� scan, aggregation, join, sorting, …
23
� Other promising areas
� Data mining, search indexing, image processing, Anti-Virus, …
SAMSUNGApplication-Aware Storage
� Application-awareness can be easily achieved in OSSD
� Integrated object (i.e., file) management
� Fluent attribute mechanism
� Furthermore, SSD is no more dumb
� 4 Cortex-R4 CPUs, SATA 6Gbps, 512MB RAM, AXI bus matrix,
24
� 4 Cortex-R4 CPUs, SATA 6Gbps, 512MB RAM, AXI bus matrix,
16 flash memory controllers, …
� What about moving some of application’s work to
OSSD?
� Issues are models for programming, execution, and
deployment
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
25
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNGPostgreSQL
DB-Aware OSSD
� PostgreSQL
� Free and open source
� Object-relational database
� Developed a plug-in for
accelerating:
Query PlannerQuery Planner
Query ParserQuery Parser
OSSD
Plug-in
Query
Optimizer
Query
Optimizer
ExecutorExecutor
26
accelerating:
� Aggregation
� Selection
� Query is processed in a similar
way to the active diskKernel
VFS
EXOFS
OSD Driver
DB-Aware OSSD
DB Proxy
ioctl
SAMSUNGDB Operators
� Aggregation
� Count, Sum, AverageSELECT count(*) FROM emp where age < 30;
Count(*)
--------
17
Result Result
27
Name Age Salary
Mc 29 3000
Kim 26 1200
Na 34 4000
Kang 28 1400
Lim 25 400
… … …
Seq. Read
Aggregate
Seq. Read
And aggregate
SAMSUNGDB Operators
� Selection, Projection
� Filtering (not yet)SELECT * FROM emp where age > 30;
Name Age Salary
----- ----- --------
Na 34 4000
Result Result
28
Seq. Read
Selection
Seq. Read
And selection
Name Age Salary
Mc 29 3000
Kim 26 1200
Na 34 4000
Kang 28 1400
Lim 25 400
… … …
SAMSUNGImplementation Mechanisms
SELECT AGGREGATE(target_col)
FROM t1 OSD
WHERE cond_col COND value;
Linux Host
PostgreSQL
� Currently,
� Running on existing exofs file system
29
WHERE cond_col COND value;
Translate inode to Object ID
Sequential Read
and
Processing Aggregate Function
exofs
OSSD ULD
OSSD LLD
SCSI/SATA
OSSD MLD
SAMSUNGExperiments
� Experimental setup
� Hardware
� AMD Atholon64x2 7750 2.7 Ghz
� 3GB RAM (PC6400)
� HDD WD3200AAKS (SATA2/7200/16M)
� OS and drivers
Linux kernel 2.6.33 (Fedora Core 13)
30
� Linux kernel 2.6.33 (Fedora Core 13)
� Target : OSC-OSD
� File system and Driver : Open-OSD exofs and drivers
� Database
� PostgreSQL 8.4.4
SAMSUNGExperiments
� Results will be available soon!!
31
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
32
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNGFuture Directions of Work
� 1. OSD SSD
� Native support of OSD commands in SSD
� OSD command tunneling
� 2. Bi-directional communication extension in SATA
� Virtually any communication protocol over SATA
33
� Virtually any communication protocol over SATA
� 3. i2SSD: open storage software platform
� JVM and deployment middleware
� Storage programming framework
SAMSUNG
Linux Host OSSD
1. OSD SSD Architecture (Revised)
exofs
OSSD ULD OSSD Cmd Handler
34
OSSD LLD
SCSI/SATA
OSD Cmd I/F (via Tunneling Tech.)
SATA Link
Object Layer
FTL
SATA
Buffer Manager
SAMSUNG1. OSD SSD: OSD Cmd Tunneling
Host SSD
� 2-phased protocol
� OSD command is transferred as a payload of the 1st
command
35
1st phase
(cmd transfer)
2nd phase (optional)
(data transfer)
SATA vendor
command
(register FIS)
D2H DataH2D Data
SAMSUNG2. Bi-Directional Comm Extension
� ATA/SCSI protocol is too limited for application-specific
extension of OSSD
� We extended SATA with a simple bi-directional
communication protocol
� Based on reserved LBAs and notification mechanism
� Serve as a link layer of other advanced protocol, e.g., TCP/IP
36
� Serve as a link layer of other advanced protocol, e.g., TCP/IP
Application Main
ATA/SCSI Trasport
Bi-directiona l
Messaging Cmd.
Application Agent
ATA/SCSI Trasport
Bi-directional
Messaging Cmd.
FTL
SATA/SAS/… Link
Bi-directional
communication
HOST HDD/SSD
SAMSUNG3. i2SSD: Intelligent and Innovative SSD
� What is i2SSD ?
� Object-based SSD capable of extending its functionalities via
dynamic application deployment
� i.e., “Open Storage Architecture for Enabling Storage-Aware
Application Deployment”
� Benefit is significantly saved time and energy
� Computation at the near of the data
37
� Computation at the near of the data
dynamically deployedapplication
service library componentsi2SSD
i2SSD Run-time Framework
SAMSUNG3. Preliminary i2SSD Architecture
i2SSD Framework
Service Component Layer
Application Layer
38
i2SSD Framework
JVM
JNI OS(Linux?)
Secu
rity
Bi-directionalcomm.
Stream EditorRegEx Match
Compression/Decompression
Encoding/Decoding
Object Layer
FTL
HIL
..
SAMSUNGOutline
� Part 1. OSD and Object-based SSD
� Introduction
� Our Practice and Experience
� Part 2. Application-Aware Storage
� Introduction
39
� Introduction
� Our Practice and Experience
� Future Directions
� Summary and Conclusion
SAMSUNGConclusion
� OSD fits well for SSD
� Free from variations in SSD internals
� Easy application-specific extension
� Advanced features (e.g., QoS) made possible
� We have developed a proof-of-concept OSSD and
40
� We have developed a proof-of-concept OSSD and
demonstrated the benefits
� The results look promising
SAMSUNGCall for Participation
� Standardization
� SCSI/SATA extensions for OSD command tunneling
� SCSI/SATA extensions for bi-directional communication
command set
� i2SSD framework and programming interfaces
41
� Call for participation of Linux community
� OSSD bring-up issues in Linux
� End-to-end storage QoS support in Linux
SAMSUNGFSWD
SAMSUNG
Thanks