Post on 11-Jan-2016
Mobile Handset Storage and File System
2
Outline
Storage and File System Basics Android File System iOS File System
3
Storage Hierarchy
Almost all computers use a storage hierarchy Put fast but expensive and small storages close
to the CPU Put slower but larger and cheaper storages far
away from the CPU
4
Primary and Secondary Storages
Primary storage (or main memory or internal memory) is the storage directly accessible to the CPU Lose information when not powered Such as cache, registers and main memory
Secondary storage (or external memory or auxiliary storage) is not directly accessible by the CPU The computer usually uses input/output channels to access
secondary storage Does not lose the data when the device is powered down Such as hard disks, CD/DVD, flash memory (e.g. sdcard)
5
Flash Memory
Flash memory is a computer memory chip that maintains stored information without requiring a power source. It belongs to the secondary storage in the computer storage hierarchy
It can be electronically erased and overwritten Mobile devices use flash memory to store data There are two major types of flash memory
NAND flash memory and NOR flash memory They are named after the NAND and NOR logic gates
6
Flash Memory History
Flash memory was invented by Dr. Fujio Masuoka (Toshiba)First presented in IEEE International Electron
Devices Meeting (IEDM) 1984Intel introduced the first commercial NOR type
flash in 1988Toshiba announced NAND flash at IEDM 1987The first NAND-based removable media format
was SmartMedia in 1995
7
NOR Flash Memory
Random access to any memory location Use blocks as the storage units (the typical
block sizes are 64KB, 128KB or 256KB) Erasure must happen on block level, a block at
a time. Write happens on byte level Long write and erasure time Developed as a replacement for ROM (read
often, rarely updated)
8
NAND Flash Memory (1)
Use page (a group of memory words) as the basic unit to store data. The typical page sizes are 512, 2048 or 4096 bytes
Associated with each page are 12 to 16 bytes for checksum
Pages are combined into blocks Read and write happen on a page level.
Erasure can only happen on a block level
9
NAND Flash Memory (2)
Write and erasure time is reduced Suitable for replacing disks Typical block sizes:
16KB: 32 pages of (512 + 16 spare bytes)128KB: 64 pages of (2048 + 64 spare bytes)256KB: 64 pages of (4096 + 128 spare bytes)512KB: 128 pages of (4096 + 128 spare bytes)Spare bytes can be used for checksum
10
NOR VS NANDNOR NAND
Performance Very slow eraseSlow writeFast read
Fast eraseFast writeFast read
Reliability Standard reliability Low reliabilityNeeds bad block management
Erase Times 10,000 – 100,000 100,000 – 1,000,000
Life Span Less than 10% the life span of NAND
Over 10 times more than NOR
Access Random Sequential
Hardware Implementation
Easy Complicated
Spare Bytes No Yes (16 bytes)
11
File System
File system is a computer program which controls how data is stored and retrieved
Primary roles:Provides an abstraction for secondary storageProvides a logical organization of filesEnables sharing data between processes, users and
machines Protects data from unwanted access
12
Why Need File System
Disks are messy physical devices Errors, bad blocks, missed seeks, etc.
The job of OS is to hide the mess from higher level software It needs to handle low-level device control (start a disk
read, etc.) It needs to provide higher-level abstractions (files,
databases, etc.) The file system handles the mess for OS
13
File Concept
A file is a logically contiguous address space which stores a collection of data. It has following attributes: File name File identifier (a unique number for the file) File type File location (pointer to file location on disk) File size File protection (controls who can read, write or execute),
etc.
14
File Operations
Most common operations:CreateWriteReadReposition within a fileDelete Truncate
15
File Protection (1)
File system must implement some kind of protection to control who can access a file and how they can access it
Types of users Owner: the user who created the file Group: the users who is in the same group with the owner Others: any other users in the system Super user: administrator of the system
Types of access are read (r), write (w) and execute (x)
16
File Protection (2)
Protection in Unix file system
17
Directories
A directory, also known as folder, is a structure which allows the user to group files into separate collections
The root directory is the first or top-most directory in tree structured directories. It is the starting point where all branches originate fromE.g., the / directory in Unix systems
19
Block (1)
A block is a sequence of bytes or bits and have a maximum length, a block size. It is the basic unit used by most file systems to store data
File systems define a block size (e.g., 4KB)Disk space is allocated in granularity of blocks
A “Master Block” stores the location of root directoryAlways at a well-known disk locationOften replicated across disk for reliability
20
Block (2)
A map stores which blocks are free, which are allocatedUsually a bitmap, one bit per block on the diskAlso stored on disk, cached in memory for
performance Remaining disk blocks are used to store files
and directories
21
Outline
Storage and File System Basics Android File System iOS File System
22
Overview
Android uses flash memory as its storage media, so it can use flash file systems such as exFAT, YAFFS2, JFFS2, etc.
Android is based on Linux kernel, so it can use a Linux file system, such as ext2, ext3, ext4, etc.
It may also use a proprietary file system developed by a manufacturer, depending on who made the device
The most commonly used file system on Android Yet Another Flash File System2 (YAFFS2)
23
YAFFS
YAFFS is a flash file system developed for NAND flashYAFFS1: designed for early NAND generations of
flash memory (512-byte page)YAFFS2: support new NAND with 2KB pages and
strictly sequential page writing order It uses chunk to manage data. Chunk is YAFFS
terminology for a page.
24
YAFFS1 Chunks
File data stored in fixed size “chunks”, i.e., NAND pages (512 bytes)
Two types of chunk:Data chunk: holding regular data file contentsFile header: a file’s metadata such as file name,
parent directory, etc.
25
YAFFS1 Tags Each chunk has tags with it. The tags comprise the
following fields (8 bytes in total):Field Bits Meaning
File ID 18 Identifies which file the chunk belongs to
Chunk ID 20Identifies where in the file this chunk belongs to. 0 means this chunk contains a file header, 1 means the first chunk and 2 is the next chunk and so on
Serial Number 2 Differentiates chunks with the same file ID and chunk ID
Byte Count 10 Number of bytes of data if this is a data chunk
Checksum 12 Checksum for tags
Reserved 2 Unused
64 Total
26
YAFFS1 Serial Number
When data is overwritten, the relevant chunks are replaced by writing new pages to the flash containing the new data. Then the old page is marked as “discarded”
If power loss/crash/other problem happens before the old page is marked as regarded, it is possible to have two pages with the same tags Solve the problem: Increase 2-bit serial number by 1 every
time a chunk is overwritten to distinguish the new data and old data
27
YAFFS1 Garbage Collection
A block with all discarded pages is an obvious candidate for garbage collection
Otherwise, valid pages are copied from a block and then mark the whole block discarded and ready for garbage collection
28
YAFFS1 Page Layout
Bytes Range Fields Details
0 - 511 Data Data, either file data or file header depending on tags
512 - 515 Tags Tags
516 Data Status If more than 4 bits are zero, this page is discarded.
517 Block Status Shows whether the block is damaged
518 - 519 Tags Tags
520 - 522 Checksum Checksum for second 256 bytes part of data
523 - 524 Tags Tags
525 - 527 Checksum Checksum for first 256 bytes part of data
29
YAFFS2 VS YAFFS1 (1) YAFFS2 is very similar in concept to YAFFS1 and
they share much of the same source code Add support for new NAND with 2KB pages Mark very newly written block with a sequence
number The sequence of the chunks can be inferred from the block
sequence number and chunk offset within the block When it detects two chunks with same file ID and chunk
ID, it can choose the new chunk by taking the greater sequence number
30
YAFFS2 VS YAFFS1 (2) Introduce concept of shrink headers for efficiency
When a file is resized to a smaller size, YAFFS1 will mark all of the affected chunks as discarded. But YAFFS2 writes a “shrink header”, which indicates that a certain number of pages before this header are invalid
Improve performance relative to YAFFS1 Write 1.5-5x Delete: 4x Garbage collection: 2x
31
Outline
Storage and File System Basics Android File System iOS File System
32
Overview
In 1985 Apple developed a new file system called hierarchical file system (HFS) for use in Mac OS
Hierarchical file system plus (HFS+) was introduced in 1998 for use in Mac OS 8.1
HFSX was introduced in Mac OS 10.3 in 2005. Now it becomes the file system for iOS
33
HFS Blocks
At the physical level, the disk is divided into blocks of 512 bytes
There are two types of blocks:Logical blocks: they are numbered from the first to
the last on the disk. And they are static and the same size as the physical blocks, 512 bytes
Allocation blocks: they are groups of logical blocks used by the HFS to track data in a more efficient way
34
HFS Structure (1) Logical blocks 0 and 1
the boot blocks which contain system startup information Logical blocks 2
contains the master directory block (MDB) which defines a wide variety of data such as date and time stamps for when the partition was created, the location of the bitmap, etc.
Logical block 3 the starting block of the bitmap which keeps track of which
allocation blocks are in use and which are free. Each allocation block is represented by a bit in the map: if the bit is set, the block is in use. Otherwise it is free to use.
35
HFS Structure (2) The extent overflow file
Keeps track of which allocation blocks are allocated to which files
Catalog fileDescribes the folder and file hierarchy on the disk.
It contains metadata about all the files and folders on the disk including information about modify, access and create times
36
HFS+ VS HFS HFS+ has three more parts in terms of the
structureAttributes file: contains attribute information of all
files and foldersStartup file: designed to assist in booting non-Mac
OS systems that don’t have HFS or HFS+ supportReserved block: reserved for use by Apple
37
HFSX VS HFS+ All Apple mobile devices use HFSX as the file
system. There is one major difference between HFSX and HFS+. HFSX is case sensitive.For example, Case_sensitive.doc and
Case_Sensitive.doc are treated as two different files. They can both exist on HFSX but not in HFS+
38
References (1) http://en.wikipedia.org/wiki/File_system https://cseweb.ucsd.edu/classes/fa06/cse120/lectures/120-fa06-l13.pdf http://codex.cs.yale.edu/avi/os-book/OS8/os8c/slide-dir/PDF-dir/ch10.pdf http://www.dubeiko.com/development/FileSystems/YAFFS/
HowYaffsWorks.pdf http://en.wikipedia.org/wiki/YAFFS http://www.forensicswiki.org/wiki/HFS%2B http://en.wikipedia.org/wiki/Mac_OS_X_Tiger http://en.wikipedia.org/wiki/HFS_Plus