Android internals 04 - “Androdized” kernel, Bionic, Logging subsystem (rev_1.1)
Windows Kernel Internals NTFS
Transcript of Windows Kernel Internals NTFS
![Page 1: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/1.jpg)
Windows Kernel Internals IIAdvanced File Systems University of Tokyo – July 2004
Dave Probert, Ph.D.Advanced Operating Systems Group
Windows Core Operating Systems DivisionMicrosoft Corporation
© Microsoft Corporation 2004 1
![Page 2: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/2.jpg)
Disk Basics
Volume exported via device objectAddressed by byte offset and lengthEnforced on sector boundariesNTFS allocation unit - clustersRound size down to clusters
© Microsoft Corporation 2004 2
![Page 3: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/3.jpg)
Storage ManagementVolumes may span multiple logical disks
Partitioning Description Benefits
spanned logical catenation of arbitrary sized volumes
size
striped(RAID-0)
interleaved same-sized volumes
read/write perf
mirrored(RAID-1)
redundant writes to same-sized volume, alternate reads
reliability, read perf
RAID-5 striped volumes w/ parity reliability, size, read perf
© Microsoft Corporation 2004 3
![Page 4: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/4.jpg)
File System Device Stack
© Microsoft Corporation 2004 4
NT I/O Manager
File System Filters
File System DriverCache Manager
Virtual MemoryManager
Application
Kernel32 / ntdlluserkernel
Partition/VolumeStorage Manager
Disk Class Manager
Disk Driver
DISK
![Page 5: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/5.jpg)
NTFS Deals with files
Partition is collection of filesCommon routines for all meta-dataUtilizes MM and Cache ManagerNo specific on-disk locations
© Microsoft Corporation 2004 5
![Page 6: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/6.jpg)
CacheManager overviewCache manager
– kernel-mode routines– asynchronous worker routines– interface between filesystems and VM mgr
Functionality– access methods for pages of file data on opened files– automatic asynchronous read ahead– automatic asynchronous write behind (lazy write)– supports “Fast I/O” – IRP bypass
© Microsoft Corporation 2004 6
![Page 7: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/7.jpg)
Datastructure Layout
Kernel
Handle
File ObjectFilesystem File Context
FS Handle Context (2)
Section Object Pointers
Data Section (Mm)
Image Section (Mm)
Shared Cache Map (Cc)
Private Cache Map (Cc)
File Object == Handle (U or K), not one per fileSection Object Pointers and FS File Context shared/stream
© Microsoft Corporation 2004 7
![Page 8: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/8.jpg)
DatastructuresFile Object
– FsContext – per physical stream context– FsContext2 – per user handle stream context, not all
streams have handle context (metadata)– SectionObjectPointers – the point of “single
instancing”• DataSection – exists if the stream has had a mapped section
created (for use by Cc or user)• SharedCacheMap – exists if the stream has been set up for
the cache manager• ImageSection – exists for executables
– PrivateCacheMap – per handle Cc context (readahead) that also serves as reference from this file object to the shared cache map
© Microsoft Corporation 2004 8
![Page 9: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/9.jpg)
Cache View ManagementA Shared Cache Map has an array of View Access
Control Block (VACB) pointers which record the base cache address of each view– promoted to a sparse form for files > 32MB
Access interfaces map File+FileOffset to a cache address
Taking a view miss results in a new mapping, possibly unmapping an unreferenced view in another file (views are recycled LRU)
Since a view is fixed size, mapping across a view is impossible – Cc returns one address
Fixed size means no fragmentation …
© Microsoft Corporation 2004 9
![Page 10: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/10.jpg)
View MappingFile Offfset
0-256KB 256KB-512KB 512KB-768KB
c1000000 <NULL> cf0c0000
VACB Array
© Microsoft Corporation 2004 10
![Page 11: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/11.jpg)
CacheManager Interface Summary
File objects start out unadornedCcInitializeCacheMap to initiate caching via Cc on a file
object– setup the Shared/Private Cache Map & Mm if
neccesaryAccess methods (Copy, Mdl, Mapping/Pinning)Maintenance FunctionsCcUninitializeCacheMap to terminate caching on a file
object– teardown S/P Cache Maps– Mm lives on. Its data section is the cache!
© Microsoft Corporation 2004 11
![Page 12: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/12.jpg)
CacheManager / FS Diagram
Cache Manager
Memory Manager
Filesystem
Storage Drivers
Disk
Fast IO Read/Write IRP-based Read/Write
Page Fault
Cache Access, Flush, Purge
Noncached IO
Cached IO
© Microsoft Corporation 2004 12
![Page 13: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/13.jpg)
File System Notes
Three basic types of IO– cached, non-cached, paging
Three file sizes– file size, allocation size, valid data length
Three worker threads– Mm’s modified page writer (paging file)– Mm’s mapped page writer (mapped files)– Cc’s lazy writer pool (flushes views)
© Microsoft Corporation 2004 13
![Page 14: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/14.jpg)
Cache Manager SummaryVirtual block cache for files not logical block cache for disksMemory manager is the ACTUAL cache managerCache Manager context integrated into FileObjectsCache Manager manages views on files in kernel virtual
address spaceI/O has special fast path for cached accessesThe Lazy Writer periodically flushes dirty data to diskFilesystems need two interfaces to CC: map and pin
© Microsoft Corporation 2004 14
![Page 15: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/15.jpg)
NTFS on-disk structure
Some NTFS system files$Bitmap$BadClus$Boot. (root directory)$Logfile$Volume$Mft$MftMirr$Secure
© Microsoft Corporation 2004 15
![Page 16: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/16.jpg)
$Mft File
Data is entirely File RecordsFile Records are fixed sizeEvery file on volume has a File RecordFile records are recycledReserved area for system filesCritical file records mirrored in $MftMirr
© Microsoft Corporation 2004 16
![Page 17: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/17.jpg)
File Records
‘Base’ file record for each fileHeader followed by ‘Attributes’Additional file records as neededUpdate Sequence ArrayID by offset and sequence number
© Microsoft Corporation 2004 17
![Page 18: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/18.jpg)
File D:¥Letters (File ID 0x200)
A B C D E F G H I J K L M N O P Q R S T U V
File ¥$Mft
100200
2000
280200
P Q R S TA B C D E F G H IJ K L M N O U V
Physical Disk
P Q R S T G H I© Microsoft Corporation 2004 18
L M U V A B C D E F J K N O
![Page 19: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/19.jpg)
File BasicsTimestampsFile attributes (DOS + NTFS)Filename (+ hard links)Data streamsACLIndexes
File Building BlocksFile RecordsNtfs AttributesAllocated clusters
© Microsoft Corporation 2004 19
![Page 20: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/20.jpg)
File Record Header
USA HeaderSequence NumberFirst Attribute OffsetFirst Free Byte and SizeBase File RecordIN_USE bit
© Microsoft Corporation 2004 20
![Page 21: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/21.jpg)
NTFS Attributes
Type code and optional nameResident or non-residentHeader followed by valueSorted within file recordCommon code for operations
© Microsoft Corporation 2004 21
![Page 22: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/22.jpg)
MFT File Record
$STANDARD_INFORMATION (Time Stamps, DOS Attributes)
$FILE_NAME - VeryLongFileName.Txt
$FILE_NAME - VERYLO~1.TXT
$DATA (Default Data Stream)
$DATA - “VeryLongFileName.Txt:A named stream”
© Microsoft Corporation 2004 22$END (Available for attribute growth or new attribute)
![Page 23: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/23.jpg)
Attribute Header
LengthFormName and name lengthFlags (Compressed, Encrypted, Sparse)
© Microsoft Corporation 2004 23
![Page 24: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/24.jpg)
Resident Attributes
Data follows attribute header‘Allocation Size’ on 8-byte boundaryMay grow or shrinkConvert to non-resident
© Microsoft Corporation 2004 24
![Page 25: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/25.jpg)
Non-Resident Attributes
Data stored in allocated disk clustersMay describe sub-range of streamSizes and stream propertiesMapping pairs for on-disk runs
© Microsoft Corporation 2004 25
![Page 26: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/26.jpg)
Some Attribute Types$STANDARD_INFORMATION $FILE_NAME $SECURITY_DESCRIPTOR $DATA $INDEX_ROOT $INDEX_ALLOCATION $BITMAP $EA
© Microsoft Corporation 2004 26
![Page 27: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/27.jpg)
Mapping Pairs
Stored in a byte optimal formatRepresents allocation and holesEach pair is relative to prior runUsed to represent compression/sparse
© Microsoft Corporation 2004 27
![Page 28: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/28.jpg)
Indexes
File name and view indexesIndexes are B-treesEntries stored at each levelIntermediate nodes have down pointers$INDEX_ROOT$INDEX_ALLOCATION$BITMAP
© Microsoft Corporation 2004 28
![Page 29: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/29.jpg)
Index Implementation
Top level - $INDEX_ROOTIndex buckets - $INDEX_ALLOCATIONAvailable buckets - $BITMAP
© Microsoft Corporation 2004 29
![Page 30: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/30.jpg)
$INDEX_ROOT
E J endR
A B C G I N P Q Z
$INDEX_ALLOCATION
A B CG I N P QZunused data
© Microsoft Corporation 2004 300x36 (00110110)
$BITMAP
![Page 31: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/31.jpg)
$ATTRIBUTE_LIST
Needed for multi-file record fileEntry for each attribute in fileResident or non-resident formMust be in base file record
© Microsoft Corporation 2004 31
![Page 32: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/32.jpg)
Attribute List (example)
• Base Record -0x200
• 0x10 - Standard• 0x20 - Attribute List• 0x30 - FileName• 0x80 - Default Data• 0x80 - Data1 “Owner”
• Aux Record -0x180
• 0x30 - FileName• 0x80 - Data “Author”• 0x80 - Data0 “Owner”• 0x80 - Data “Writer”
© Microsoft Corporation 2004 32
![Page 33: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/33.jpg)
Attribute List (example cont.)
© Microsoft Corporation 2004 33
Code FR VCN Name (Not Present)0x10 0x200 $Standard0x30 0x200 $Filename0x30 0x180 $Filename0x80 0x200 0 $Data0x80 0x180 0 “Author” $Data0x80 0x180 0 “Owner” $Data0x80 0x200 40 “Owner” $Data0x80 0x180 “Writer” $Data
![Page 34: Windows Kernel Internals NTFS](https://reader035.fdocuments.us/reader035/viewer/2022081407/62963f4d74a64c02d33311a3/html5/thumbnails/34.jpg)
Discussion
© Microsoft Corporation 2004 34