This presentation is to provide a technical understanding of the Microsoft Extended File Allocation Table (exFAT) for forensics examiners.
It will also provide general knowledge of exFAT.
1
One of the is to provide the background and history of the file system, and the relationship to the other FAT versions in the family.
2
3
D4CS stands for Digital Forensics & Cyber Security
FCM 760 Fall 2009
4
This presentation was given 5 times within the 14 months since the SANS paper was published.
It will now be given twice in 2014, with some updates
At the 4/19 Computer Forensics Show, the session was recorded and can be found on the AT&T Tech Channel
5
Both books were published in 2012, other books have been published, and may mention exFAT, but these two send the reader to the SANS paper for more detailed and in-depth information.
6
Brian Carrier’s book is considered by some as the gold standard and bible on explaining file systems.
It has become a little outdated since it is about 9 years old, no 2nd edition, and other file systems have surfaced since the book’s release.
Earlier in 2014 exFAT support was planned/announced for TSK
7
This is the Sleuth Kit Wiki, on this page it is noted that exFAT support was added to TSK, and one of the docs used as a guide during development was the SANS paper.
8
Microsoft published a patent that included the exFAT 1.00 specification.
This presentation and the paper attempt to stick to the terminology used in the patent/specification as close as possible.
Links to the patent and my paper will be on a later slide, and references to the paper will also be on my blog.
The presentation will focus on desktop/server version of exFAT which in 2014 is still Version 1 00 as is still released on Windows 8Version 1.00, as is still released on Windows 8
There are differences between implementations of other vendors, including incompatibilities, and the Windows phone has advances features and also uses compression.
The technology behind this SD card feature is called Content Protection for Recordable Media (CPRM)Recordable Media (CPRM).
Content Protection for Pre-recorded Media (CPPM),
9
10
http://www.webopedia.com/TERM/O/OSR_2.html
OEM Service Release 2 – Windows 95B
According to Wikipedia, there was a 8 bit FAT originally
Ray Duncan (1988). The MS-DOS Encyclopedia - version 1.0 through 3.2. Microsoft Press. ISBN 1-55615-049-0.
Although we talk about FAT12/16/32, there were many flavors as the FAT family evolved.
11
exFAT is specifically designed for Removable media, but can be used for fixed media as well.
NTFS is not recommended for removable media, especially because of the lazy write problem.
Faster I/O through less file system overhead
Limitation on how many times you can write to a single electronic gate (e g nand)Limitation on how many times you can write to a single electronic gate (e.g. nand)
FAT in general is simpler, so in the case of embedded systems, stick with a variation of FAT instead of implementing NTFS
12
SD = Standard Definition
HD = High Definition
Quad HD is 4 times Full HD (Double wide, Double Long)
Amount of data being recorded depends on many factors, including frames per second, size, color size, resolution, and compression rations.
Definitely will exceed the 4GB file size limitation.
13
Source: https://www.sdcard.org/consumers/cards
14
15
•You need to be able to locate the evidence, just in general
•This includes re-assembly when a file is fragmented
•Also includes recovery of deleted files
•You also need to know the hiding places where it can be hidden
•For example, unallocated space
•You need to validate what you found is correct, in order (proper assembly), and complete (no missing pieces)
•CP (Child Porn) when created uses cameras, and as camera memory gets cheaper, and moves to exFAT, relevant evidence is going to be on exFAT
16
Don’t be a money pressing a button, need to know what is under the covers
17
If the OS can’t recognize the file system, then it thinks the media is not formatted.
When this slide was built, it was 2010, on a Windows XP machine, that did not have the hotfix.
This example is Microsoft specific, and with XP being retired, and later systems getting the support, this situation should not occur often anymore on Microsoft Systems.
Back when this happened this message would lead one to believe that the mediaBack when this happened, this message would lead one to believe that the media was blank and unformatted.
18
Any evidence with exFAT would probably be pushed aside with the lack of tools, documentation and expertise to process it.
Tools that were available were raw acquisitions and then data carving
Data carving easier and automatic when the file is contiguous
19
Linux and Open Source is used a lot for examinations
Commercial tools are lacking, but picking up
Little documentation or publications on exFAT internals, 4 years later and SNAS paper appears the authoritative resounse.
exFAT Computer Compatibility
http://gopro.com/support/articles/exfat-computer-compatibility
We are not there yet
https://www.cyberfetch.org/sites/default/files/EnCase%20Forensic%20v6.18.0.59%20Test%20Report pdf0Test%20Report.pdf
Test Results for Deleted File Recovery and Active File Listing:
EnCase Forensic Version 6.18.0.59
This report was prepared for the Department of Homeland Security Science and
20
This report was prepared for the Department of Homeland Security Science and Technology Directorate Cyber Security Division by the Office of Law Enforcement Standards of the National Institute of Standards and Technology.
For additional information about the Cyber Security Division and ongoing projects, please visit www.cyber.st.dhs.gov.
http://www.cftt.nist.gov/presentations/AAFS-2013-Lyle-DFR.pptx
AAFS = American Academy of Forensic Sciences
http://www.cftt.nist.gov/presentations/AAFS-2013-Lyle-DFR.pdf
6 Vendors, not named in the presentation
21
Encase: http://www.cyberfetch.org/sites/default/files/EnCase Forensic v6.18.0.59 Test Report.pdf
FTK: http://www.cyberfetch.org/sites/default/files/FTK v3 3 0 33124 Test Report.pdf
Access to Test Images
Layout of test ImagesLayout of test Images
Each test case is repeated at least four times to characterize the tool’s behavior for different file system families. These include FAT, exFAT, NTFS and ext. The NTFS and exFAT images contain a single partition. The FAT and ext images each contain three partitions. Each partition has the same pattern of files created and deleted for a gi en test case The FAT and e t cases (three partitions) ha e three times asa given test case. The FAT and ext cases (three partitions) have three times as many files as the NTFS and exFAT cases (one partition). The FAT images contain a FAT-12, a FAT-16 and a FAT-32 partition.
The FAT partitions were created on a Windows Vista system. Some partitions marked as FAT-12 in the partition table, appear to have a FAT table that is actually FAT 16 (thi did t i ifi tl ff t t t lt ) Th NTFS i lFAT-16 (this did not significantly affect test results). The NTFS images were also created on a Microsoft Windows Vista system.
The ext partitions were created on a Fedora Linux system.
The exFAT partition and HFS+ partitions were created on a Mac running Snow Leopard, OSX Version 10.6. 22
23
24
When we use the term “Mega”, is it 1,000,000 (1006) 0r do we mean “Mega” 1,048,576 (220) ?
25
A quick note on exponents, since we will get our hands dirty with math
Some simple numbers should be like learning the times table in school
26
In some cases you might see ZB or ZIB, technically they are really different, but are close.
So when we say 1 kb of disk, they mean 1,000 bytes, but when we say 1 kb of memory, they mean 1024 bytes.
IEC 60027-2 A.2 and ISO/IEC 80000
http://physics nist gov/cuu/Units/binary htmlhttp://physics.nist.gov/cuu/Units/binary.html
27
Just another slide
It is suggested that in English, the first syllable of the name of the binary-multiple prefix should be pronounced in the same way as the first syllable of the name of the corresponding SI prefix, and that the second syllable should be pronounced as "bee."
28
Being off by 15% when talking about an "exabyte" means being off by about bytes, or 150 petabytes.
29
exFAT uses 16 bit Unicode strings
This is the terminology as used in the specifications leaked in the patent
When reading the paper, and as we discuss here, these are the ground rules in terminology
30
It is important to note that Pentium processers use the little-endian format, so numbers stored in the file system are stored in little-endian. This can be significant because you need to change the order of the bytes in order to read the values from a hex dump.
This could have issues with support of exFAT in other architectures, and could affect acquisitions.
The exFAT specification requires little endian.
31
This is how Microsoft does Math, and then everyone uses these numbers not knowing the full context
232 sectors * 29 bytes per sector (512B) = 241 = 2,199,023,255,552 (2TB)
http://support.microsoft.com/kb/184006
32
http://support.microsoft.com/kb/955704
Volume size of 64ZB is architecturally incorrect. Currently it cannot exceed 128PiB because:
1) With 232 clusters (32 bit fat indices) tracking clusters with a maximum of 225 in size = 257 = 128 PiB [32+25]
2) With LBA 48 as the maximum addressable block on the FS with a 512B physical2) With LBA-48 as the maximum addressable block on the FS, with a 512B physical sector, the file system supported would be 257 = 128 PiB [48+9]
Note that with #2, we could go further with native AF 4K sectors, however it is interesting how the numbers add up.
For file size, the current architecture uses 64 bit numbers for the length of file, based on that the maximum (theoretical) file size is really 264-1 = 16EiB
Now, since the architecture limits the filoe system to less than 128 PiB, and PIB is smaller than EiB, the maximum file size is almost the volume size minus overhead and metadata.and metadata.
33
http://en.wikipedia.org/wiki/Windows_CE
Microsoft Windows CE (now officially known as Windows Embedded Compactand previously also known as Windows Embedded CE
Small footprint, limited API
Windows XPE, XP Embedded – Different, uses desktop code but not all features
WinCE code is used to derive code for other embedded systems including the phone
http://support.microsoft.com/default.aspx?scid=kb;EN-US;Q166915p pp p ; ;Q
34
35
You never really see another sector size other than 512 bytes, but everyone just assumes that it is only 512 (in earlier versions of Windows NT, there were weird sector sizes)
The 4096 size is special to support a device that is used for paging and supports 4K pages. But with the standard format, you can’t adjust sector size
http://en.wikipedia.org/wiki/Advanced_Format – Advanced format is for any sector size > 520 bytes.
Clusters (or blocks) are 64KiB max in FAT32
However, for Windows 95 OSR2, 98 and ME, the FAT32 clusters had a maximum of 32KiB, so for backward compatibility it was recommended to not exceed 32KiB
ExFAT and FAT32 Root Directories not restricted in size, other than space available on the volume.
Max files on FAT32 volume ≈ 228 while exFAT ≈ 232 This is based on a maximum of one file per allocation unit (each cell in the FAT is one allocation unit)
36
one file per allocation unit (each cell in the FAT is one allocation unit)
Since exFAT supports empty files that don’t take up space in the cluster heap (length = 0 first cluster = 0) Max Files theoretically could be more
Ref: http://support.microsoft.com/kb/955704
This new format, called Advanced format, provides via hardware 4K physical sector size.
An OS/FS can either use emulation (512e) or native (4Kn)
exFAT will support 4K sized sectors.
37
This command (help format) was issued on a Windows 7 32-bit system.
This snippet is for the allocation unit size.
Most “supports” largest clusters for 512b sectors at 64K, including NTFS, FAT & FAT32. FAT & FAT32 appear to support a larger allocation units of 128K and 256K when sectors are > 512b (probably AF sectors).
Although in some cases 64K allocation units are supported, not all OS support it, and in some cases 64K+ not supported and must be a power of 2 thus 32Kand in some cases 64K not supported, and must be a power of 2, thus 32K.
Differences between Windows 95/98/ME and Windows NT4/2000/XP, 7 & 8
Even though a FAT32 could lead to 8TB (for 32K) and 16TB for 64K cluster sizes, when putting into a MBR, LBA is 32 bits, a block is 512b, so all file systems in a MBR is restricted to 2TB
Might get 8TB if AF (4K sectors) are used.
38
Microsoft in the KB for Windows XP support indicated a capacity to 64ZiB and a file size maximum to 64ZiB.
In reality, the file system can only support up to 128PiB, and the file size up to 16EiB.
Microsoft documentation indicates a maximum file system size as 512TiB
The recommended maximum volume size is 512 TBThe recommended maximum volume size is 512 TB.
http://support.microsoft.com/?kbid=955704
The volume size is limited by a 32-bit FAT and a 25-bit cluster size giving a 57-bit addressable volume size
The file size is limited by the 8-byte (64-bit) number that holds the file size.The file size is limited by the 8 byte (64 bit) number that holds the file size.
The volume label and file names are all 16 bit unicode
Filenames to a maximum of 255 characters
Subdirectory is max at 256MiB, Directory records are 32 bytes, and the smallest fileset is 3x32 = 96 bytes and assumes no ACL and a filename < 16 characters in length.
exFAT better optimized, reduce the “write” actions
For media that use OEM parameters this may be a method to convey device39
With TexFAT there will be 2 FATS and 2 BITMAPS, with exFAT 1.0 – which does not have TexFAT (Transactional FAT) support, there is ony 1 FAT and 1 BITMAP, where previous FAT versions had 2 FATs.
To be released later, but it is 6 years and we are still at VV.MM 01.00
TexFAT and ACL already exist in Windows CETexFAT and ACL already exist in Windows CE
40
Any FS is limited, even FAT32 and NTFS.
This is Windows only, we are not talking GUID Partition Table (GPT)
Although a MBR uses a 4 byte sector count, remember that the FS can be larger if you make the sectors larger (512 vs. 4096) and this causes a lot of confusion on how big a FS fits.
A FAT32 filesystem could reach 8 TiB in size (2**28 x 32K), but with 512 sector sizes a MBR can only support 2TB (with 4K size a MBR can support 16TiB)sizes, a MBR can only support 2TB. (with 4K size, a MBR can support 16TiB)
http://support.microsoft.com/kb/314463
http://techcosupport.com/press/maximum-size-of-a-fat-32-partition/
GUID Limits:
http://support.microsoft.com/kb/302873http://support.microsoft.com/kb/302873
http://msdn.microsoft.com/en-us/library/windows/hardware/dn640535(v=vs.85).aspx
A GPT GUID Partition uses a 64 bit number for the number of logical blocks
In theory, a GPT disk can be up to 2^64 logical blocks in length. Logical blocks are commonly 512 bytes in size.
41
This would be 264 * 29 = 273 which is 270 = 1 ZiB and 23 = 8 for ≈ 8Zib
maximum partition size of 264−1 sectors. For disks with 512-byte sectors, that would be 9.4 ZB (9.4 × 1021 bytes) or 8 ZiB−512 bytes (9,444,732,965,739,290,426,880 bytes or 18,446,744,073,709,551,615 (264−1) sectors × 512 (29) bytes per sector)
Windows would not format FAT32 beyond 32GB, it required using a FAT32 format on a different OS
Some Windows utilities did not work properly with volume spaces GT 32GB, but you can mount a device that was GT 32GB
Limitations of FAT32 File System: http://support.microsoft.com/kb/184006
SDXC predecessor (SDHC) had a max spec of 32GB. SDXC picks up from 32GB. (But starts around 48GB 32GB will till be SDHC for a while)(But starts around 48GB, 32GB will till be SDHC for a while)
4GB maximum file size barrier existed in both FAT and FAT32.
SD 4.0 Specification – 300MB/s I/O speeds
http://www.flashmemorysummit.com/English/Collaterals/Proceedings/2009/20090813_S204_Lin_Yee.pdf
Starting at 104 mega bytes per second, and later to 300 mega bytes per second
http://www.letsgodigital.org/en/20985/sdxc-cards/
Microsoft set limits on FAT32 volume size
In one argument, older utilities could not format the volume correctly or could not determine the proper size
42
In another argument, since the larger volumes had a much larger FAT, massive reads of the FAT would be required to find free space. For example, with a 32K cluster size and a 32GB media, the FAT would be about 4MB, and for a heavily used (low free space) volume there could be a lot of I/O to find free clusters. FAT32 limited this overhead by adding a hint of freespace using the FAT32 File System Information sector, but it was a hint and not always to be relied upon, just to point the software to where free cluster might be
3rd party file utilities may provide conversion to and from exFAT, but no Convert command, and current convert command doesn’t work even to change exFAT to NTFS or even FAT32.
Mostly a Microsoft Desktop and Server World – there is Linux, and MAC, Microsoft dominates
43
There are discussions of creation of exFAT on a Vista or Windows 7 machine that can’t be seen on Vista. This is usually a case of creating the media on a machine with exFAT support and then trying to read the media on a different machine without exFAT support. The common mistake is creation of the file system on removable media with a Vista SP1 (or higher machine) and trying to read it on a machine with Vista RTM.
44
45
The SDXC media will not be backward compatible\e-solutuions/volkswagon for in vehicle entertainment systems
DCF 2.0 – Design Rule for Camera File System
Camera and Imaging Products Association (CIPA) – DC-009-2010
Japan Electronics and Information Technology Industries Association (JEITA) CP3461BCP3461B
Exchangeable image file format (officially Exif, not EXIF according to JEIDA/JEITA/CIPA specifications) is a standard that specifies the formats for images, sound, and ancillary tags used by digital cameras (including smartphones), scanners and other systems handling image and sound files recorded by digital cameras The specification ses the follo ing e isting file formats ith the additioncameras. The specification uses the following existing file formats with the addition of specific metadata tags: JPEG Discrete cosine transform (DCT)
[1] for compressed image files, TIFF Rev. 6.0 (RGB or YCbCr) for uncompressed image files, and RIFF WAV for audio files (Linear PCM or ITU-T G.711 μ-Law PCM for uncompressed audio data, and IMA-ADPCM for compressed audio data).
[2] It is not supported in JPEG 2000, PNG, or GIF. (Source Wikipedia)
46
This is just a selection of some dates, not exhaustive.
Volkswagon and Audi licensed Tuxera drivers to get exFAT support.
http://www.thesixthaxis.com/2014/04/30/the-hidden-features-and-changes-of-playstation-4-firmware-1-70/
The 1.70 PS4 firmware update is rumored to have exFAT support
47
Tuxera
http://www.tuxera.com/products/tuxera-exfat-embedded/
Provides exFAT & NTFS to be integrated in embedded systems.
They do software development and integrate the support
They have a working relationship with MS, and have licensed exFAT. Their development then gets integrated for other companies.
However, there are developers working on their own to build and generate drives for Linux.
48
The Linux community is very hostile to Microsoft, complaining that a Microsoft standard was forced on them and Microsoft expects to get paid.
However, that is what they are stuck with.
Meanwhile the community is trying to build exFAT drivers for the various Linux distributions in order to support exFAT.
IN 2013 someone leaked the source code to Samsung’s exFAT kernal basedIN 2013 someone leaked the source code to Samsung s exFAT kernal based drivers. Eventually Samsung put those drivers under a GPL license and released them as open source.
Even through there is now a GPL license for Samsung’s exFAT implementation, I would expect that Microsoft will want their royalties and there may be legal issue ahead.
E er since the Tom Tom settlement companies are afraid of iolating Microsoft’sEver since the Tom Tom settlement, companies are afraid of violating Microsoft’s patent rights.
49
50
51
52
I use a CPAP machine, and it has a SD card to record my sleep patterns
These are the uses, many of these devices take USB, SD, CF, and even memory stick
Most of these are embedded systems, but produce or consume media that can/or was processed on desktop systems
53
SANDISK ships high capacity CF cards pre-formatted as exFAT
PNY 128GB Turbo USB flash
DigiStore SSD
Some Kingston Memory
Even some magnet disk media, example Western Digital
Pre-formatted file system is not usually specified in the product specs, so it is hard to determine unless you see discussions or go out and buy & test everything
In 2010 a 64GB SANDISK SDXC card was selling on Amazon for $350, in May 2014 I bought 2 of these at $49.50 each (with free tax & free shipping)
54
New Devices may accept SDXC, but older devices might not.
Mentioning memory cards such as camera cards is important because the target market for exFAT is removable storage.
Back in 2009 I believed that this was going to be a big driver towards exFAT adoption but flash memory in any type of memory card or even SSD drives mayadoption, but flash memory in any type of memory card, or even SSD drives may have contributed to the adoption rate and prevalence.
55
With Sony adopting the XC memory stick to exFAT, plus the SD market, is almost 90% of the market today.
http://anythingbutipod.com/2009/01/next-generation-sdxc-details/
Jan 8, 2009
56
July 2012
57
The 137GiB comes from LBA-28 addressing.
228 = 268,435,456
228 * 29 = 128GiB = 137GB = 137,438,953,472
144PB comes from LBA-48 addressing
248 = 281,474,976,710,656
248 * 29 = 128PiB = 144PB = 144,115,188,075,855,872
58
SD – up to 2GB – FAT
SDHC – 2GB to 32GB FAT32
SDXC – 32GB to 2TB exFAT
These are marketing hype and represent maximums
http://www.dpreview.com/news/2009/1/8/sdxc
http://www.computerworld.com/s/article/9125622/Memory_card_standard_could_provide_up_to_2TB_on_an_SD_card
The 64GB SDXC card, for instance, can store a full 16 hours of 1080p High-Definition footage (1920x1080 9Mbps H.264 AVC compression) or over 4000 RAW images (based on 14MB file size), ideal for situations that demand continuous burst-images (based on 14MB file size), ideal for situations that demand continuous burstmode shooting and non-stop video recording.
Source: http://dk.transcend-info.com/About/press/10044
The size of a photo will vary based on the camera resolution and the effectiveness of compression.
Transcend (from the statement above) indicates 4,000 Raw images based on 14MB file size. This was for a 64GB card, while the SD press releases were saying 4,000 images per 2TB card, something is off.
And in the case of just taking the Nikon D7100 DX, RAW images are really almost59
Nand gate wear and tear, less write, longer the memory may last
Although the SD association states that the official, standard and only file system for SDXC is exFAT, users will format the card using other file systems.
Some cameras may allow a SDXC card to be formatted as FAT32, but others will indicate that the SD Card is not formatted properly and ask to format it.
Yet, a user may format the SD Card in another file system, and use it in a non-compliant device such as a slot on the laptop where only the OS will read and writecompliant device, such as a slot on the laptop where only the OS will read and write the card.
60
Write Endurance (Program Erase Cycles)
Limited, maybe up to a million, writes
Writes require the storage area to be erased first, almost like an EPROM
Flash memory, nand and nor gates, should not be full formatted unless needed – do a quick format. Good for forensics because data is not erased
Degrags should not be done either, flash memory doesn’t have moving parts so and the extra writes wear down the solid state chipsthe extra writes wear down the solid state chips.
Writes need to be limited, the less writing the longer the memory will last and the better the performancebetter the performance
Write Cliff
Blocks are rotated and pre-erased, but if you run out of empty blocks, then you have to wait for a block to be erased before you can reuse it for a different set of values.
Wear Leveling
Methods of providing a block of gates, and rotating through the gates.
61
Why a 2TB limit when the CF has a 144PB limit?
Looks like SDXC uses a MBR partition to separate the protected area from the user data area, and that limits volume to 2TB
The follow-on the SDXC will need to use a GPT, a partition would probably still be required with a separate protected area.
CD does not have DRM/copy protection, so extra partition probably not required. Then issue is superfloppy or GPTThen issue is superfloppy or GPT
Format of a SDXC card puts standard boot code in the boot sector, while cards from the factory ha all those fields filled up with F4
Need to format card using SD Formatter utility and inside the camera – both cases to see what is going into the sector.
AU Sizes vary based on size of total volume.
62
63
Currently use exFAT 1.00, but if a later version of exFAT is in use, it will check the version # and not mount the FS unless it can suppoort it
Checksums protect against corruption and viruses
If there is a problem with critical directory entries, the FS should not mount.
The dirty flag used to be in the 2nd FAT index in FAT32
64
65
4 Regions defined on the volume
The FAT tables reside outside the cluster heap
Everything except the data region is measured in and addressed as sectors.
Data region is measured and addressed as blocks, blocks are called clusters
66
FAT and Cluster heap have their own offsets, which allow alignment, if needed to force these region on a designated boundary line
Might be needed in SD and other flash memory
The specification for exFAT says the # of FATs is either 1 or 2.
For legacy FAT it is recommended to be 2, could be 1, but could even be more although rarely seenalthough rarely seen.
Since there are offsets, I could build a 3rd, 4th or more FAT – just stick space between the 2nd FAT and the cluster heap start and have an area of slack space.
67
Details follow in the next slides
A mirror of the VBR follows, and is a backup VBR
In case the first gets corrupted
FAT32 had a mirror also, the mirror was at sector 6
68
When you take the volume length (64-bit) * 4K sector, that is 64+12 = 276.
270 = 1 ZiB, 26 = 64, thus based on this value, a file system of 64 ZiB. However, the current architecture specification cannot produce a file that big unless some parameter somewhere gets changed.
69
If there was no restriction, then the size of a cluster could be 4255
70
If the sector size is > 512 bytes, all space on the first sector of the VBR (Main Boot Sector) is not used (Only the first 512 bytes)
71
Unlike the first sector, the other 8 boot sectors can use the entire sector and the signature marker is moved to the last 8 bytes of the sector
72
If a virus modified the boot record, and doesn’t fix the checksum, there should be a mount failure
73
Repeats over and over again, 4 bytes = 32 bit checksum
Can be used to determine if the VBR was modified
3 bytes in the VBR are not calculated in the checksum
This sector does not have a signture
74
The BITMAP is used to track cluster allocation, and the FAT is only required for re-assembling the original file. If the original file is contiguous, then the FAT isn’t needed for THAT file. We will see later that a flag in the directory record is used to tell the FS whether the FAT should be used or ignored.
For Contiguous Cluster Allocation see Patent: US8606830B2 (Contiguous File Allocation in an Extensible File System)
For TexFAT see Patent: US7613738B2 (FAT Directory Structure for use in Transaction Safe File System)
For Extensible File System see Patent: US8583708B2 (Extensible File System)
75
Because there is no floppy support, there is only one possible media descriptor value
Cluster 0 and 1 are not defined, so 0 & 1 are not significant (Same as legacy FAT)
Since the FAT is no longer used for cluster allocation, 0 (zero) is no longer significant (used to mean the cluster was unused/free/unallocated)
FF..F9 thru FF..FE were also EOC (end of cluster chain) markers in FAT32, but are unused in exFATunused in exFAT.
Values 2 thru FF..F6 are cluster addresses.
The Cell’s content is called an index or indices
1st indices contains FFFFFFF8 (Media Descriptor)
2nd indices contains FFFFFFFF and is not used to hold dirty volume flags
76
77
The 3 main critical records: Allocation Bitmap, UP-Case Table, and Root Directory will use FAT chains.
The Root Directory can grow and since it is dynamic in its growth, most likely will fragment.
The UP-CASE Table and Allocation bitmap should be static and not grow or change, although theoretically they could probably be relocated and moved somewhere else on the volume.
However, in the SD standard, the allocation bitmap must be within the first 4MB of the cluster heap.
The locations (cluster addresses) of the 3 special metadata files may change, this is based on one formatting and in reality these files could eventually end up in any cluster.
These 3 special files (4 if TexFAT) will use chains even if unfragmented becauseThese 3 special files (4 if TexFAT) will use chains even if unfragmented because even though they appear in the cluster heap, they are not true user files, and are defined by special directory records and not file records to point to these special system files. These special files don’t have the INVALID FAT flag.
78
If there are 2 FATs in a TexFAT Transactional Safe exFAT environment, then each FAT is paired with a allocation bitmap
The allocation BITMAP is pointed to by a 0x81 entry.
To locate an empty cluster, each 512 byte sector would hold 4096 allocation bits (512 bytes x 8 bits)
More efficient but still require reading many sectors if the FS is large and the earlyMore efficient, but still require reading many sectors if the FS is large and the early part of the FS is allocated.
79
When files on legacy are fragmented, and deleted, the deletion wipes out the FAT chain because each FAT cell has to be zeroed out to indicate that the cell is no longer allocated.
Since allocation is moved from the FAT to the Allocation Bitmap, the FAT cells for the chain remain intact.
There is still the possibility of cells being overlaid by other file fragments after the delete.
This is not in the spec, buit this behavior has been observed.
80
If the files are made larger, then code can be hidden into those files
The SD Specification indicates that the Allocation Bitmap must be within the first 4MB of the file system.
81
This is an eye chart, but the idea is to show how to get to the bitmap.
You start at the VBR (BPB), go to the root directory, look up the 0x81 entry to get the cluster address, and then go into the BITMAP table.
The first byte of a directory record is the entry type, here we see x’83’, X’82’ and X’81’
82
We will see details of the directory entry construction later, including what we mean by an entry type.
If there are issues with the critical entries, then the file system should not mount.
In FAT the largest directory size is 221 Which equals 2,097,152
http://read.pudn.com/downloads77/ebook/294884/FAT32%20Spec%20(SDA%20Contribution) pdfntribution).pdf
In FAT, with a directory size of 221 and a director record size of 25 (32 bytes) the number of 32 character entries in the directory is 216 = 65,536
Keep in mind that the maximum number of files will be related to the FAT itself since not more than a single file can occupy a cluster.
In exFAT the largest directory size is 228 = 256MiB
exFAT directory size limit is 27 (128) times the size of the FAT limitation
83
Benign directory records
Fake secondary records
Zero length/zero cluster files
Phantom / orphan Files
84
The first byte of every directory entry is the “entry type” and describes the directory entry.
85
When a file set is not in use, it is usually (but not always) a deleted file
When a volume label is not in use, it means no volume label
In a file set, it could be caused by renaming a file with a longer file name.
Only files have secondary entries so far
Missing Benign entries usually won’t prevent the file system from being mounted.
0x80 is not defined.
In FAT32, file deletion is done by overlaying the first byte with 0x’E5’, or X’05” if the first byte of the filename is already an 0x’E5’.
For KANJI character set based names, the value 0x05 is stored in DIR_Name[0] - if required - to represent 0xE5.
86
Primary and Critical
In legacy FAT, the Volume label is in the Root Directory, and has an attribute
87
Since we use 16 bit unicode without string termination, we need the length of the volume label – in unicode characters.
88
Primary and Critical. If the FS can’t find the BITMAP table, it can’t mount the FS
Since there is no flag, this file will always have a FAT chain, even if it is one cluster, will always have a cluster chain ending with EOC
89
This was a small volume. 63 bytes can support maximum of 63x8 = 504 clusters.
90
Filenames are stored case insensitive, so when a search is done, the filenames are converted to upper case (folded). The UP-CASE table is used to convert the filename to all uppercase.
91
The UP-Case table is less than 6K – imagine if it was in a 32K cluster, now imagine if it was in a 32MB cluster, the amount of available slack space.
92
File Entry Set would have a File, Stream Extensions, and up to 17 File Name Extension for a total of 19.
Later, when a new exFAT version comes out, the ACL will be another secondary entry bringing this up to 20.
As more file secondary entries are added, let’s say one for encryption, this increases to a max of 255 secondaries.
93
Attributes and Timestamps in later slides
Checksum is across the Primary and all secondaries in the set.
94
Modified, Access, and Create.
Timestamps are NOT stored in this order, but MAC is a common acronym in the literature.
Timestamps are not one single field like NTFS which uses a 64 bit value. exFAT combines pieces to make a UTC value.
TZ offset is absent in Vista SP1, and does not appear in the exFAT 1.00 spec.
Note: By default, the creation time is tunneled if a file is deleted, and a file with the same name is created within 15 seconds.
(See KB172190 http://support.microsoft.com/kb/172190)
95
The standard DOS Date/Time, also used in the previous FAT versions, does not count to the second, but double seconds.
To get seconds, a 33 bit number would have been needed.
The OS doesn’t always update last access.
And even NTFS last access is disabled in some versions, can modify behavior with” fsutil behavior set disablelastaccess 0fsutil behavior set disablelastaccess 0
Not sure if FAT32/exFAT is relaiable
96
97
FAT and exFAT timestamp behavior varies, but is just not reliable as far as last accessed.
TSK research shows some differences between OS, so timestamp analysis could be very inconsistent
Even in later Windows releases, NTFS doesn’t even update the Last Accessed on READ for performance reasons but this behavior can be restored via a registry keyREAD for performance reasons, but this behavior can be restored via a registry key.
98
These are pretty much the same as previous FAT versions.
Since we have a separate volume label entry, there is no attribute for it, and since we don’t have 8.3 support, there is no LFN (Long File Name) attribute either because everything is LFN.
Reserved1, which is mask 0x08 was ATTR_VOLUME_ID (0x08) in legacy FAT
99
The update behavior on the 10ms Modified is also not predictable, sometimes it is just set to zero.
Note that the create time is really 3B866244 (reversed because of little-endian)
100
In order to validate the analysis in reverse engineering the FS, I had to write a C program to format the directory entries.
This is an example of the output.
All the timestamps are even because of the double seconds. But since the create is 168, this means that the create time was really 12:18:09.68
Secondary count is 4, meaning that this file set is 5 entries, 1 File, 1 Stream, and 3 filenamefilename.
101
There is 2 file lengths, one is supposed to be the physical file length and the other the amount of data actually written into the file so far (Valid Data Length - VDL)
These are two 64 bit length and are similar to the two lengths in NTFS in the $FILE_NAME 0x30 attribute.
Length of name is needed because there is no string termination, but the file name (max 255) may require multiple directory entries (we will see later).
This is where the FS indicates whether the FAT is used if the FAT Invalid flag is setThis is where the FS indicates whether the FAT is used, if the FAT Invalid flag is set, then the FAT is ignored.
In legacy FAT
The cluster number of the first cluster of the file is recorded in the directory entry associated with
the file. For zero-length files, the first cluster number in the associated directory entry is set to 0.
exFAT also supports a first cluster of zero if the length is zero.
One of the Lengths is called “DataLength” Field
102
One of the Lengths is called DataLength Field
The specification states: If the corresponding file directory entry describes a directory, then the valid value for this field is the entire size of the associated allocation, in bytes, which may be zero. Further, for directories, the maximum value for this field is 256MB. The other length field is called “ValidDataLength” field says that if this is for a directory, then this value must match the DataLength field.
103
Since these values can vary based on the format parameters, for reference this is what the samples in this presentation is using.
104
Another output from the C program.
Allocation possible indicates that the directory entry specifies a cluster address field
FAT invalid indicates that this file does not use the FAT
This file is 18MB and required 143 clusters to store the file.
As we said before, there are 3 filename entries (each holds 15 characters of the filename), and as we see above, the filename is 40 characters in length.
105
Allocation not possible indicates that there is no cluster address in the entry.
FAT Invalid has no meaning
106
Filename is 40 characters (80 bytes) and takes 3 entries to store it.
Notice that in Uni-Code the file name is stored in mixed case
107
When the entries are not in use, some may be overwritten, and some may not. This means that a complete set may not exist.
108
109
There are discussions of creation of exFAT on a Vista or Windows 7 machine that can’t be seen on Vista. This is usually a case of creating the media on a machine with exFAT support and then trying to read the media on a different machine without exFAT support. The common mistake is creation of the file system on removable media with a Vista SP1 (or higher machine) and trying to read it on a machine with Vista RTM.
Microsoft distributes a specification, each vendor writes their own drivers, so variations between vendors, causing compatibility issues are occurring
Users try for format drive on Windows system, drive is >32GB, the only options are: exFAT & NTFS, they format in exFAT and then find out their device doesn’t work in other places due to lack of exFAT support.
The drive of a user to get away from a device in FAT32 is the 4GB barrier.
110
New, but 8 years old, misunderstood
More forensics tools need exFAT support
Implementations across vendors are inconsistent and might not implement all features
Needs to be fixed before it gets worse
Even utilities for Disk Partition, Defragmentation, File Recovery, and commands like CHKDSK need exFAT supportCHKDSK need exFAT support
More evidence is going to show up in exFAT format, need to acquire the right tools and get experience
111
112
I need followers
113
Since NTFS has a smaller maximum cluster size (64K) 216 (29 * 27 = 216) while exFAT maximum is set to 225. Then the question is: What happends to NTFS?
114
http://www.snia.org/sites/default/files2/SDC2012/presentations/File_Systems/JRTipton_Next_Generaltion-3.pdf
115
My paper on exFAT and the Microsoft Patent that exposes the specification
116
117
I encountered these other sites that have information on exFAT.
I include them here to provide more information for the reader
118
119
120
121
Top Related