1 ©All rights reserved to JSOF Ltd.
Reverse Engineering Archeology:
Multiple Devices, Multiple Versions
CONFidence 2020- September 8th, 2020
©All rights reserved to JSOF Ltd.
Who are we?
2
JSOF is a software security consultancy
• Shlomi Oberman, co-founder, JSOF
• Moshe Kol, Security researcher, JSOF; Finder of Ripple20
• Ariel Schön, Security researcher, JSOF
©All rights reserved to JSOF Ltd.
Agenda
• Ripple20
• Reverse engineering process:
• Multiple binaries
• Wrap-up
4
©All rights reserved to JSOF Ltd.
Ripple20
5
• Series of 19 zero-day vulnerabilities in Treck TCP/IP*
• Amplified by the supply chain
• 100’s of millions of devices
• Medical, ICS, Home, Enterprise, Transportation, Utilities
©All rights reserved to JSOF Ltd.
Ripple20
6
CVE-2020-11896
CVE-2020-11897
CVE-2020-11898
CVE-2020-11899
CVE-2020-11900
CVE-2020-11901
CVE-2020-11902
CVE-2020-11903
CVE-2020-11904
CVE-2020-11905
CVE-2020-11906
CVE-2020-11907
CVE-2020-11908
CVE-2020-11909
CVE-2020-11910
CVE-2020-11911
CVE-2020-11912
CVE-2020-11913
CVE-2020-11914
• 4 critical remote code execution vulnerabilities
©All rights reserved to JSOF Ltd.
100’s of Millions of Devices Affected
7
And many more...
©All rights reserved to JSOF Ltd.
Ripple20 Research
8
• Reverse engineering 7 different devices with multiple versions
• Every device has a different configuration
• Ongoing research Sep’19 - Jun’20 ( 9 months )
• Some strange architectures and firmwares involved
2 whitepapers released (CVE-2020-11896/CVE-2020-11901)
©All rights reserved to JSOF Ltd.
Challenge
• 1 library many versions
• Little did we know…
• Need symbols, debug, binary…
• Multiple data points
• Lots of history
9
©All rights reserved to JSOF Ltd.
Challenge
• Multiple firmwares/binaries
• Security/Archeology project
• Library dating to pre-2000
10
©All rights reserved to JSOF Ltd.
How did we start?
• Browsing to Treck’s website
• Looking for datasheets, manuals, demos
11
12 ©All rights reserved to JSOF Ltd.
Binary #1 – Freescale demo
©All rights reserved to JSOF Ltd.
Freescale 5280 demo
• Contains headers and static library
• The headers provide useful comments and structure definitions.
13
©All rights reserved to JSOF Ltd.
Freescale 5280 demo
• Static library contains 202 object files:
tr8023.o trarp.o trarpchk.o trautoip.o trbase64.o trbootp.o trbtdhcp.o trbuffer.o trcmplib.o trdevice.o
trdhcp.o trdialer.o trdsplib.o treap.o trethcom.o trether.o trethtag.o trfs.o trftp.o trftpd.o
trhttp.o trhttpd.o tricmp.o trigmp.o trindrmc.o trindrv.o trinscdr.o trip.o tripfrag.o triphc.o
triptunl.o trlist.o trlock.o trlog.o trloop.o trlqm.o trmime.o trmoblip.o trmschap.o trnat.o
trnetid.o trntstat.o trping.o trpop3.o trppp.o trramfs.o trrelay.o trresolv.o trrip.o trromfs.o
…
• Architecture: Motorola 68030 big-endian
14
©All rights reserved to JSOF Ltd.
Freescale 5280 demo
• Object files have function names:
trip.o 15
©All rights reserved to JSOF Ltd.
Freescale 5280 demo
• But non-local function calls are missing:
16
©All rights reserved to JSOF Ltd.
Freescale 5280 demo
• Some can be recovered using the relocation table:
17
©All rights reserved to JSOF Ltd.
Freescale 5280 demo
• In summary:
• Useful data point
• Cannot be debugged easily
18
19 ©All rights reserved to JSOF Ltd.
Binary #2 – Win32 demo
©All rights reserved to JSOF Ltd.
Win32 Demo
• Treck (used to) offer Windows 32-bit demo app
20
©All rights reserved to JSOF Ltd.
Win32 Demo
• Supports many useful features:
• IPv4
• IPv6
• DHCP client
• TCP
• UDP
• ICMP
• IPSEC
• Mobile IPv6
21
©All rights reserved to JSOF Ltd.
Win32 Demo: Finding Treck
• No debug symbols.
• Able to recover some function names using debug strings:
• Applies mostly to IPv6 functions
22
©All rights reserved to JSOF Ltd.
Win32 Demo: Finding Treck
• To locate the IPv4 code base, we searched for EtherType constants in the binary.
• Recall Ethernet packet format:
EtherType Protocol
0x0800 IPv4
0x0806 ARP
0x86dd IPv6 23
©All rights reserved to JSOF Ltd.
Win32 Demo: Finding Treck
• Using this technique we were able to locate tfEtherRecv.
x86 is little-endian architecture!
24
©All rights reserved to JSOF Ltd.
Win32 Demo: Results
• We reverse engineered large parts of the network stack
• We found some vulnerabilities
• We wanted to test if other devices are affected
25
26 ©All rights reserved to JSOF Ltd.
Binary #3 - Digi dev board
©All rights reserved to JSOF Ltd.
Digi Connect ME 9210
• A “Veteran of the Digi Community” mentioned online that Digi Connect ships with Treck TCP/IP stack in Digi forum:
27
©All rights reserved to JSOF Ltd.
Digi Connect ME 9210
• Digi Connect ME devices come in two flavors:
• Running embedded Linux
• Running proprietary NET+OS
• The network stack of NET+OS 7.5 is Treck TCP/IP.
• We bought the Connect ME 9210 development kit.
28
©All rights reserved to JSOF Ltd.
Digi Connect ME 9210
• Runs Digi's new 32-bit NS9210 processor (ARM9).
• Have debugging capabilities using JTAG.
• Comes with eclipse-based IDE to write software:
29
©All rights reserved to JSOF Ltd.
Digi Connect ME 9210
• We compiled some basic example and examined the resulting ELF file.
• ELF comes with debug symbols!
• We developed an exploit for CVE-2020-11896 on this device.
• Disadvantage: relatively old Treck version (4.7).
30
31 ©All rights reserved to JSOF Ltd.
Binary #4 – Intel AMT
©All rights reserved to JSOF Ltd.
Intel ME
• In a quest for newer versions, we looked at Intel ME.
• Treck powers the AMT module.
• We speculated that since Intel is a security-aware company, they must have updated their Treck software.
32
©All rights reserved to JSOF Ltd.
We thought we had 1-days
• Intel binary had some “defensive programming”
• We thought we had 1-days that still existed in the wild (we were mostly wrong)
• Maybe fixes, maybe ifdef, maybe they are paranoid
33
©All rights reserved to JSOF Ltd.
Intel ME: Patch-diffing
• INTEL-SA-00241 describes a vulnerability that looks related:
• We wanted to patch-diff AMT versions to find it.
34
©All rights reserved to JSOF Ltd.
Intel ME: Patch-diffing
• We obtained two ME firmware versions:
• Intel ME 12.0.32.1421 Corporate/5MB
• Intel ME 12.0.55.1521 Corporate/5MB
• Used the ME Analyzer tool to unpack the firmware and extract the AMT module.
35
©All rights reserved to JSOF Ltd.
Intel ME: Patch-diffing
• We used BinDiff as our patch-diffing tool.
• Challenge:
• diff is large.
• We want to focus on Treck-related code only.
36
©All rights reserved to JSOF Ltd.
Intel ME: Finding Treck
• tfUseEthernet initializes the Ethernet link layer.
Two references for the string “ETHDIX”.
Initializes a struct with function pointers – tfEtherRecv among them.
*Decompiled code taken from the Digi Connect device
37
©All rights reserved to JSOF Ltd.
Intel ME: Finding Treck
• We signed the tfUseEthernet function structure.
• Using the “ETHDIX” string we found the image base address.
• We developed a Ghidra script to mark Treck-related code, then extracted Ghidra symbols to IDA for diffing.
38
©All rights reserved to JSOF Ltd.
Marking Treck-related code
• Traverse call-graph from known-Treck entry points.
• Function pointers in tfUseEthernet as entry points.
• Luckily, library functions reside in separate module(s).
• To gain more coverage we considered parents of functions with many xrefs (e.g. tfLock).
tfEtherRecv
tfFreePacket
tfFreeSharedBuffer
tfArpIncomingPacket
tfFreePacket
tfArpResolveSearch
tfLock
…
tfIpIncomingPacket
…
tf6IpIncomingPacket
…
39
©All rights reserved to JSOF Ltd.
Intel ME: A vulnerability
• A fixed bug was found in the DHCPv6 client:
• During option 24 processing in tf6DhcpSaveReplyInfo.
• Function accepts single argument (buff) and computes the total label length.
• Matches the description of CVE-2019-0131 shown earlier: • Adjacent access
• Infoleak/DoS
40
©All rights reserved to JSOF Ltd.
Intel ME: A vulnerability
• We also found that Treck got the fix wrong:
• Still OOB access.
• We reported the issue to Treck. This is CVE-2020-11905.
41
©All rights reserved to JSOF Ltd.
1 days, 0 days, Any-days
• Some of the vulnerabilities fixed only in Intel. Also, Intel has exploit mitigations.
• Digi had old code; Intel had new code. Intel had some code (no DNS)
• Until disclosure, we thought some bugs were 1-days and Intel was most updated.
• Treck told us they are 0-days. The story of AMT is unclear.
42
©All rights reserved to JSOF Ltd.
1 days, 0 days, Any-days
• Few types of Treck supply-chain vulnerabilities:
• True 0-days
• 0-days only fixed in AMT code (to our knowledge)
• N-days that exist in the wild and fixed upstream - Any-days • Never publicly reported as far as we know
• We don’t know if considered security fix previously
• Support package updates
• No support no security
43
44 ©All rights reserved to JSOF Ltd.
Binary #5 – HP printer
©All rights reserved to JSOF Ltd.
HP OfficeJet Pro 8720
• Searching for some common Treck function names on Google yields interesting results.
• We found that some HP printers run Treck.
• We wanted to check if they are affected by the vulnerabilities.
45
©All rights reserved to JSOF Ltd.
HP firmware unpacking
• RFU (remote firmware update) file obtained from HP’s public FTP server.
• Multi–stage unpacking.
• Bizarre file formats.
• Whole process is described in our blog.
Unpacking PJL/PCL layer
Unpacking binary S-Records layer
Removing Flash OOB data
Locating firmware section table
Dump decompressed section
46
©All rights reserved to JSOF Ltd.
HP OfficeJet Pro 8720: A crash
• We found that CVE-2020-11896 crashes the printer.
• Apparently, there is an ifdef that disallows fragmented data over an IP-in-IP tunnel.
• However, sending those packets cause tfKernelError to run.
• Vulnerability variant.
Crash!
47
48 ©All rights reserved to JSOF Ltd.
Binary #6 – APC UPS
©All rights reserved to JSOF Ltd.
Schneider Electric APC UPS
• Downloaded firmware update files from APC website.
• Not encrypted/compressed.
• Reverse engineered some parts of the file format:
• Image base address
• CRC16 fields
49
©All rights reserved to JSOF Ltd.
The AOS binary
Header crc file crc
Image base
Image base + 0x400
File size
0x400
50
©All rights reserved to JSOF Ltd.
The AOS binary: Which processor?
• Loaded into Ghidra.
• Choosing 16-bit x86 protected mode “kind of” works.
• Disassembler cannot resolve far-calls.
• Obscure architecture.
51
©All rights reserved to JSOF Ltd.
The AOS binary: Which processor?
• Strange memory addressing.
• Can’t be protected mode - too many segments, no GDT.
• Can’t be real mode - shifting by 4 does not work.
• We opened the old books to find olden x86 witchcraft – no luck.
• Do you have a moment to learn about unreal mode?
52
©All rights reserved to JSOF Ltd.
The AOS binary: Which processor?
• A pattern emerges when looking at the strings/function calls (but mostly strings).
• LSB of a pushed string corresponds to LSB of the offset of the string within the binary.
• Shifting the segment word by 8 does the trick. We saw that we always land on a function this way. LinearAddress = (segment << 8) + offset
push cs push 0xf2e push cs push 0xf1b call 0xc466:0x382 tfKernelError
53
©All rights reserved to JSOF Ltd.
The AOS binary: Loading into RE tool
• We must fix the far-call issue to reverse engineer the firmware.
• We tried to change Ghidra’s processor module, recompile it.
• Only partial success, no strings.
• We tried to specify the segment granularity on radare2 – better, still lacks strings.
54
©All rights reserved to JSOF Ltd.
The AOS binary: Loading into RE tool
• We found someone who faced the same issue on http://www.openrce.org/forums/posts/753.
• Mystery solved: Turbo186!
• Solution: use IDA’s segment selectors.
• Thanks igor skochinsky.
55
©All rights reserved to JSOF Ltd.
The AOS binary: Loading into IDA
• We wrote an IDA python script to create segment selectors which emulate the “shifting by 8”.
• Now we have strings, switch statements, far-calls working.
• We can start reverse engineering.
• No decompiler for 16-bit x86.
56
©All rights reserved to JSOF Ltd.
The AOS binary: Heap Functions
57
• Even after segment fixing, many far-calls point to non-mapped regions
• Comparing with Digi firmware, we concluded these are far-calls to heap utility functions
• malloc(), free(), etc.
• The binary contains debug strings with the function names
• But without references…
• Because of the 8-bit segment shift, we can search for undefined push instructions
• Found and re-mapped these functions to their proper dynamic location
©All rights reserved to JSOF Ltd.
New Vulnerability
• Found newer Treck version than Digi’s.
• Also new vulnerability (CVE-2020-11901: Bad RDLENGTH).
• Bad fix for a previously found vulnerability.
• Didn’t exist in AMT because they don’t use this feature.
58
59 ©All rights reserved to JSOF Ltd.
Binary #7 – GE MDS
©All rights reserved to JSOF Ltd.
Bonus Binary: GE MDS
• General Electric communication device for utilities (water/power).
• Used Google search + n-gram slices to find the architecture
• cpu_rec works too!
• Runs Blackfin processor and uses Treck.
• Didn’t use extensively, didn’t teach us anything new.
60
61 ©All rights reserved to JSOF Ltd.
Wrap-up
©All rights reserved to JSOF Ltd.
Take-aways
• Supply chain is complicated
• Obscurity doesn’t work
• Well, mostly.
• Know your upstream, patch your upstream
• Deeper in the supply-chain higher impact
62
©All rights reserved to JSOF Ltd.
Take-aways
• Software providers security SLA
• Report security issues?
• Timeline?
• Product support vs security support?
• Two way? What if user finds a vulnerability?
• Proprietary vs. OSS?
63
©All rights reserved to JSOF Ltd.
Inconsistent patching
• One vendor patches and another doesn’t.
• Patch-gapping on steroids!
64
©All rights reserved to JSOF Ltd.
Conclusions
• Complex reverse engineering process
• Forks in software library can unveil more vulnerabilities
• Supply chain makes security difficult
• Proprietary update process is obscure
65
66 ©All rights reserved to JSOF Ltd.
Thanks for listening! [email protected]
Top Related