Cache on deliverymining memcached

marco@sensepost.com

• Large percentage of data remains relatively constant

• Wikipedia page contents

• Youtube video links

• FB Profile data

• Poorly designed solutions regenerate data on each request

• Don’t regenerate, rather regurgitate

The need for caching

• Large percentage of data remains relatively constant

• Wikipedia page contents

• Youtube video links

• FB Profile data

• Poorly designed solutions regenerate data on each request

• Don’t regenerate, rather regurgitate

The need for caching

Memcached

• memcached.org

• Written for LJ (2003) by Brad Fitzpatrick

• Non-persistent network-based KV store

• Why do we care? Mom&pop don’t need the cache.

Memcached

• memcached.org

• Written for LJ (2003) by Brad Fitzpatrick

• Non-persistent network-based KV store

• Why do we care? Mom&pop don’t need the cache.

Basic KV

• Slabs are fixed size

• Dst slab determined by value size

• Users don’t care about slabs

• Miners care about slabs

Trivial protocol

• set• get• stats

• ...• ????

Binary and UDP protocols also exist, these were not touched.

• ASCII-based• Long-lived• Tiny command set

Trivial protocol

• set• get• stats

• ...• ????

Binary and UDP protocols also exist, these were not touched.

• ASCII-based• Long-lived• Tiny command set

Trivial protocol

• set• get• stats

• ...• ????

Binary and UDP protocols also exist, these were not touched.

• ASCII-based• Long-lived• Tiny command set

Trivial protocol

• set• get• stats

• ...• ????

Binary and UDP protocols also exist, these were not touched.

• ASCII-based• Long-lived• Tiny command set

Trivial protocol

• set• get• stats

• ...• ????

Binary and UDP protocols also exist, these were not touched.

• ASCII-based• Long-lived• Tiny command set

1998

• Blank ‘sa’

• Anonymous ftp

• system/manager

Goals

• Connect to memcached

• Find all slabs

• Retrieve keynames from each slab

• Retrieve each key

Lies, damn lies, and stats

• stats cmd has subcmds

• items

• slabs

• ...

This gets us the slabs_ids

stats slabsSTAT 1:chunk_size 80<...>STAT 2:chunk_size 104<...>STAT 3:chunk_size 136<...>STAT 4:chunk_size 176<...>STAT 6:chunk_size 280<...>STAT 8:chunk_size 440<...>STAT 9:chunk_size 552<...>STAT 9:cas_badval 0STAT active_slabs 7

Retrieving key names

Rely on two {poorly|un}documented features

Retrieving key names

Feature #1:

Remote enabling of debug mode

Retrieving key names

Feature #2:

“stats cachedump”

Retrieving key names

Feature #2:

“stats cachedump”

Retrieving key names

Feature #2:

“stats cachedump”

Slabs IDSlabs ID

Retrieving key names

Feature #2:

“stats cachedump”

Key limitKey limit

Retrieving key names

Feature #2:

“stats cachedump”

Key listKey list

Retrieving key names

Feature #2:

“stats cachedump”

This gets us key names

And this gets us?• No need for complex hacks. Memcached serves up

all its data for us.

• What to do in an exposed cache?

• Mine

• SQLi is too hard for me

• Overwrite

• Client-side

• Server-side

Mining the cache

• go-derper.rb – memcached miner

• Retrieves up to k keys from each slab and their contents, store on disk

• Applies regexes and filters matches in a hits file

• Supports easy overwriting of cache entries

• [demo]

Two issues

Two issues

• Finding caches

• Again with the simple approach

• Pick a cloud network, scan for memcacheds on port 11211 with a mod’ed .nse

Two issues

http://www.rhythm.com/~keith/autoStereoGrams/vortexas.gif

• Linking apps to caches

• Who’s %$!#ing cache is this?

• Cached high scores suck. Where’s the good stuff?

• Is it live?

Results #1

IPs scanned 2^16# of caches found 229Retrieved Items 7.3GBAverage uptime ~50days

Total bandwidth used 9PBTotal entry count 288 millionTotal Bytes stored 136TB

Highest bandwidth 247TBHighest entry count 133 millionHighest Bytes Stored 19.3GB

Results #2

• HTML

• JavaScript

• Data

• Email

• Passwords (clear-text, crypt’ed, MD5)

• Objects found

• Serialized Java

• Pickled Python

• Ruby ActiveRecord

• .Net Object

• JSON

Globworld

Globworld

Globworld

Globworld

Gowalla

Gowalla

Gowalla

Gowalla

Gowalla

Gowalla

Bit.ly Pro

Bit.ly Pro

Bit.ly Pro

PBS

PBS

PBS

PBS

Sidebar: serialized objs

• Python’s pickle intentionally insecure

• But they’re exposed!

• Pickle shellcode

cossystem(S'echo hostname'tR.

• [demo]

Sidebar: serialized objs

• Python’s pickle intentionally insecure

• But they’re exposed!

• Pickle shellcode

cossystem(S'echo hostname'tR.

• [demo]

Fixes?• FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW.

FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. (VPC)

• Hack code to disable stats facility (but doesn’t prevent key brute-force)

• Hack code to disable remote enabling of debug features

• Switch to SASL

• Requires binary protocol

• Not supported by a number of memcached libs

• Salt your passwords with a proper scheme (PHK’s MD5 or Bcrypt)

• Also, FW.

Random thoughts

• This can’t be new

• Inject tracker images / strings

• Trace Refers / hit Google

• Key guessing or prediction

• Your data ends up in places you never expected.

Places to keep looking

• Improve data detection/sifting/filtering

• Spread the search past a single provider

• Caching providers (?!?!)

• Other cache software

• Other infrastructure software

Questions?

www.sensepost.com/labs/tools/poc/go-derper