IO-DSSE: Scaling Dynamic Searchable Encryptionto Millions of Indexes By Improving Locality

Ian MiersThe Johns Hopkins University Department of Computer Science

imiers@cs.jhu.edu

Payman MohasselVisa Research∗

pmohasse@visa.com

Abstract—Free cloud-based services are powerful candidatesfor deploying ubiquitous encryption for messaging. In the case ofemail and increasingly chat, users expect the ability to store andsearch their messages persistently. Using data from a major mailprovider, we confirm that for a searchable encryption schemeto scale to millions of users, it should be highly IO-efficient(locality) and handle a very dynamic message corpi. We observethat existing solutions fail to achieve both properties simulta-neously. We then design, build, and evaluate a provably secureDynamic Searchable Symmetric Encryption (DSSE) scheme withsignificant reduction in IO cost compared to preceding workswhen used for email or other highly dynamic material.

I. INTRODUCTION

The last few years have seen incredible success in deploy-ing seamless end-to-end encryption for messaging. BetweeniMessage, WhatsApp, and SRTP for video, over 2 billionusers routinely encrypt their messages without necessarily evennoticing. Unfortunately, this trend has been largely limited toephemeral or semi-ephemeral communication mediums. Whereusers expect messages to be available for later recall, we haveseen little progress.

The quintessential example of this is email, where messagesare searched for months or years after being received. But emailis not the only medium filling this role: Google Hangouts, Slack,and Facebook Messenger all operate in an archive and searchparadigm—indeed the latter two are to some extent used asan email substitute because of this. In this setting, end-to-endencryption is not seamless: by using it, users lose the abilityto store messages in the cloud and search. Since users areunwilling to sacrifice features for security, this is a majorimpediment to deploying end-to-end encryption.

Symmetric Searchable Encryption (SSE) [5, 7, 8, 10, 15, 19]provides a potential solution to this problem as it allows a clientto outsource an encrypted index of documents (e.g. emails)to an untrusted server and efficiently search the index forspecific keywords. The efficiency of SSE stems from its useof fast symmetric-key operations and its privacy guarantees

∗ Work done while at Yahoo Labs.

typically allow for some information leakage on search/accesspatterns which is captured using a leakage function. Thedynamic variants of SSE (DSSE) [4, 13, 14, 20] also allowfor efficient updates to the encrypted database. The question iswhether dynamic SSE can be used efficiently in a cloud-scaleapplication?

Several works [4, 5] have addressed scaling SSE schemesto a very large index—terabytes of data—but to the best of ourknowledge, none have examined deploying millions of smallindexes on shared hardware. This is the exact problem facedby a cloud provider wishing to deploy search for encryptedmessaging or email without deploying large amounts of newhardware.

Since SSE schemes typically use fast symmetric cryp-tography, the primary performance issue for SSE is IO andspecifically poor locality: unlike a standard inverted index wherethe list of document IDs for a given keyword k can be stored ina single location, SSE schemes typically place each documentID for a given keyword in a distinct random location in orderto hide the index structure. This results in a large increase inIO usage, since searching for a keyword found in, e.g., 500documents results in 500 random reads, rather than one singleread retrieving a list of 500 document IDs. Similarly, insertinga batch of, e.g., 100 documents containing some 150 totalkeywords requires 100× 150 = 15, 000 writes, rather than 150writes, one per keyword.

While the same locality issues arise in SSE schemes forvery large indexes, the solution space is far more constrainedin our setting and the main techniques introduced in themost promising approaches [4, 5] are not applicable for purelydynamic indexes (i.e. indexes that are initially empty). As aresult, deploying existing SSE schemes for search for mail ormessaging—where every entry is inserted dynamically into aninitially empty index—-would result in an order of magnitudeincrease in IO usage. Since existing mail search is already IObound, the cost of doing so is prohibitive. This is a markeddeparture from the cost of deploying end-to-end encryption forephemeral communication, which comes at little operationalcost.

Unfortunately, this increased cost cannot be handled bysimply using better hardware. First, high performance storagesystems are prohibitively expensive relative to profit fromfree communications services. Second, caching is relativelyineffective as each individual index is used infrequently. Third,even given cost effective storage (e.g. much cheaper SSDs)and the willingness of providers to upgrade infrastructure, anyinfrastructure-related improvement yields similar cost savings

Permission to freely reproduce all or part of this paper for noncommercial purposes is granted provided that copies bear this notice and the full citation on the first page. Reproduction for commercial purposes is strictly prohibited without the prior written consent of the Internet Society, the first-named author (for reproduction of an entire paper only), and the author’s employer if the paper was prepared within the scope of employment.NDSS ’17, 26 February - 1 March 2017, San Diego, CA, USACopyright 2017 Internet Society, ISBN 1-891562-46-0http://dx.doi.org/10.14722/ndss.2017.23394

IO to insert n documentswith k keywords

IO for a searchreturning n documents supports deletes

index type Static Dynamic Static DynamicUnencrypted Index O(k) O(k) O(1) O(1) yesCash et. al [4] O(k) O(nk) O(1) O(nk) dynamic onlyStefanov et. al [20] O(nk) O(nk) O(nk) O(nk) yesThis work1 O(k) O(k) O(1) O(1) partial dynamic

TABLE I: Informal description of IO costs of SSE schemes assuming n, the number of documents mapped to a given keyword,is smaller that the block size used for storage. When inserting n documents containing k keywords into an existing index, allexisting SSE schemes write each posting (i.e. keyword, document ID pair) to a distinct random location. As a result, when addingn documents to the index entry for a given keyword, there are n writes to distinct random locations. This is repeated for eachof the k keywords. For search, returning the complete entry in the index for a given keyword requires each of the n randomlocations be read. When used for messaging or email, all insertions and searches are done against the dynamic index and anysavings in the static case do not apply.

for non-encrypted search. Existing SSE schemes simply requireconsiderably more resources than unencrypted search.

We now examine in detail the specific challenges for emailand then detail our proposed solution.

A. Why Email is Different

Cloud-based mail offers a unique setting. First, while storageat this scale is cheap, access to it is not: existing mail search isalready IO bound. As a result, we must minimize the IO costof maintaining the index. As we will see, this is the drivingdesign constraint behind our work.

Second, unlike traditional settings for SSE, there is noinitial corpus of documents to index. Instead all documents(i.e. sent and received emails) are added after database creationvia updates2—the typical user receives between 30 and 200messages a day.3 This exacerbates the IO problem, since allexisting schemes end up performing one random write to thedisk per keyword for each new email and one random read perdocument ID returned in a search. This is in marked contrastto a standard index wherein multiple entries are packed intoone block and read/written in one shot, resulting in efficientIO usage.

Third, the query rate is exceedingly low. For Yahoo webmail,we see on the order of 250 searches per second total acrossall users on mail content from on the order of a few hundredmillion monthly active users. Searches on public metadata suchas sender, date sent, “has attachment”, etc. are far more frequent,but searches on mail content, i.e., what would be stored in anencrypted index, are surprisingly rare. Many schemes (e.g. [20])assume hundreds of queries per second, where it is economicalto load the index into memory. At an average two queries perinbox per month, storing the index in memory is neither costeffective nor remotely practical given the sheer number of userseach needing their own separate index.

If the IO problems are resolved, however, the prospects forencrypted mail search are actually fairly good: search for email

1We assume the obliviously updatable index is of fixed size and thus theoverhead it imposes is constant. This is the case if the key word list is fixedor the server has limited space. Otherwise there is a cost that is logarithmic inthe size of the OUI but independent of total index size or search frequency.

2Since the server knows the contents of existing (unencrypted) email, addingexisting unencrypted email to the index is unnecessary and for some schemesill-advised.

3The mean is 30 with a standard deviation of 148.

is in many ways easier than what is typically asked of encryptedsearch. We analyze detailed usage numbers furnished by YahooWebmail. We conclude that searchable encryption for cloud-based email is surprisingly plausible in terms of functionalityif cost is disregarded: the size of the datasets are manageable,search is not frequently used, and most queries are exceedinglysimple. In particular, the combination of single keyword plusintersection on public metadata (date, sender, “has attachment”),seems to cover the vast majority of searches. While conjunctivequeries would of course be an improvement, they are not strictlynecessary. Moreover the availability of cheap storage coupledwith the fact that search is infrequent, means the cost of indexentries for deleted files is relatively small. This allows us toeschew many of the complexities of fully dynamic searchableencryption because we can afford to delete entries on a best-effort basis. The scalability issues become even easier whenone considers that end-to-end encrypted mail is likely personaland the majority of email is not. As a result, only a smallfraction of mail will need to be encrypted or indexed usingSSE.

Why existing schemes fail Encrypted databases such as BlindSeer [16], Cryptdb [17] and others aim to provide complexprivate queries as a privacy-preserving replacement to traditionallarge database applications. Just as those database clusters areunsuited for mail search, so too are their privacy-preservingcounterparts.

The DSSE scheme of Stefanov et al. [20] provides impres-sive privacy protections including efficient deletion of entries,but as each individual index entry (i.e. a mapping from a wordto a single document) is stored in a random location, the IOexpansion relative to standard search is very large. This is notscalable in the context of deploying hundreds of millions ofindexes on a few thousand servers.

This problem is not isolated to Stefanov et al.’s scheme.Indeed, Cash et al. [4] note “One critical source of inefficiencyin practice (often ignored in theory) is a complete lack oflocality and parallelism: To execute a search, most priorSSE schemes sequentially read each result from storage ata pseudorandom position, and the only known way to avoidthis while maintaining privacy involves padding the server indexto a prohibitively large size.” [4]. Although one can arrive atthis from varying perspectives (disk IO, latency, or memoryrequirements), it is a central problem that renders all previousschemes unusable in this context.

2

f4

f4

f9

f1 f8 f9

k1

k2

k3

Data centerFull Block Index

keyword queries

f3

f4 f1

f1 f8

k1

k7

k3

overflow:

f1, f5 ,f8,f12k1

k4

k1

f4, f1 ,f3,f9

f2, f3 ,f8,f11

f3 f3 f5 f7 f9 f1, f5 ,f8,f9

Local DeviceObliviously Updatable Index

…

Fig. 1: IO-DSSE logical architecture: an index mapping keywords to document IDs is stored locally, then the largest entriesoverflow into the obliviously updatable index. When the entry for a keyword in the OUI is full, index entries are inserted into theFull Block Index in chunks. Keyword searches query all three indexes. Note: encryption and the obliviously updatable indexconstruction hide this view from the server.

While Cash and Tessaro [6] show there is a fundamentaltrade-off between locality and index size, in many settingsa large increase in overall performance can be obtained byallowing a small blow-up in index size. Cash et al. [4] offersuch an improvement: instead of storing each index entry ata random location on disk, they store entries sequentially infixed-size blocks, drastically reducing the number of disk readsby allowing many index entries for a given keyword to beretrieved at once. Unfortunately, this technique cannot readilybe used to handle updates. The act of appending a new indexentry to an existing (partially-filled) block leaks significantinformation on updates (which are frequent). This means thesetechniques are not applicable to entries inserted into the indexafter the initial index creation.

As a result, while Cash et al. make use of an efficient on-diskindex for the starting document corpus, all updates to the indexare stored in memory because each index entry is written to adistinct random location. The scheme is not designed to dealwith frequent updates in an IO-efficient manner and assumesthat most of the corpus is indexed statically, at initialization.Thus, when the scheme is used for dynamic updates, it incursthe very same locality issues the authors identify. Bridging thisgap between locality and privacy for highly dynamic indexesis the goal of our construction. See Table I for details.

B. Our Contribution

A logical extension of Cash et al.’s scheme is to bufferpartial blocks locally (client-side) and only upload to the serveronce the block is full. However our experiments in Section IV-Cindicate that a storage-limited client (e.g. a mobile device) willoverflow its local storage in less than 100 days even for theaverage user. A second approach is to offload this buffer to theserver in a way that hides when it is updated but still allows thebuffer to be searched. Oblivious RAM (ORAM) [2, 9, 11, 12, 18]is a natural choice. Index entries would be buffered in a smallORAM cache and then written to the full index.

ORAM, however, is not an ideal choice. First, in mostSSE schemes, access patterns for search are already leaked,yet in ORAM we must pay for the full ORAM overhead notjust for reads and writes associated with index updates, butfor searches as well.4 Second, ORAM must remain securefor arbitrary access patterns, while we only need to obscurebatched updates to the index (many emails/keywords can bebatched and inserted in the index at the same time).

Thus, in Section III, we construct an obliviously updatableindex (OUI), which provides for ORAM-like properties forupdates to the index (both reads and writes) but allows simplenon-oblivious reads for search. An overview of this approachis given in Figure 1. This is accomplished by using a weakersecurity requirement tailored to the SSE leakage function.Through optimizations and batching, we achieve a 94% IOsavings vs. the generic approach using Path ORAM [21].

To summarize our contributions:

1) We examine the requirements and feasibility of search onencrypted email and more generally, the class of IO-boundDSSE schemes that exhibit a low query rate and highupdate rate. This examination is supported with realisticdata from a large webmail provider.

2) Motivated by IO-efficient SSE for highly dynamic corpi,we introduce the concept of an Obliviously UpdatableIndex (OUI). As a proof of concept, we provide a newconstruction for an OUI, with proof of security, whichoffers a 94% savings compared to a naive implementationusing ORAM. We obtain our final solution by combiningthis with the state-of-the-art IO-efficient SSE that indexesthe full blocks.

3) We evaluate our solution using real-world data from100,000 mail users, showing we achieve a 99% reductionin the IO cost vs. the state-of-the-art in SSE schemes such

4Because we need to know the content of an index entry to update it andmust read that from a server, we cannot use “write-only” [1] ORAM whichassumes reads are not observable.

3

as that of Cash et al. which, for purely dynamic insertions,write each document-keyword pair to a random location.We also report the storage required for a typical mail userboth on the client side and the server side.

C. No free lunch

These improvements are, of course, not free. First, we mustallow for slightly more leakage in search than the scheme of [4],since we leak when an entry in the OUI is full and needs to bepushed to the full-block index. Second, we can only deal withdeletes from the OUI: once an entry is written to the staticindex, it is stored until the index is rebuilt. Thus we do notprovide a fully dynamic index. Given the low cost of storagerelative to the size of emails and the fact that for the averageuser emails are contained fully within the partial index for atleast 11 days, we believe this is an acceptable trade-off.

II. BACKGROUND

A. Hash Tables

A hash table is a data structure commonly used for mappingkeys to values. It often uses a hash function h that maps a keyto an index (or a set of indexes) in a memory array M wherethe value associated with the key may be found. The keywordis not in the table if it is not in one of those locations. Moreformally, we define a hash table H = (hsetup, hlookup, hwrite)using a tuple of algorithms.

• (h,M)← hsetup(S): hsetup takes as input an initial setS of keyword-value pairs and outputs a hash function hand a memory array M storing the key-value pairs.

• M′ ← hwrite(key, value,M, h): If (key, value) al-ready exists in the table it does nothing, else it stores(key, value) in the table. If M and h are known from thecontext, we use the shorter notation hwrite(key, value).

• value ← hlookup(key,M, h): hlookup returns value if(key, value) is in the table. Else it returns ⊥. If M and hare known from the context, we use the shorter notationhlookup(key).

B. Oblivious RAM

We recall Oblivious RAM (ORAM), a notion introducedand first studied in the seminal paper of Goldreich andOstrovsky [12]. ORAM can be thought of as a compiler thatencodes the memory into a special format such that accesseson the compiled memory do not reveal the underlying accesspatterns on the original memory. An ORAM scheme consistsof protocols (SETUP,OBLIVIOUSACCESS).

• 〈σ,EM〉 ↔ SETUP〈(1λ,M),⊥〉: SETUP takes as input thesecurity parameter λ and a memory array M and outputsa secret state σ (for the client), and an encrypted memoryEM (for the server).

• 〈(M[y], σ′),EM′〉 ↔ OBLIVIOUSACCESS〈(σ, y, v),EM〉:OBLIVIOUSACCESS is a protocol between the client andthe server, where the client’s input is the secret state σ,an index y and a value v which is set to null in case theaccess is a read operation (not a write). Server’s inputis the encrypted memory EM. Client’s output is M[y]and an updated secret state σ′, and the server’s output is

an updated encrypted memory EM′ where M[y] = v, ifv 6= null.

Correctness Consider the following correctness experiment.Adversary A chooses memory M. Consider the encrypteddatabase EM generated with SETUP (i.e., 〈σ,EM〉 ↔ SETUP〈(1λ,M),⊥〉). The adversary then adaptively chooses mem-ory locations to read and write. Denote the adversary’sread/write queries by (y1, v1), . . . , (yq, vq) where vi = nullfor read operations. A wins in the correctness game if〈(Mi[yi], σi),EM

′〉 are not the final outputs of the protocolOBLIVIOUSACCESS〈(σi−1, yi, vi),EMi−1〉 for any 1 ≤ i ≤ q,where Mi,EMi, σi are the memory array, the encrypted memoryarray and the secret state, respectively, after the i-th accessoperation, and OBLIVIOUSACCESS is run between an honestclient and server. The ORAM scheme is correct if the probabilityof A in winning the game is negligible in λ.

Security An ORAM scheme is secure if for any adversaryA, there exists a simulator S such that the following twodistributions are computationally indistinguishable.

• RealA(λ): A chooses M. The experiment then runs〈σ,EM〉 ↔ SETUP〈(1λ,M),⊥〉. A then adaptively makesread/write queries (yi, v) where v = null on reads, forwhich the experiment runs the protocol 〈(M[yi], σi),EMi〉↔ OBLIVIOUSACCESS〈(σi−1, yi, v),EMi−1〉 . Denotethe full transcript of the protocol by ti. Eventually, theexperiment outputs (EM, t1, . . . , tq) where q is the totalnumber of read/write queries.

• IdealA,S(λ): The experiment outputs (EM, t′1, . . . , t′q)↔

S(q, |M|, 1λ).

C. Path ORAM

Path ORAM [21] is a tree-based ORAM construction withhigh practical efficiency. We use Path ORAM as a componentin our SSE construction. We only review the non-recursiveversion of Path ORAM where the client stores the positionmap locally and hence only a single binary tree T is neededto store the data on the server.

Notations Let M be a memory array of size at most N = 2L

that we want to obliviously store on the server. We use M[i]to denote the ith block in M. Let T denote a binary tree ofdepth L on the server side that will be used to store M. Theclient stores a position map position where x = position[i] isthe index of a uniformly random leaf in T . The invariant PathORAM maintains is that M[i] is stored in a node on the pathfrom the root to leaf x which we denote by P (x). We alsouse P (x, `) to denote the node at level ` on path P (x), i.e.the node that has distance ` from the root. There is a bucketassociated with each node of the tree T , and each bucket canat most fit Z memory blocks.

The client holds a small local stash denoted by S, whichcontains a set of blocks that need to be pushed into the server’stree.

ORAM Setup We assume that memory array M is initiallyempty. Client’s stash S is empty. All the buckets in the treeT are filled with encryptions of dummy data. The position

4

OBLIVIOUSACCESS〈(σ, y, v),EM〉:

1: x← position[y]

2: position[y]R← {0, . . . , 2L}

3: for ` ∈ {0, . . . , L} do4: S ← S ∪ READBUCKET(P (x, `))5: end for6: data← Read block y from S7: if v 6= then8: S ← (S − {(y, data)}) ∪ {(y, v)}9: end if

10: for ` ∈ {L, . . . , 0} do11: S′ ← {(y′, data′) ∈ S : P (x, `) =

P (position[y′], `)}12: S′ ← Select min(|S′|, Z) blocks from S′.13: S ← S − S′14: WRITEBUCKET(P (x, `), S′)15: end for

Fig. 2: Read/Write Ops in path ORAM

map position is initialized with uniformly random values in{0, . . . , 2L}. This encrypted tree is denoted by EM.

Read/Write Operations To read M[y] or to write a value vat M[y], the client first looks up the leaf position x from theposition map and reads all the buckets along the path P (x). Itthen updates position[y] to a fresh random value in {0, . . . , 2L}.If it is a read operation, the encryption of (y, v) will be foundin one of the buckets on P (x), which the client decypts tooutput v. It also adds all the buckets on P (x) to its local stash.If it is a write operation, the client also adds (y, v) to its localstash.

The client encrypts all the blocks in the stash and insertsas many as possible into the buckets along P (x), insertingeach block into the lowest bucket in the path possible whilemaintaining the invariant that each block y′ remains on thepath P (position[y′]).

Figure 2 describes the read/write operations in more detail.The READBUCKET protocol has the server return the bucketbeing read to the client who decrypts and outputs the blocks inthe bucket. The WRITEBUCKET protocol has the client encryptand insert all the blocks in its input set into a bucket and sendit to the server.

D. Searchable Symmetric Encryption

A database D is a collection of documents di each ofwhich consist of a set of keywords Wi. A document can be awebpage, an email, or a record in a database, and the keywordscan represent the words in the document, or the attributesassociated with it. A symmetric searchable encryption (SSE)scheme allows a client to outsource a database to an untrustedserver in an encrypted format and have the server performkeyword searches that return a set of documents containing thekeyword. For practical reasons, SSE schemes often return aset of identifiers that point to the actual documents. The clientcan then present these identifiers to retrieve the documents anddecrypt them locally.

More precisely, a database is a set of document/keyword-setpair DB = (di,Wi)

Ni=1. Let W = ∪Ni=1Wi be the universe of

keywords. A keyword search query for w should return all diwhere w ∈Wi. We denote this subset of DB by DB(w).

A searchable symmetric encryption scheme Π consists ofprotocols SSESETUP, SSESEARCH and SSEADD.

• 〈EDB, σ〉 ← SSESETUP〈(1λ,DB),⊥〉: SSESETUP takesas client’s input a database DB and outputs a secret stateσ (for the client) and an encrypted database EDB whichis outsourced to the server.

• 〈(DB(w), σ′),EDB′〉 ← SSESEARCH〈(σ,w),EDB〉:SSESEARCH is a protocol between the client and theserver, where the client’s input is the secret state σ andthe keyword w he is searching for. The server’s input isthe encrypted database EDB. The client’s output is theset of documents containing w, i.e., DB(w) as well anupdated secret state σ′, and the server obtains an updatedencrypted database EDB′.

• 〈σ′,EDB′〉 ← SSEADD〈(σ, d),EDB〉: SSEADD is aprotocol between the client and the server, where theclient’s input is the secret state σ and a document d tobe inserted into the database. The server’s input is theencrypted database EDB. The client’s output is an updatedsecret state σ′, and the server’s output is an updatedencrypted database EDB′ which now contains the newdocument d.

Correctness Consider the following correctness experiment.An adversary A chooses a database DB. Consider the en-crypted database EDB generated using SSESETUP (i.e. 〈EDB,K〉 ← SSESETUP〈(1λ,DB),⊥〉). The adversary then adap-tively chooses keywords to search and documents to add tothe database. Denote the searched keywords by w1, . . . , wt.A wins in the correctness game if 〈(DBi(wi), σi),EDBi〉 6=SSESEARCH〈(σi−1, wi),EDBi−1〉 for any 1 ≤ i ≤ t, whereDBi,EDBi are the database and encrypted database, respec-tively, after the ith search, and SSESEARCH and SSEADD arerun between an honest client and server. The SSE scheme iscorrect if the probability of A in winning the game is negligiblein λ.

Security Security of SSE schemes is parametrized by a leakagefunction L, which explains what the adversary (the server) learnsabout the database and the search queries, while interactingwith a secure SSE scheme.

An SSE Scheme Π is L-secure if for any PPT adversaryA, there exists a simulator S such that the following twodistributions are computationally indistinguishable.

• RealΠA(λ): A chooses DB. The experiment then runs〈EDB, σ〉 ← SSESETUP〈(1λ,DB),⊥〉. A then adap-tively makes search queries wi, which the experimentanswers by running the protocol 〈DBi−1(wi), σi〉 ←SSESEARCH〈(σi−1, wi),EDBi−1〉. Denote the full tran-scripts of the protocol by ti. Add queries are handledin a similar way. Eventually, the experiment outputs(EDB, t1, . . . , tq) where q is the total number of search/addqueries made by A.

• IdealΠA,S,L(λ): A choose DB. The experiment runs(EDB′, st0)← S(L(DB)). On any search query wi from

5

A, the experiment adds (wi, search) to the history H ,and on an add query di it adds (di, add) to H . It thenruns (t′i, sti) ← S(sti−1, L(DBi−1, H)). Eventually, theexperiment outputs (EDB′, t1, . . . , t

′q) where q is the total

number of search/add queries made bt A.

III. OUR CONSTRUCTION

As discussed earlier, three important security/efficiencyrequirements for any searchable symmetric scheme for emailto be practical are (1) dynamic updatability, (2) low latencyon search and hence high IO efficiency, and (3) no leakage onupdates (send/received email).

The SSE construction of [5] accommodates dynamic updatesand does not leak keyword patterns on updates as each newkeyword-document pair is added to the index using a freshlyrandom key generated by a PRF that takes a counter as input.This meets requirements (1) and (3). They also address theIO efficiency issue by storing document IDs in larger blocksand hence retrieving many documents IDs using a single diskaccess. This provides a partial solution to the IO efficiency andfast search, but the solution is not suitable for a highly-dynamicapplication such as mail.

In particular, the main challenge is to pack multiple updates(i.e. a set of new keyword-document pairs) into a large block,before pushing them into the encrypted index. The naivesolution of storing the partial blocks in the same index andadding new keyword-document pairs on each update leaks theupdate pattern which is a major drawback for an update-heavyapplication such as email.

Our construction consists of two pieces: the obliviouslyupdatable index (OUI), a small dynamic encrypted index forpartial blocks, and the full-blocks index (FBI), a large append-only encrypted index for full blocks.

The full-block index holds a mapping from an encryptedkeyword to a fixed-size block containing document IDs. Newentries can be added to this index only when we have afull block. This is the basic approach taken by [5] for staticencrypted search, though they expand on it to deal with farlarger data sets than we wish to.

The main technical challenge in our construction is thedesign of OUI for managing partially-filled blocks until theyare full and can be pushed to the FBI. In particular, note thatthe blocks in OUI need not be full and are instead paddedto some fixed size. When a block is full of real data (i.e. nopadding) its contents are transferred to the full-block index.This allows messages to be added and deleted from the OUIby updating the requisite block. Of course, we must do so ina way that does not leak which blocks are being updated, orelse we fail to meet the basic requirements for secure searchby leaking update patterns.

A. An Obliviously Updatable Index

ORAM forms a generic starting point for our obliviouslyupdatable index. In particular, storing partial blocks in ORAMwould allow us to update them privately. However, as a genericapproach, ORAM is an overkill. An index built on top ofORAM would hide not only reads and writes resulting froman index update, but also reads resulting from an index lookup.

This is stronger than the typical protection provided by SSEschemes (which leak the “search pattern”), and in our case(similar to prior work) this information is already revealed viasearches against the full-block index. So we gain no additionalprivacy by hiding the search pattern only in the OUI and willrealize considerable efficiency gains by not doing so.

As a concrete starting point, consider a basic constructionof Path ORAM [21]. In Path ORAM, entries (called blocksin our construction) are stored as leaves in a full binary tree.Each time an entry is read or written, the entire path from itsleaf to root is read, the entry is remapped to a random leafin the tree, and the original path is rewritten with the entryplaced at as close to the lowest common ancestor of the oldand new paths as possible.

The position map which keeps track of the current leafposition for each entry is typically stored on the server side inits own ORAM. This leads to many round trips of interactionfor each read/update which is a non-starter for a real-timeapplication such as email. We note that for email, it is feasible tostore the position map client side. As shown in the experimentsin Section IV-C, this storage will not exceed 70MB in 10 yearsfor the 95th percentile user and for most users is closer to35MB.

Even with the position map stored on the client side, aread or write entails reading everything on the path from a leafto the root, performing some client side operations, and thenwriting back along that path. In other words, in Path ORAM(and ORAM in general), entries are shuffled both in case ofreads and writes.

At first glance, in the case of a lookup in the obliviousindex, we can simply omit the complicated machinery for aread of the ORAM (which we only need for reads and writesfor index updates) and directly access a single entry in the tree.We do not care if repeated index accesses to the same locationare identifiable. However, there are two issues with this. First,the position map only stores what leaf an entry was mapped to,not the particular point along the path from leaf to root whereit is stored. This can be fixed by storing, for each keyword,additional information to locate the entry.

The larger issue is that the reshuffling that occurs on aread provides privacy protections not just for the read (whichis not important for us) but for subsequent reads and writes.If reads are not shuffled, then an observer can identify whenfrequently looked up index entries are updated. As a result,we cannot simply have a “half-ORAM”: to get completelyoblivious writes, we must at some point reshuffle the locationswe read from.

Crucially, in our obliviously updatable index, we need notdo this on every read (as in ORAM), rather we can defer theshuffling induced by a non-oblivious read to the beginning ofan update. We call these deferred reads.

This enables considerable savings. First, since updates canbe batched (i.e., we collect a bunch of updates to various entrieslocally and only commit them to the server later), we can shiftthe computational and bandwidth load to idle time (e.g. whena mobile device is plugged in and connected to wifi) at thecost of temporary client storage. Second, repeated searches forthe same term only result in one deferred read. Third, searches

6

for terms that are mapped to the same leaf also only result inone shuffle operation. Finally, because paths overlap even fordistinct leaves, we will realize considerable savings: e.g. for10 deferred read shuffles, we will end up transmitting the rootof the tree once instead of 10 times, the children of root twiceinstead of 5 times, etc. Looking forward to our evaluation, thisresults in over 90% savings in accesses compared to the simplePath ORAM.

We note that write-only ORAM constructions [1] do notsolve our problem. Write-only ORAM is used in settings wherereads leave no record (e.g. where an adversary only has accessto snapshots of an encrypted disk, which reveals changes due towrites but not reads.). In these settings, the initial read neededto determine the contents of the block being appended to canbe done in the clear. We cannot do that here since we mustrequest the partial bock from the server before appending to it.

To summarize, our obliviously updatable index is a modifiedPath ORAM scheme with the following changes:

• We keep the position map on the client slide and augmentthe position map to allow us to locate index entries insidea given path from leaf to the root.

• On non-oblivious reads: we lookup the entry directly fromits position in the tree (i.e. one disk access), but add theleaf to the list of deferred reads.

• On batched reads and updates: we read all paths in theset of deferred reads since the last batch updated and allthe paths associated with the updates themselves. We thenremap and edit the entries on these paths as in standardPath ORAM and write them back to the ORAM at once.

Security Intuition Deferred shuffling for reads ensures thatwhen a non-deferred read/write happens, the system is in theexact same state as it would be in full Path ORAM. Intuitively,this models the effect of shuffling a deck of cards: no matterwhat the previous state was and how the deck was rigged, theshuffle is still good.

More formally, our approach means that after the deferredread, the position map entries are statistically independent ofeach other, and we retain the security condition for Path ORAMthat

∏Mj=1 Pr(position(aj)) = ( 1

2L

M) for non-deferred oper-

ations. Of course, we have leaked substantial information aboutthe prior state of the index, but that leakage is allowed in SSE!Rather than proving this separately, we will capture it in theproof of security for the SSE scheme itself.

B. The Full Protocol

Next, we describe our full DSSE scheme for email whichis a combination of the OUI described above and a separateindex for full blocks. A detailed description follows.

Let H = (hsetup, hlookup, hwrite) be a hash table imple-mentation, E = (KG,Enc,Dec) be a CPA-secure encryptionscheme and F : K ×M → C be a pseudorandom function.Let W be the universe of all keywords, and L = log(|W |).

Setup. (Figure 3) For simplicity, we assume that the DBis initially empty, and documents are dynamically added. Ifnot, one can run the SSEADD protocol we describe shortlymultiple times to populate the client and server storages withthe documents in DB.

〈σ,EDB〉 ↔ SSESETUP〈(1λ,⊥),⊥〉:

1: Client runs (hc,Mc) ← hsetup() to setup a localhash table.

2: Server runs (hs,Ms)← hsetup() to setup an append-only hash table.

3: for w ∈ |W | do4: posw

R← {0, . . . , 2L}5: countw, rw, `w ← 0, Bw ← ∅6: Client runs Mc ← hwrite(w, [posw, `w, countw,rw, Bw],Mc)

7: end for8: kf ← K(1λ), ke ← KG(1λ), ka ← KG(1λ)9: Client and server run the setup for a non-recursive

Path ORAM. Server stores the tree T , and clientstores the stash S.

10: Client outputs σ = (Mc, S, kf , ka, ke)11: Server outputs EDB = (Ms, T )

Fig. 3: Setup for our DSSE scheme

The client generates three random keys kf , ke, and ka, onefor the PRF F , and the other two for the CPA-secure encryptionscheme.

The client and server initialize the obliviously updatableindex, i.e., a non-recursive Path ORAM for a memory of size|W |. We denote the tree stored at the server by T , and thecorresponding stash stored at the client by S. For all referencesto Path ORAM we use the notation introduced in Section II-C.The server also sets up an initially empty full-block index, i.e.,an append-only hash table that will be used to store full blocksof document IDs.

For every w ∈W , the client stores in a local hash table thekey-value pair (w, [posw, `w, countw, rw, Bw]), where Bw is ablock storing IDs of documents containing w (initially empty),posw stores the leaf position in {0, . . . , 2L} corresponding tow (chosen uniformly at random), `w stores the level of the nodeon path P (posw) that would store the block for documentscontaining w (initially empty), countw stores the number of fullblocks for keyword w already stored in the append-only hashtable (initially 0), and rw is a bit indicating whether keywordw is searched since the last push of the client’s block to PathORAM (initially 0).

The client’s state σ will be the hash table Mc, the stash Sfor the Path ORAM, and the keys ke, kf , ka.

Search. (Figure 4) The client will store the matching documentsin the initially empty set R = ∅. To search locally, theclient first looks up w in its local hash table to obtain[posw, `w, countw, rw, Bw], and lets R = R ∪Bw.

It then asks the server for the bucket in the tree T at nodelevel `w and on path P (posw), i.e., P (posw, `w). It decryptsthe blocks in the bucket using ke. If it finds a tuple (w,Ow) inthe bucket, it lets R = R ∪Ow. If rw is not yet set, the clientlets rw = 1 to indicate that w was searched for.

For i = 1, . . . , countw, the client sends Fkf (w||i) to theserver, who looks it up in the append-only hash table and

7

SSESEARCH〈(σ,w),EDB = (T,Ms)〉:

1: R← ∅2: [posw, `w, countw, rw, Bw]← hlookup(w,Mc)3: R← R ∪Bw4: U ← READBUCKET(P (posw, `w)5: Read (w,Ow) from U6: R← R ∪Ow7: rw ← 18: hwrite(w, [posw, `w, countw, rw, Bw],Mc)9: for i ∈ {1, . . . , countw} do

10: Client sends Fkf (w||i) to server11: Server returns Ciw ← hlookup(Fkf (w||i),Ms)12: Aiw ← Decka(Ciw)13: R← R ∪Aiw14: end for15: Client outputs R

Fig. 4: Search for our DSSE scheme

returns the encrypted full block Aiw. The client decrypts usingka and lets R = R ∪ Aiw. The client then outputs R. SeeFigure 4 for details.

Update. (Figure 5) Let idd be the document identifier associ-ated with d. For every keyword w in d, the client looks up win its local hash and adds idd to Bw. It then checks whether itslocal storage has reached the maximum limit maxc or not. Ifnot, the update is done. Else, we need to push all the documentblocks to the server.

But before doing so, we need to finish the ORAM accessfor all reads done since the last push. In particular, for all non-zero rw’s, the client needs to read the whole path P (posw),re-encrypt all the buckets using fresh randomness, update poswto a fresh random leaf, and write the buckets back to the treeusing the Path ORAM approach.

Then, for every non-empty block Bw in its local hash, theclient performs a full ORAM write to add the documents inBw to the ORAM block Ow for the same keyword. If Owbecomes full as a result, maxb documents IDs in the blockare removed and inserted into Acountw+1

w , and inserted to theappend-only hash table using a keyword Fkf (w||countw + 1).See Figure 5 for details.

C. Security Analysis

As noted in Section II, security of an SSE scheme isdefined with respect to a leakage function L on the database ofdocuments DB as well as the history of search/update operationsin the index. We first specify the leakage function for ourconstruction.

The Leakage Function Recall that DB = (di,Wi)Ni=1 is the

database of document-keyword pairs and W = ∪Ni=1Wi is theuniverse of keywords.

During the setup, L(DB) outputs the size of database |DB|,i.e., the total number of initial document-keyword pairs in thedatabase. For simplicity we can assume this is zero initially. Oneach search query wi, the leakage function L(DBi−1, H), leaks

SSEADD〈(σ, idd),EDB〉:

1: for w ∈ d do2: [posw, `w, countw, rw, Bw]← hlookup(w,Mc)3: Bw ← Bw ∪ {idd}4: hwrite(w, [posw, `w, countw, rw, Bw],Mc)5: sizec ← sizec + 16: end for7: if sizec < maxc then8: return9: else

10: U ← {w ∈ |W | : rw == 1}11: for w ∈ U do12: [posw, `w, countw, rw, Bw] ← hlookup(w,

Mc)13: for ` ∈ {0, . . . , L} do14: S ← S ∪ READBUCKET(P (posw, `))15: end for16: end for17: for (w,Ow) ∈ S do18: O′w ← Ow ∪Bw19: if |O′w| > maxb then20: countw ← countw + 121: O′′w ← first maxb items in O′w22: hwrite(Fkf (w||countw), O′′w,Ms)23: O′w ← O′′w −O′w24: end if25: S ← (S − {(w,Ow)}) ∪ {(w,O′w)}26: end for27: for ` ∈ {L, . . . , 0} do28: S′ ← {(w′, Ow′) ∈ S : P (x, `) =

P (posw′ , `)}29: S′ ← Select min(|S′|, Z) blocks from S′.30: S ← S − S′31: WRITEBUCKET(P (x, `), S′)32: for (w,Ow) ∈ S′ do33: `w ← `34: rw ← 035: Bw ← ∅36: posw

R← {0, . . . , 2L}37: hwrite(w, [posw, `w, countw, rw, Bw],

Mc)38: end for39: end for40: sizec ← 041: end if

Fig. 5: Update for our DSSE scheme

the pattern of searches for wi in the history H . Note that thisdoes not leak wi itself, but only the location of all search/updatequeries for wi in the sequence all previous read/updates. Onan update query di, the leakage function L(DBi−1, H) leaksthe total number of keywords in di, and also the total numberof keywords in di for which the number of documents in theindex has reached the multiple of our designated block-size.Again, this does not leak what the actual keywords are andhow they are related to previous searches/updates (no leakageof patterns). This leakage on updates simply captures the factthat the server learns when full-blocks are pushed from the

8

partial-block index to the full-block index.

Theorem 1: Our SSE Scheme is L-secure (see definitionin Section II-D), if F is a pseudorandom function, and E is aCPA-secure encryption scheme.

Proof Sketch: First, we need to describe a simulator S thatgiven access to the leakage function describe above, simulatesthe adversary A (i.e. untrusted server’s) view in the real world.

Description of the Simulator: The simulator initializes alocal position map that it uses for bookkeeping, just as thehonest client would.

On each update query di (or many updates if they arebatched), the simulator learns the number of keywords in di,n. It also learns the total number of keywords in di that justreached a full block (m) and need to be pushed to the full-blockindex. It also knows (from the state it is keeping) the locationof the leaf for all deferred reads since the last update. Thesimulator behaves similar to the honest client, except that hegenerates the leaf location for each keyword being updateduniformly at random and the m locations in the full-blockindex where the newly filled blocks will go also uniformly atrandom. It keeps record of all these locations in its positionmap. Also, for all encrypted blocks it needs to send, it simplygenerates fresh encryptions of dummy values using a randomencryption key it generates.

On each search query for wi, S learns (from the leakagefunction) all previous occurrences of search/update for wi. Ifthis is the first occurrence, it chooses a random entry on theORAM tree for the OUI, a sequence of random locations thathave not yet been looked up in the full-block index, and alsostored these locations for bookkeeping. But if it is not the firstoccurrence, S has previously stored the locations in the partial-block and full-block index it had sent to the server. It simplysends the same locations to the server again. This completesthe description of the simulator.

We need to show that the simulated view above is indistin-guishable from the view of the adversary in the real execution.We do so using a sequence of hybrids.

H0: In the first hybrid, the adversary is interacting with thesimulator described above. In other words, all ”random”locations are generated uniformly at random and indepen-dently, and all encrypted values are dummy values.

H1: In the second hybrid, all locations that were generateduniformly at random are instead generated by the clientusing a PRF with a random key kf not known to theadversary.Note that the advantage of an adversary in distinguishingthese two hybrids is bounded by the advantage of anadversary in breaking the PRF security.

H2: In the third hybrid, all dummy encryptions are replaced bythe encryption of actual document identifiers (chosen bythe adversary) using a semantically secure symmetric-keyencryption as prescribed in the real protocol.The advantage of the adversary in distinguishing the twogames is bounded by the security of the CPA-secureencryption scheme used to encrypt document identifiers.

It only remains to show that the view of the adversary inH2 is identical to the real protocols described earlier in this

Section. We consider adversary’s view for the search queriesand update queries separately. On update queries, both in thereal protocol and in H2, the location to be updated in the OUIis generated using a PRF as described in the protocol, and so isthe location to be added to the full-block index for all keywordsthat have reached a full block during this update. Similarly, inboth cases, the real document keywords used for generatinglocations and the real document identifiers are encrypted. Sothe two views are identical.

In case of read queries, if the keyword is being searchedfor the first time, again the leaf location to be looked up inboth the real protocol and H2 are generated using a PRF, andfor repeated reads, in both cases, the exact same locations inthe OUI and the full-block index are looked up. Hence the twoviews are identical. This concludes the proof sketch.

IV. EVALUATION

In this section we assess the feasibility and performance ofour SSE scheme for email, using experiments that are based onreal data. The two important requirements we focus on in ourevaluation are (1) storage usage both on the server side andthe client side, (2) IO performance on the server side. Again,as existing (non-encrypted) mail search is already IO bound,our primary concern is the second criteria.

A. Real World Data

Note that the performance of any SSE scheme criticallydepends on two main pieces of information: (1) the distributionand frequency of updates to the index, i.e., new keyword-document pairs added to the search index, and (2) distributionand frequency of the keywords being searched.

Data on Index Updates for Email Our email data comesfrom Yahoo Webmail. Yahoo mail has hundreds of millions ofmonthly active users and maintains a 30 day rolling window ofcustomer email for research. This dataset consists of the sentand received emails of over one hundred thousand users whoopted into email collection for research. For privacy reasons,we extract only the frequency of keywords, not the actual wordsthemselves and all analysis of the data itself was performeddirectly on the map-reduce cluster containing the dataset: onlythe counts of each token, not the tokens themselves, wereexported. On average, users receive 30 emails per day with astandard deviation of 148. Furthermore, each message containsan average of 143 (unstemmed) keywords in it with a standarddeviation of 140.

For all keyword data, we strip HTML from the messagesand use the standard tokenizer and stopword list from ApacheLucene.5 No stemming or other filtering was applied. Betweenthis and the fact that the dataset spanned more than just English,our data represents an upper bound for the “index everything”approach. We identified a total of 62,131,942 keywords in ourdataset across all 100K users’ email. As the estimated numberof words in the English language is in the order of one million,there is ample room for stemming and filtering to drasticallyreduce this. Nevertheless, we stick with this number for ourexperiments as an empirical worst case measurement.

5We used the list StopAnalyzer.ENGLISH_STOP_WORDS_SET.

9

Approximating query data For two reasons, we do not haveaccess to comprehensive query data. First, there is no suchdataset of users’ queries for which users explicitly opted in.Second, existing mail search supports more than single keywordqueries and as such queries are often in natural language. It isunclear how to appropriately generate keyword searches fromsuch data. As such, for the sake of experiments, we assumesearch terms are selected uniformly at random from the set ofall indexed words.

Given the low number of queries, we anticipate that theoverall effect of this is minimal. The primary concern in ourevaluation is the effect of the large number of updates onstorage and performance requirements.

B. IO Performance of IO-DSSE

We now examine the IO performance of IO-DSSE relativeto the operating conditions we observe at Yahoo. Our goal isto measure the ability of our scheme to scale out to millions ofconcurrent instances on arbitrary (and likely proprietary) cloudinfrastructure. As a result, we model such an system abstractlyas a key value store and measure the number of reads andwrites against it. We implement our scheme in Python againstthis abstraction.

Our measurements are taken over 30 days of trafficgenerated using the sampled distribution of keywords discussedabove and is repeated for 50 iterations. We assume that thesystem pushes all client-side email messages to the serverevery day. This models a mobile device that has access to afree Internet connection when at home. Given that we fix thedistribution of keywords, we are left with three variables: thenumber of searches, the number of emails per day, and thenumber of keywords per email.

For simplicity, we fix the number of keywords at 350 (this istwo standard deviations above the average), and vary the numberof email messages starting with the average and incrementingby the standard deviation. This has the net effect of changingthe total number of updates per day which (along with thedistribution of keywords) is the actual controlling variable forperformance. Similarly, we fix the number of searches at oneper day, modeling an active user (at Yahoo the average usersearches message content once per month.).

Finally, we somewhat arbitrarily fix the Path ORAMparameters, assuming a height of 17 at 4 buckets per levelwith each ORAM entry containing a block of 500 identifiers(at 64 bits per file ID, this gives us blocks that approximatelyfit in a 4KB disk block). Real deployments should tune theseparameters dependent on system architecture and testing. Westress again that our goal is not to see how our system handleslarge indexes—individual mail inboxes are at most tens ofgigabytes—but to measure the resources used by a small index.This allows us to see how costly it is to deploy in a setting withhundreds of millions of inboxes and thus indexes supported bya minimal number of servers.

Our experiment measures the actual IO savings of ourscheme against:

• An encrypted index which, for updates, stores eachkeyword-document pair at a random location. This repre-

sents the state of the art schemes under purely dynamicinsertion [5, 20], which perform identically.

• Our solution but with a naive implementation of anObliviously Updatable Index built with Path ORAM.

Because we are mainly concerned with the IO cost, it wasnot necessary to implement the comparison systems. For thecurrent state of the art encrypted index, the number of randomwrites will simply be equal to the number of inserted keywords,and the number of random reads equal to the number of searchresults. This provides a lower bound on the number of accessesrequired for all of the DSSE schemes we compare with.

For a solution that uses the Path ORAM scheme as the OUI,the cost of a search is dominated by one ORAM read and oneORAM write. An ORAM “read” would be recorded as manyreads by our code as each level in the path generates a distinctaccess. Thus, instead of one non-deferred read as in our scheme,we charge oramHeight reads and oramHeight writes. Byread or write, we mean the access of a single key-value pairin the underlying server-side data structure.

A third approach we could compare against is a variantof Cash et al. scheme [4] where full-blocks are stored on theserver-side, and partial blocks are stored locally on the clientside. This would exhibit better performance than our schemein the short term, because it reduces IO costs compared to anOUI. However, when the client-side storage becomes full, theblocks need to be pushed to the index even if they are onlypartially full and indeed maybe almost empty. As we will seein the storage discussion, this does not work in practice asthe client would need to regularly push partially-filled blocksto the static index, hence defeating the IO efficiency gains ofpacking many documents identifiers into one block.

Using our implementation, we measure:

Total IO savings: The total amount of IO saved compared tothe existing approach of SSE schemes (including [5, 20]under purely dynamic insertion). This includes both searchand update.

IO savings for search: The amount of savings on read dueto search, ignoring deferred reads. This represents theimmediate cost of a search and also the associated latencysavings.

IO savings vs. ORAM: Total IO savings when using an OUIvs. ORAM. This is the savings due to our optimizedobliviously updatable index that does not require fullORAM security.

The results show (see Figure 6) a 99% percent reductionin the IO cost of our scheme compared to a scheme that doesone random read per search result and one random write perkeyword-document pair. This is slightly different from the naiveestimate for the savings of simply batching IO into contiguousblocks of 500 (our chosen block size), i.e., x−

x500

x = 0.998 dueto both the overhead of access to our obliviously updatableindex and the fact that entries will not be packed perfectly,instead being added in smaller groups as they arrive. However,some entries will still be in the stash locally and therefore willnot be read from the index. These two issues appear to roughlycancel out.

We show a 94% percent reduction in the reads requiredby the server for a search query for our scheme vs. simply

10

Fig. 6: Experimental and simulated results. IO savings for IO-DSSE vs existing approach and vs an obliviously updatable indexconstructed with PATH ORAM.

construction an obliviously updatable index with ORAM. Thisshows that optimizations stemming from the relaxation ofORAM’s security properties are effective.

This is both an important cost saving and a large reductionin the latency of serving the first page results. These resultsare exactly as predicted, since we do one read instead of 17(the length of a full path for the chosen parameters) and save17−1

17 = .941. Finally, our experiments show a 20% to 82%reduction in the IO needed to return a search result. Since searchterms are selected uniformly at random from the set of results,many searches have only 1 or 2 documents associated with them,needing only the small corresponding number of reads from thenaive index vs. one read from the obliviously updatable indexin our scheme. For those, our scheme offers little advantage. Aswe increase the number of received messages, the total numberof indexed documents and therefore the expected number ofresults per search increases and our scheme becomes moreefficient. The variance in the number of results per search termalso accounts for the variance in the measured results. We againnote that, given the relative infrequence of searches against theindex compared to updates, this is not the crucial metric tooptimize for.

C. Simulated Long-term Storage Usage

We now examine the storage requirements for IO-DSSEboth on the server side (i.e. how large the obliviously updatableindex needs to be) and on the client side in terms of storedmetadata. The necessary size of the obliviously updatable indexis a function of the distribution of keywords, the rate of arrivalof keywords, and the amount of available local storage thatcan be used to buffer results on the client.

Client storage Because the local client is trusted, we are underno constraints as to how the storage is laid out and need notobscure its access pattern. As a result, it does not make a

difference if all of the storage is used holding an entry for asingle keyword appearing in two million emails or one millionkeywords each in one email. To a first approximation, clientstorage is directly proportional to the number of documentskeyword pairs (as we store less than 8 bytes per uniquekeyword). As shown in Figure 6, we use 62.7MB ± 13.2KBfor a 95th percentile user and 33.4MB ± 11.8KB for a 50th

percentile user.

Server storage for partial-block index Of course, once theclient’s storage is full, we must evict entries into the obliviouslyupdatable index, starting with the most full. For OUI, keyworddistribution comes into play. At some point the index will befull of infrequent words and we will be forced to evict partialblocks into the full-block index. The main questions we needto answer are 1) how often does this happen and 2) how largeof a partial-block index we need to ensure it does not happentoo soon.

To measure this directly would involve experiments span-ning the expected lifetime of a user’s mail account, whichis prohibitive to evaluate with a real implementation. Instead,we conduct a Monte Carlo style simulation. We draw wordsat random, according to the measured distribution of tokensdescribed in the previous section and measure how long ittakes before we are forced to evict a partially full block fromthe OUI. Our simulator merely keeps track of how space isallocated locally (client-side) and in the OUI and at what pointeach becomes full and forces an eviction. The simulator doesnot provide actual search results as the intention here is toassess storage requirements.

We assume a local store of 128MB and 64-bit emailidentifiers. We assume the cost of adding a new keywordto the index is 100 bits. Rather than specifying a fixed sizefor the OUI, we simply measure how many blocks would beneeded to fit the entire index.

11

Fig. 7: Simulated long term storage for IO-DSSE. Left: the size of the obliviously updatable index needed (i.e. server side storage)as time goes on for users of different activity levels. Right: the size of the client’s storage under the same conditions.

Our simulation validates that the general approach ofsplitting the index into a partial-block and full-block indexis viable, showing that even for a 95th percentile user, 2GBof partial-block index is sufficient to hold ten years worth ofemail or nearly 100 years of email for the average user.

As the graph in Figure 6 shows, however, there appearsto be no upper bound on the size of the index, short of thetotal number of observed words. This validates the followingintuition: evicting frequent words from the OUI when the blockis full ultimately does not free enough space for infrequentwords. That space will immediately be occupied by (in manycases the same) frequent word. The long tail of keywordscauses problems. To accommodate this, providers must either(1) limit the number of indexed words (e.g. to English words),or (2) limit the amount of indexed email. In the case of alimited set of keywords, the obliviously updatable index wouldhave a fixed size.

Deletes Recall, we can only delete emails from the obliviouslyupdatable index, so any index entries that have been evictedfrom the OUI are permanent. How long does this give us tofully delete a message (i.e. all index entries resulting fromthat message’s arrival)? For a 95th percentile user, the mostfrequent word is evicted 6724.8± 3.22490 times in 3650 daysor 2 evictions per day. For such a user, all index entries for anemail can only be deleted within a day of arrival. For the 50th

percentile user, on the other hand, where the most frequentword is evicted 319.4±0.843274 times in 3650 days, all indexentries for an email can be removed if the email is deletedwithin an expected 11 days of arrival.

V. RELATED WORK AND ATTACKS

A. Related work on searchable encryption

Searchable encryption has been studied in an extensive lineof works [5, 7, 8, 10, 15, 19]. Very few works have focusedon efficiency or locality. Cash et al. [5] provide the bestsuch approach. As we stressed in the introduction, however,this approach does not help in the dynamic case: all existing

techniques insert each document ID associated with a givenkeyword into a random location in the index.

B. Attacks

Recently Zang et al. [22] construct a highly effective queryrecovery attack on SSE schemes. The attack leverages thefact that an attacker who can insert entries into the index canconstruct them such that the subset of attacker files returneduniquely identifies the queried keyword. Effectively, this is amore efficient version of the attacker inserting one unique fileper possible keyword which contains only that single keyword.As this attack requires an adversary to insert files into the index,it cannot be mounted by an adversary who wishes to passivelysurveil many users. None-the-less, our scheme is subject to theattack, and indeed if used for email, highly susceptible due tothe ease with which an attacker can insert files. This makesthe scheme useful for end-to-end encryption settings whichprotect against dragnet surveillance. However, it should not beused in scenarios where greater protection against an activelymalicious mail server is needed. Also, as the attack dependsonly on observing retrieved files, there appears to be no simplecountermeasures. Forward private SSE schemes [3] do thwartthe adaptive version of the attack that requires injecting fewerfiles, but again at a large locality cost. Moreover as Zang etal. make clear, the non-adaptive attack still works even withforward privacy and that attack is readily mountable in thesetting of email.

We are hopeful that some method of injecting noise into theresults or merely detecting when this attack has been mountedwill be developed. As these techniques likely operate overthe logical structure of the index, our scheme should be fullycompatible with them. But absent such countermeasures, SSEin the email setting is only secure against passive adversariesregardless of its IO efficiency.

VI. EXTENSIONS AND CONCLUSIONS

Encrypted search for email or similar messaging systemsrepresents a major obstacle for E2E encrypted applications. All

12

existing built solutions place a prohibitively high IO cost onupdating the index on message arrival: requiring one randomwrite per document-keyword pair and one random read persearch. Using a hybrid approach where updates are done toa dynamic ORAM-like index and then evicted to a chunkedindex typically used for static searchable encryption, we areable to reduce the total IO usage by 99%, and by building adynamic index that does not protect read privacy, we are ableto achieve a 94% reduction in the upfront costs of search.

Our approach, of course, is still more expensive than non-encrypted search, and deploying for email is, in the end, a cost-benefit analysis between the value of protecting user privacyand the operating cost. But this is at least now a trade-off that isfar easier to make given our performance improvements. Indeed,without such a reduction in IO cost, the cost of encrypted searchfor email is too high.

Achieving this comes at some cost. First, we must slightlyrelax the leakage function for searchable encryption: an attackerlearns when entries are moved from the partial to full-blockindex, and we leak slightly more to an active attacker. Second, atpresent, we only support deletes from the obliviously updatableindex, and third, we only provide single keyword search.

Future work and extensions The techniques of Cash et. al [5]can readily be applied to our approach to get conjunctive search.We can simply use their (or any other similar) scheme directlyfor the full-block index. In their scheme, a keyword is associatedwith a key used specifically for computing intersection tagsxtag based on indexes and set of all such tags xSET is storedby the server. Because no additional data is associated withthe index entries on the server, and tags can safely be addedto xSET without additional leakage, this technique can beapplied to our approach simply by having the server store thetag set.

To deal with deletes, it is possible to incrementally rebuildthe index. Instead of appending to the full index on eviction,we can with some small probability overwrite a block in thefull index with one evicted from the partial index, storingthe overwritten block locally and then incrementally feedingthe non-deleted entries back. Thus the entire index wouldperiodically be refreshed. However, careful analysis of thespecific rate of deletion is needed to check if this approachprovides any practical benefit.

Finally, it is an interesting question whether similar relax-ations to ORAM security can be used to build an obliviouslyupdatable index with something other than Path ORAM andwith even better efficiency.

ACKNOWLEDGMENTS

This work was supported in part by The National ScienceFoundation under award CNS-1228443. We would also like tothank David Cash, Seny Kamara, Charalampos Papamanthouand the anonymous reviewers for discussions and helpfulcomments.

REFERENCES

[1] BLASS, E.-O., MAYBERRY, T., NOUBIR, G., AND ONARLIOGLU, K.Toward robust hidden volumes using write-only oblivious ram. InProceedings of the 2014 ACM SIGSAC Conference on Computer andCommunications Security (2014), ACM, pp. 203–214.

[2] BONEH, D., MAZIERES, D., AND POPA, R. A. Remote obliviousstorage: Making oblivious ram practical.

[3] BOST, R. Sophos - forward secure searchable encryption. CryptologyePrint Archive, Report 2016/728, 2016. http://eprint.iacr.org/2016/728.

[4] CASH, D., JAEGER, J., JARECKI, S., JUTLA, C., KRAWCZYK, H.,ROSU, M.-C., AND STEINER, M. Dynamic searchable encryption invery-large databases: Data structures and implementation. In Networkand Distributed System Security Symposium (NDSS’14) (2014).

[5] CASH, D., JARECKI, S., JUTLA, C., KRAWCZYK, H., ROSU, M.-C.,AND STEINER, M. Highly-scalable searchable symmetric encryptionwith support for boolean queries. In Advances in Cryptology–CRYPTO2013. Springer, 2013, pp. 353–373.

[6] CASH, D., AND TESSARO, S. The locality of searchable symmetricencryption. In Advances in Cryptology–EUROCRYPT 2014. Springer,2014, pp. 351–368.

[7] CHASE, M., AND KAMARA, S. Structured encryption and controlleddisclosure. In Advances in Cryptology-ASIACRYPT 2010. Springer, 2010,pp. 577–594.

[8] CURTMOLA, R., GARAY, J., KAMARA, S., AND OSTROVSKY, R.Searchable symmetric encryption: improved definitions and efficientconstructions. In Proceedings of the 13th ACM conference on Computerand communications security (2006), ACM, pp. 79–88.

[9] DAMGARD, I., MELDGAARD, S., AND NIELSEN, J. B. Perfectly secureoblivious ram without random oracles. In Theory of Cryptography.Springer, 2011, pp. 144–163.

[10] GOH, E.-J., ET AL. Secure indexes. IACR Cryptology ePrint Archive2003 (2003), 216.

[11] GOLDREICH, O. Towards a theory of software protection and simulationby oblivious rams. In Proceedings of the nineteenth annual ACMsymposium on Theory of computing (1987), ACM, pp. 182–194.

[12] GOLDREICH, O., AND OSTROVSKY, R. Software protection andsimulation on oblivious rams. Journal of the ACM (JACM) 43, 3(1996), 431–473.

[13] KAMARA, S., AND PAPAMANTHOU, C. Parallel and dynamic searchablesymmetric encryption. In Financial cryptography and data security.Springer, 2013, pp. 258–274.

[14] KAMARA, S., PAPAMANTHOU, C., AND ROEDER, T. Dynamicsearchable symmetric encryption. In Proceedings of the 2012 ACMconference on Computer and communications security (2012), ACM,pp. 965–976.

[15] NAVEED, M., PRABHAKARAN, M., AND GUNTER, C. A. Dynamicsearchable encryption via blind storage. Cryptology ePrint Archive,Report 2014/219, 2014. http://eprint.iacr.org/.

[16] PAPPAS, V., KRELL, F., VO, B., KOLESNIKOV, V., MALKIN, T., CHOI,S. G., GEORGE, W., KEROMYTIS, A., AND BELLOVIN, S. Blind seer:A scalable private dbms. In Security and Privacy (SP), 2014 IEEESymposium on (2014), IEEE, pp. 359–374.

[17] POPA, R. A., REDFIELD, C., ZELDOVICH, N., AND BALAKRISHNAN,H. Cryptdb: protecting confidentiality with encrypted query processing.In Proceedings of the Twenty-Third ACM Symposium on OperatingSystems Principles (2011), ACM, pp. 85–100.

[18] SHI, E., CHAN, T.-H. H., STEFANOV, E., AND LI, M. Obliviousram with o ((logn) 3) worst-case cost. In Advances in Cryptology–ASIACRYPT 2011. Springer, 2011, pp. 197–214.

[19] SONG, D. X., WAGNER, D., AND PERRIG, A. Practical techniques forsearches on encrypted data. In Security and Privacy, 2000. S&P 2000.Proceedings. 2000 IEEE Symposium on (2000), IEEE, pp. 44–55.

[20] STEFANOV, E., PAPAMANTHOU, C., AND SHI, E. Practical dynamicsearchable encryption with small leakage. IACR Cryptology ePrintArchive 2013 (2013), 832.

[21] STEFANOV, E., VAN DIJK, M., SHI, E., FLETCHER, C., REN, L., YU,X., AND DEVADAS, S. Path oram: An extremely simple oblivious ramprotocol. In Proceedings of the 2013 ACM SIGSAC conference onComputer & communications security (2013), ACM, pp. 299–310.

[22] ZHANG, Y., KATZ, J., AND PAPAMANTHOU, C. All your queriesare belong to us: The power of file-injection attacks on searchableencryption. Cryptology ePrint Archive, Report 2016/172, 2016. http://eprint.iacr.org/2016/172.

13