Robust and Efficient Elimination of Cache and Timing SideChannels

Benjamin A. Braun1, Suman Jana1, and Dan Boneh1

1Stanford University

ABSTRACTTiming and cache side channels provide powerful attacksagainst many sensitive operations including cryptographicimplementations. A popular strategy for defending againstthese attacks is to modify the implementation so that thetiming and cache access patterns of every hardware instruc-tion is independent of the secret inputs. However, this solu-tion is architecture-specific, brittle, and difficult to get right.In this paper, we propose and evaluate a robust low-overheadtechnique for timing and cache channel mitigation that re-quires only minimal source code changes and works acrossmultiple platforms and architectures. We report the exper-imental results of our solution for C, C++, and Java pro-grams. Our results demonstrate that our solution success-fully eliminates the timing and cache side-channel leaks andincurs significantly lower performance overhead than exist-ing approaches.

1. INTRODUCTIONDefending against cache and timing side channel attacks

is known to be a hard and important problem in computersecurity. Timing attacks can be used to extract crypto-graphic secrets from running systems [26, 15, 33, 14], spyon Web user activity [12], and even undo the privacy of dif-ferential privacy systems [23, 5]. Attacks exploiting timingside channels have been demonstrated for remote attack-ers, where the adversary and the target are separated bya network [26, 15, 33, 14] and for local attackers, wherethe adversary runs unprivileged spyware on the target ma-chine [33, 8, 11, 42, 7, 44].

One way to defend against remote timing attacks is tomake sure that the timing of any externally observable eventsare independent of any data that should be kept secret. Sev-eral strategies have been proposed to achieve this, includingapplication-specific changes [27, 10, 10], static transforma-tion [17, 19] and dynamic padding [6, 44, 18, 28, 23].

These strategies do not defend against local timing at-tacks where the attacker spies on the target applicationby measuring the target’s impact on the local cache andother resources. Some strategies for defending against lo-cal attacks include static partitioning of resources [25, 34,40, 41], flushing state [47], obfuscating cache access pat-terns [8, 10, 13, 37, 32], and moderating access to fine-grained timers [31, 30, 39]. We survey these methods inrelated work (Section 8).

A popular approach for defending against local and re-mote timing attacks is to ensure that the low-level instruc-tion sequence does not contain instructions whose perfor-

mance depends on secret information. This is enforced bymanually re-writing the code, as was done in OpenSSL1, orby changing the compiler to ensure that the generated codehas this property [19].

This popular strategy can fail to ensure security for sev-eral reasons. First, the timing properties of instructions candiffer in subtle ways from one architecture to another caus-ing this approach may produce an instruction sequence thatis unsafe for some architectures. Second, this strategy canfail for languages like Java where the JVM can optimizethe bytecode at runtime and inadvertently introduce secret-dependent timing variations. Third, manually ensuring thata certain code transformations prevent timing attacks can bedifficult, as was the case when updating OpenSSL to preventthe Lucky-thirteen timing attack [29].

Our contribution. We propose the first low-overhead,cross-architecture defense that can protect against both localand remote timing attacks with minimal application codechanges. We show that our defense is language-independentby applying the strategy to protect applications written inJava and C/C++. Our defense requires relatively simplemodifications to the underlying OS and can run on off-the-shelf hardware.

The key insights behind our solution (Section 4) are that:

• The necessary time padding can be minimized by ac-curately accounting for the different causes for timingvariations. We eliminate timing variations that areindependent of protected data (e.g., caused by inter-rupts, the OS scheduler, or non-secret execution flow).Our accounting ensures that the time pad is the min-imum needed to prevent timing variation that dependon secret data.

• The OS scheduler can be leveraged to isolate a sen-sitive function’s execution from other untrusted pro-cesses without incurring significant performance over-head. Dynamic resource isolation techniques like pagecoloring can further help in isolation of shared resources.Lazy state cleansing mechanisms for shared resourcescan amortize the costs of state cleansing while stillmaintaining the security guarantees.

• We fully implemented our approach in Linux and showthat execution times are independent of secret dataand that performance overhead is low. For exam-ple, the performance overhead to protect the entirestate machine running inside a SSL/TLS server against

1In the case of RSA private key operations, OpenSSL usesan additional defense called blinding.

arX

iv:1

506.

0018

9v1

[cs

.CR

] 3

1 M

ay 2

015

known timing- and cache-based side channel attacks isless than 8.5% in connection latency.

• We show that our solution can defend applicationswritten in different languages like C, C++, and Javawith minimal application code changes. For Java ourdefense is implemented by inserting a small number ofnative code calls into the bytecode.

Overall we obtain an efficient application-independent so-lution that defends against a wide range of timing- andcache-based side channel attacks.

2. KNOWN TIMING ATTACKSBefore describing our proposed defense we briefly survey

different types of timing attackers. In the previous sectionwe discussed the difference between a local and a remotetiming attacker: a local timing attacker, in addition to mon-itoring the total computation time, can spy on the targetapplication by monitoring the state of shared hardware re-sources such as the local cache.

Concurrent vs. non-concurrent timing attacks.In a concurrent attack, the attacker can probe shared re-

sources while the target application is operating. For ex-ample, the attacker can inspect the state of the shared re-sources at intermediate steps of a cryptographic operation.The attacker’s process can control the concurrent access byadjusting its scheduling parameters and its core affinity inthe case of symmetric multiprocessing (SMP).

Concurrent, local attacks are the most prevalent class oftiming attacks in the research literature. Such attacks areknown to be able to extract the secret/private key againsta wide-range of ciphers including RSA [33, 4], AES [37, 43,22, 32], and ElGamal [46]. These attacks exploit informa-tion leakage through a wide range of shared hardware re-sources: the L1 or L2 data cache [37, 33, 22, 32], instructioncache [1, 46], branch predictor cache [2, 3], floating-pointmultiplier [4], and the L3 cache [43, 24].

A non-concurrent attack is one in which the attacker onlygets to observe the timing information or shared hardwarestate at the beginning and the end of the sensitive compu-tation. For example, a non-concurrent attacker can extractsecret information using only the aggregate time it takes thetarget application to process a request.

All existing remote attacks are non-concurrent, howeverthis is not fundamental. A hypothetical remote, yet con-current, attack would be one in which the remote attackersubmits a request to the victim application at the same timethat another non-adversarial client makes requests to thevictim application. The attacker would then use the tim-ing information collected from this setup to learn somethingabout the non-adversarial client’s communication with theserver.

There are several known local non-concurrent attacks aswell. Osvik et al. [32], Tromer et al. [37], and Bonneau andMironov [11] present two types of local, non-concurrent at-tacks against AES implementations. In the first, prime andprobe, the attacker “primes” the cache, triggers an AES en-cryption, and “probes” the cache to learn information aboutthe AES private key. The spy process primes the cache byloading its own memory content into the cache and probesthe cache by measuring the time to reload the memory con-tent after the AES encryption has completed. This attack

involves the attacker’s spy process measuring its own timinginformation to indirectly extract information from the vic-tim application. Alternatively, in the evict and time strat-egy, the attacker measures the time taken to perform thevictim operation, evicts certain chosen cache lines, triggersthe victim operation and measure its execution time again.By comparing these two execution times, the attacker canfind out which cache lines were accessed during the victimoperation. Osvik et al. were able to extract an 128-bit AESkey after only 8,000 encryptions using the prime and probeattack.

3. THREAT MODELWe allow the attacker to be local or remote and to exe-

cute concurrently or non-concurrently with the target appli-cation.

We assume that the attacker can only run spy processesas a different non-privileged user (i.e., without super-userprivileges) than the owner of a target process. We also as-sume that the spy process cannot bypass the standard user-based isolation provided by the OS. We believe that theseare very realistic assumptions because if either one of theseassumptions fail, the spy process can steal the user’s sensi-tive information without resorting to side channel attacks inmost existing OSes.

We assume that the operating system and the underlyinghardware are trusted. Similarly, we assume that the attackerdoes not have physical access to the hardware and cannotmonitor side channels such as electromagnetic radiations,power use, or acoustic emanations. We are only concernedwith timing and cache side channels since they are the easiestside channels to exploit without physical access to the victimmachine.

4. OUR SOLUTIONIn our solution, a developer first annotates the sensitive

computation(s) in their code that they would like to protectat the granularity of functions. For the rest of the paper,we refer to such functions as protected functions. Oursolution ensures that the code inside the protected func-tions, all other functions that may be invoked as part oftheir execution, and all the secrets that they operate on areprotected from timing attacks. We also support nested pro-tected functions i.e. one protected function can call otherprotected functions. Our solution instruments the protectedfunctions such that our stub code is invoked before and afterexecution of each protected function.

We defend a protected function against all known formsof timing attacks—remote, local non-concurrent, and localconcurrent attacks—using two high-level mechanisms.

Time padding. We use time padding to defend againstremote timing attacks and any local attack that measurethe end-to-end runtime of a protected function. Essentially,we pad the protected function’s execution time to its worst-case runtime. Time padding has been used in past propos-als [6, 44, 18, 28, 23], however previous solutions incurredhigh performance overheads. Our main contributions hereare twofold:

• Computing the padding amount adaptively by sepa-rating out the secret-dependent sources of timing vari-ations from external secret-independent sources (i.e.,OS scheduler, interrupt handlers, etc.) and

• Designing a safe padding mechanism that avoids theinformation leakage occurring under naive padding ap-proaches.

We show that time padding can be implemented efficientlywhile ensuring strong security guarantees.

A safe time padding scheme defends against attackersthat measure the duration of a protected function using thetimestamp counter or any other measure of real time. How-ever, most modern hardware often also contain performancecounters that keep track of different performance events suchas the number of cache evictions or branch mispredictionsoccurring on a particular core. A local attacker with accessto these performance counters may infer the secrets usedduring the sensitive computation despite time padding. Oursolution, therefore, restricts access to performance monitor-ing counters so that a user’s process cannot see detailedperformance metrics of another user’s processes. We do notrestrict, however, a user from using hardware performancecounters to measure the performance of their own processes.

Shared resource isolation. Our solution also prevents in-formation leakage through shared resources across differentusers’ processes. We do this by dynamically reserving exclu-sive access to a physical core (including all per-core cachessuch as L1 and L2) while it is executing a sensitive function.This ensures that a local attacker does not have concurrentaccess to any per-core resources while a protected functionis accessing them. We lazily cleanse the state left by theprotected function in any of the per-core resources beforehanding them over to untrusted processes. Finally, we usepage coloring to ensure that protected functions of a userdo not perform any accesses outside of a reserved portionof the L3 cache, and to ensure that this reserved portion isnot shared with other users. This ensures that the attackercannot infer information about protected functions throughthe L3 cache.

We describe the components of time padding and sharedresource isolation in detail next.

4.1 Time padding

4.1.1 Computing the adaptive padding thresholdOne key insight behind our solution is that there are two

major sources of variation in a protected function’s execu-tion time: secret-dependent variations and secret-independentvariations. For example, program delays caused by the OSscheduler preemptions or interrupt handling are indepen-dent of accesses to a program’s secret data inside protectedfunctions. Existing implementations of time padding incurextremely high overheads because they try to pad the exe-cution time up to the worst-case in the presence of a largenumber of secret-independent variations. However, in oursolution, we separate out the secret-independent variationsfrom the secret-dependent ones and compute the paddingamounts for these two cases separately. Most modern OSeslike Linux keep track of the number of such external pre-emptions, and hence we can adapt the amount of paddingdepending on how many times a protected function is pre-empted. The padding mechanism we present will pad tosome amount of time which the attacker observes, Tobserved,where Tobserved = Text preempt + Tmax. Tmax is the worstcase execution time of the protected functions when no ex-ternal preemptions occur, and Text preempt is the worst-casetime spent during preemptions given the set of preemptions

time

Padding target:

LeakFigure 1: Time leakage due to naive padding

(preempt1, preempt2, ...preemptn) that occur during the ex-ecution of the protected function or the added padding. Notethat the number of external preemptions, n, is independentof the secret and thus the attacker does not learn anythingsensitive by observing the total padded time Tobserved.

In practice, we estimate Tmax through offline profiling ofthe protected function. Since this value is machine-specific,we perform this profiling on any machine which will runprotected functions. Also, even though Tobserved does notleak information about secrets, padding to this value will becostly if Text preempt is high, due to frequent or long-runningpreemptions during the protected function. Therefore, weminimize external events that can delay the execution of theprotected function. We describe the main external sourcesof delay during a protected function’s execution and how wedeal with each one of them in detail below.

CPU frequency scaling. Modern CPUs include mech-anisms to change the operating frequency of each core dy-namically at runtime depending on the current workload tosave power. If a core’s frequency decreases in the middle ofthe execution of a protected function or it enters the haltstate, it will take longer in real-time, increasing Tmax. Toreduce such variations, we disable CPU frequency scalingand low-power CPU states when a core executes a protectedfunction.

Paging. If an attacker can cause memory paging eventsduring the execution of a protected function, she can ar-bitrarily slow down the protected function. To avoid suchcases, our solution forces a process executing the protectedfunction to lock all of its pages in memory and disables pageswapping. Our solution currently does not allow processesthat allocate more memory than is physically available inthe target system to use protected functions.

Hyperthreading. Hyperthreading is a technique sup-ported by modern processor cores where one physical coresupports multiple logical cores. The OS can schedule taskson these logical cores independently and the hardware takescare of sharing the underlying physical core. We observedthat protected functions executing on a core with hyper-threading enabled can encounter large amounts of slowdown.This slowdown is caused because the other concurrent pro-cesses executing on the same physical core can interfere withaccess to some of the CPU resources.

One potential way of avoiding this slowdown is to config-ure the OS scheduler to prevent another process from run-ning concurrently on a physical core with a process in themiddle of a protected function. However, such a mechanismresults in high overheads either due to waiting on anotherrunning process to be scheduled off of a virtual core prior torunning a protected function, or due to the cost of actively

unscheduling or migrating a process running on another vir-tual core. For our current prototype implementation, wesimply disable hyperthreading as part of system configura-tion.

Preemptions by other user processes. Under regu-lar circumstances, a protected function can be preemptedby other user processes. This can delay the execution ofthe protected function as long as the process is preempted.Therefore, we need to minimize such preemptions while stillkeeping the system usable.

In our solution, we prevent preemptions by other user pro-cesses during the execution of a protected function by usinga scheduling policy that prevents migrating the process toa different core and prevents other user processes from pre-empting a process while the process is running a protectedfunction.

Preemptions by interrupts. Another common source ofpreemption is the hardware interrupts served by the core ex-ecuting a protected function. One way to solve this problemis to block or rate limit the number of interrupts that can beserved by a core while executing a protected function. How-ever, such a technique may make the system non-responsiveunder heavy load. For this reason, in our current prototypesolution, we do not apply such techniques.

Note that some of these interrupts (e.g., network inter-rupts) can be triggered by the attacker and thus can be usedby the attacker to slow down the protected function’s exe-cution. However, in our solution, such an attack increasesText preempt, and hence degrades performance, but does notcause information leakage.

4.1.2 Safely applying paddingOnce the padding amount has been determined using the

techniques described above, waiting the target amount mightseem easy at first glance. However, there are two major is-sues that make application of padding complicated in prac-tice as described below.

Handling limited accuracy of padding loops. Figure 1shows that a naive padding scheme that repeatedly mea-sures the elapsed time in a tight loop until the target time isreached leaks timing information. This is because the loopcan only break when the condition is evaluated, and henceif one iteration of the loop takes u cycles then the paddingloop leaks timing information mod u.

Our solution guarantees that the distribution of runningtimes of a protected function for some set of private inputs isindistinguishable from the same distribution produced whena different set of private inputs to the function are used. Wecall this property the safe padding property. We over-come the limitations of the simple wait loop by performinga timing randomization step before entering the simple waitloop. During this step, we perform m rounds of a random-ized waiting operation. This goal of this step is to ensurethat the amount of time spent in the protected functionbefore the beginning of the simple wait loop, when takenmodulo u, the stable period of the simple timing loop (i.e.disregarding the first few iterations), is close to uniform.This technique can be viewed as performing a random walkon the integers modulo u where the runtime distribution ofthe waiting operation is the support of the walk and m is thenumber of steps walked. Prior work by Chung et al. [16] hasexplored the sufficient conditions for the number of steps in

a walk and its support that produce a distribution that isexponentially close to uniform.

For the purposes of this paper, we perform timing ran-domization using a randomized operation with 256 possibleinputs that runs for X + c cycles on input X where c isa constant. We make this operation concrete in Section 5.We then choose m to defeat our empirical statistical testsunder pathological conditions that are very favorable to anattacker as shown in Section 6.

For our scheme’s guarantees to hold, the randomness usedinside the randomized waiting operation must be generatedusing a cryptographically secure generator. Otherwise, ifan attacker can predict the added random noise, she cansubtract it from the observed padded time and hence derivethe original timing signal, modulo u.

A padding scheme that pads to the target time Tmax us-ing a simple padding loop and performs the randomizationstep after the execution of the protected function will notleak any information about the duration of the protectedfunction, as long as the following conditions hold: (i) nopreemptions occur; (ii) the randomization step successfullyyields a distribution of runtimes that is uniform modulo u;(iii) The simple padding loop executes for enough iterationsso that it reaches its stable period. The security of thisscheme under these assumptions can be proved as follows.

Let us assume that the last iteration of the simple waitloop take u time. Assuming the simple wait loop has iter-ated enough times to reach its stable period, we can safelyassume that u does not depend on when the simple waitloop started running. Now, due to the randomization step,we assume that the amount of time spent up to the start ofthe last iteration of the simple wait loop, taken modulo u, isuniformly distributed. Hence, the loop will break at a timethat is between the target time and the target time plus u−1.Because the last iteration began when the elapsed executiontime was uniformly distributed modulo u, these u differentcases will occur with equal probability. Hence, regardlessof what is done within the protected function, the paddedduration of the function will follow a uniform distributionof u different values after the target time. Therefore, theattacker will not learn anything from observing the paddedtime of the function.

To reduce the worst-case performance cost of the random-ization step, we generate the required randomness at thestart of the protected function, before measuring the starttime of the protected function. This means that any vari-ability in the runtime of the randomness generator does notincrease Tmax.

Handling preemptions occurring inside the paddingloop. The scheme presented above assumes that no ex-ternal preemptions can occur during the the execution ofthe padding loop itself. However, blocking all preemptionsduring the padding loop will degrade the responsiveness ofthe system. To avoid such issues, we allow interrupts to beprocessed during the execution of the padding loop and up-date the padding time accordingly. We repeatedly updatethe padding time in response to preemptions until a “safeexit condition” is met where we can stop padding.

Our approach is to initially pad to the target value Tmax,regardless of how many preemptions occur. We then re-peatedly increase Text preempt and pad to the new adjustedpadding target until we execute a padding loop where nopreemptions occur. The pseudocode of our approach is show

// At the return point of a protected function:// Tbegin holds the time at function start// Ibegin holds the preemption count at function start

1 for j = 1 to m2 Short-Random-Delay()3 Ttarget = Tbegin + Tmax

4 overtime = 05 for i = 1 to ∞6 before = Current-Time()7 while Current-Time() < Ttarget, re-check.8 // Measure preemption count and adjust target9 Text preempt = (Preemptions()− Ibegin) · Tpenalty

10 Tnext = Tbegin + Tmax + Text preempt + overtime11 // Overtime-detection support12 if before ≥ Tnext and overtime = 013 overtime = Tovertime

14 Tnext = Tnext + overtime15 // If no adjustment was made, break16 if Tnext = Ttarget

17 return18 Ttarget = Tnext

Figure 2: Algorithm for applying time padding to a protectedfunction’s execution.

in Figure 2. Our technique does not leak any informationabout the actual runtime of the protected function as thefinal padding target only depends on the pattern of preemp-tions but not on the initial elapsed time before entering thepadding loops. Note that forward progress in our paddingloops is guaranteed as long as some form of preemptions arerate limited on the cores executing protected functions.

The algorithm computes Text preempt based on observedpreemptions simply by multiplying a constant Tpenalty bythe number of preemptions. Since Text preempt should matchthe worst-case execution time of the observed preemptions,Tpenalty is the worst-case execution time of any single pre-emption. Like Tmax, Tpenalty is machine specific and can bedetermined empirically from profiling data.

4.1.3 Determining worst-case execution timeWe estimate Tmax, the worst-case execution time of a pro-

tected function using profiling data collected during an of-fline profiling phase. Since the worst-case execution timedepends on the target machine, we perform this offline pro-filing on individual target machines.

To gather profiling information, we run an applicationthat invokes protected functions with a input generatingscript either provided by the application or the system ad-ministrator. We instrument the protected functions in theapplication so that the worst-case performance behavior isstored in a profile file. We compute the padding parametersbased on the output profiling results. To reduce the possi-bility of overtimes occurring due to uncommon inputs, it isimportant that we profile for both common and uncommoninputs.

To be conservative, we obtain all profiling measurementsfor the protected functions under high load conditions (i.e.in parallel with programs that stress test both memory andCPU). From these measurements, we compute Tmax so thatit is a worst-case bound when at most a κ fraction of profil-

ing readings are excluded, where κ is a security parameter.Higher values of κ increase the frequency of overtimes but re-duce Tmax, hence κ represents a performance-security trade-off. For our prototype implementation we set κ to 10−5.

4.1.4 Handling overtimesDue to the immense size of the input-space of most pro-

tected functions, despite our best efforts, we might misssome pathological input that take significantly longer timethan other inputs. If such a pathological input appears inthe wild, the protected function may take longer than esti-mated worst-case bound and this will result in an overtimeand leak information. We therefore augment our techniqueto detect overtimes, that is, when the elapsed time of theprotected function, even taking the interrupts into account,is greater than the computed padding target. When an over-time is detected, we pad to a significantly larger value thanone would expect the function to take. This ensures thateach overtime only leaks 1 bit of information.

To further limit leakage, if a sufficient number of over-times are detected, we can also alert the system administra-tor and potentially refuse to service requests. The systemadministrator can then act on this alert by updating thesecret inputs (e.g., secret keys) and increasing the parame-ters of the model Tmax to avoid future overtimes. We sup-port updating the padding parameters of a protected func-tion on the fly without restarting running applications. Thepadding parameters are stored in a file that has the sameaccess permissions as the application/library containing theprotected function. This file is memory-mapped when thecorresponding protected function is called for the first time.Any changes to the memory-mapped file will immediatelyimpact the padding parameters of any application invokingthe protected function except the ones that are in the middleof the padding step.

4.2 Shared resource isolationIn our solution, isolation of shared resources are imple-

mented in two ways —isolating shared resources betweenconcurrent processes and cleansing state left in shared re-sources before handing them over to other untrusted pro-cesses. Isolating shared resources help in both preventingtiming attacks from other concurrent processes, and alsoimproving performance by minimizing variations in the run-time of protected functions. For example, consider the casewhere we disable hyperthreading during a protected func-tion’s execution to improve performance as described earlier.This also ensures that an attacker cannot run spy code thatsnoops on per-core state while a protected function is execut-ing. Also, preventing preemptions from other user processesduring the execution of protected function ensures that thecore and its L1/L2 caches are dedicated for the protectedfunction.

To enforce resource isolation, we also make several mod-ifications throughout the entire life-cycle of processes thatinvokes protected functions. We described the details be-low.

Changes to system initialization. We use page coloringto dynamically isolate the protected function’s data in theL3 cache. To provide page coloring, at boot time, the OSinitializes physical page allocators that do not allocate pageshaving any of C reserved ”secure” page colors, unless thecaller specifically requests a secure color. Pages are colored

based on which L3 cache sets a page maps to. Therefore, twopages with different colors are guaranteed never to conflictin the L3 cache in any of their cache lines.

Changes to system configuration. In order to supportpage coloring, the system configuration script disables trans-parent huge pages and sets up access control to huge pages.An attacker that has access to a huge page can evade theisolation provided by page coloring, since a huge page canspan multiple page colors. Hence, we bar access to hugepages (transparently or by request) to non-privileged users.

Also as part of page coloring, the script disables memorydeduplication features, such as kernel same-page merging.This prevents a secure-colored page mapped into one pro-cess from being transparently mapped as shared into anotherprocess. Disabling memory deduplication has been used inthe past in hypervisors to prevent leakage of informationacross different virtual machines [36].

Changes to process state. During the initialization of theprocess calling protected functions, a kernel module routineis called that remaps all pages allocated by the process inprivate mappings (i.e., the heap, stack, text-segment, librarycode, and library data pages) to pages that are not sharedwith processes of any other user and that have a page colorreserved by the user. The remapping transparently changesthe physical pages that a process accesses without modifyingthe virtual memory addresses, and hence requires no specialapplication support. If the user has not yet reserved anypage colors or there are no more remaining pages of anyof her reserved page colors, the OS allocates one of the re-served colors for the user. Also, the process is flagged witha ”secure-color” bit. We modify the OS so that it recognizesthis flag and ensures that the future pages allocated to aprivate mapping for the process will come from a reservedpage color for the user. Note that since we only remap pri-vate mappings, we do not protect applications that access ashared mapping from inside a protected function.

This strategy for allocating page colors to users has theminor potential downside that such a system restricts thenumbers of different users’ processes that can concurrentlycall protected functions. We believe that such a restrictionis a reasonable trade-off between security and performance.

Changes when a protected function returns. To en-sure that an attacker does not see the tainted state in aper-core resource after a protected function, when a pro-tected function returns we mark the CPU as “tainted” withthe user ID of the caller process. The next time a the OSattempts to schedule a process from a different user on thecore, it will first flush all per-CPU caches, including the L1instruction cache, L1 data cache, L2 cache, Branch Transla-tion Buffer (BTB), and Translation lookaside buffer (TLB).Such a scheme ensures that the overhead of flushing thesecaches can be amortized over multiple invocations of pro-tected functions by the same user.

5. IMPLEMENTATIONWe implement a prototype implementation of our protec-

tion mechanism for the Linux OS running on the Intel SandyBridge architecture. We describe the different componentsof our implementation below.

5.1 Programming APIWe implement a new function annotation FIXED TIME

for the C/C++ language that indicates that a function shouldbe protected. The annotation can be specified either in thedeclaration of the function or at its definition. Adding thisannotation is the only change to a C/C++ codebase thata programmer must make to use our solution. We wrotea plugin for the Clang C/C++ compiler that handles thisannotation. The plugin automatically inserts a call to thefunction fixed time begin at the start of the protected func-tion and a call to fixed time end at any return point of thefunction. These functions protect the annotated functionusing the mechanisms described in Section 4.

Alternatively, a programmer can also call these functionsexplicitly. This is needed for protecting ranges of code withinfunction such as the state transitions of the TLS state ma-chine (see Section 6.1). We provide a Java native interfacewrapper to both fixed time begin and fixed time end func-tions, for supporting protected functions written in Java.

5.2 Time paddingFor implementing time padding loops, we read from the

timestamp counter in x86 processors to collect time measure-ments. In most modern x86 processors, including the one wetested on, the timestamp counter has a constant frequencyregardless of the power saving state of a processor. We gen-erate pseudorandom bytes for the randomized padding stepusing the ChaCha/8 stream cipher [9]. We use a value of15 µs for Tpenalty as this bounds the worst-case slowdowndue to a single interrupt we observed in our experiments.

The randomized wait operation we use takes an input Xand simply performs X+c noops in a loop, where c is a largeenough value so that the loop takes one cycle longer for eachadditional iteration. We observe that c = 46 is sufficient toachieve this property.

Some of the OS modifications specified in our solution areimplemented as a loadable kernel module. This module sup-ports an IOCTL to mark the core as tainted at the end of aprotected function. The module also supports a IOCTL callthat enables fast access to the interrupt and context-switchcount. In the standard Linux kernel, the interrupt countis usually accessed through the proc file system interface.However, such an interface is too slow for our purposes. In-stead, the kernel module allocates a page of counters that ismapped into the calling process’ virtual address space andis also pointed to in its task struct. We modify the kernel tocheck on every interrupt and context switch if the currenttask has such a page, and if so, to increment the correspond-ing counter in that page.

Offline profiling. We provide a profiling wrapper script,fixed time record .sh, that computes worst-case executiontime parameters of each protected function as well as theworst-case slowdown on that function due to preemptionsby different interrupts or kernel tasks.

The profiling script automatically generates profiling in-formation for any protected functions in an executable byrunning the application on a user-provided inputs. Dur-ing the profiling process, we run a variety of applicationsin parallel to create a stress-testing environment that trig-gers worst-case performance of the protected function. Toallow the stress testers to maximally slow down the userapplication, we reset the scheduling parameters and CPUaffinity of a thread at the start and end of every protectedfunction. One stress tester generates interrupts at a highfrequency using a simple program that generates a flood of

UDP packets to the loopback network interface. We also runthe mprime2, systester3, and the LINPACK benchmark4 tocause high CPU load and large amounts of memory con-tention.

5.3 Shared resource isolationIsolating a processor core and core-specific caches.We disable hyperthreading in Linux by selectively disablingvirtual cores. This prevents any other processes from inter-fering with the execution of a protected function. As partof our prototype, we also implement a simple version of thepage coloring scheme as described in Section 4.

We prevent a user from observing hardware performancecounts showing the performance behavior of other users’ pro-cesses. The perf events framework on Linux mediates accessto hardware performance counters. We configure the frame-work to allow accessing per-cpu performance counters onlyby the privileged users. Note that an unprivileged user canstill access per-process performance counters that measurethe performance of their own processes.

For ensuring that a processor core executing a protectedfunction is not preempted by other user processes, As speci-fied in Section 4, we depend on a scheduling mode that pre-vents other userspace processes from preempting a protectedfunction. For this purpose, we use the Linux SCHED FIFOscheduling mode at maximum priority. In order to be ableto do this, we allow unprivileged users to use SCHED FIFOat priority 99 by changing the limits in the /etc/securi-

ty/limits.conf file.One side effect of this technique is that if a protected func-

tion manually yields to the scheduler or perform blockingoperations, the process invoking the protected function maybe scheduled off. Therefore, we do not allow any blockingoperations or system calls inside the protected function. Asmentioned earlier, we also disable paging for the processesexecuting protected functions by using the mlockall() sys-tem call with the MCL_FUTURE.

We detect whether a protected function has violated theconditions of isolated execution by determining whether anyvoluntary context switches occurred during the protectedfunction’s execution. This usually indicates that either theprotected function yield the CPU manually or performedsome blocking operations.

Flushing shared resources. We modify the Linux sched-uler to check the taint of a core before scheduling a userprocess on a processor core and to flush per-core resourcesif needed as described in Section 4.

To flush the L1 and L2 caches, we iteratively read overa segment of memory that is larger than the correspondingcache sizes. We found this to be significantly more efficientthan using the WBINVD instruction, which we observed costas much as 300 microseconds in our tests. We flush theL1 instruction cache by executing a large number of NOPinstructions.

Current implementations of Linux flush the TLB duringeach context switch. Therefore, we do not need to separatelyflush them. However, if Linux starts leveraging the PCIDfeature of x86 processors in the future, the TLB would have

2http://www.mersenne.org/3http://systester.sourceforge.net4https://software.intel.com/en-us/articles/intel-math-kernel-library-linpack-download/

to be flushed explicitly. For flushing the BTB, we leverageda “branch slide” consisting of alternating conditional branchand NOP instructions.

6. EVALUATIONTo show that our approach can be applied to protect a

wide variety of software, we have evaluated our solution inthree different settings and found that our solution success-fully prevents potential timing and cache attacks in all ofthese settings. We describe these settings below.

Encryption algorithms implemented in high level in-terpreted languages like Java. Traditionally, crypto-graphic algorithms implemented in interpreted languageslike Java have been harder to protect from timing attacksthan those implemented in low level languages like C. One ofthe key reasons behind this is the fact that virtual machinesfor the high level interpreted languages may contain data-dependent timing variations that are very hard to detect.While developers writing low level code can use features suchas in-line assembly to carefully control the machine code oftheir implementation, such low level control is simply notpossible in a higher level language.

We show that our techniques can take care of these issues.We demonstrate that our defense can make the computationtime of Java implementations of cryptographic algorithmsindependent of the secret key with minimal performanceoverhead.

Sensitive data structures. Besides cryptographic algo-rithms, timing channels also occur in the context of differentdata structure operations like hash table lookups. Hash ta-ble lookups may take different amount of time depending onhow many items are present in the bucket where the desireditem is located. It will take longer time to find items inbuckets with higher number of items than in the ones withless items. This signal can be exploited by an attacker tocause denial of service attacks [21].

We demonstrate that our technique can prevent timingleaks in two different hash table implementations: associa-tive arrays in C++ STL and HashMaps in Java.

Cryptographic operations and SSL/TLS state machine.Implementations of cryptographic primitives other than thepublic/private key encryption or decryption routines mayalso suffer from side channel attacks. For example, a cryp-tographic hash algorithm like SHA-1 takes different amountof time depending on the length of the input data. In fact,such timing variations have been used as part of several ex-isting attacks against SSL/TLS protocols (e.g., Lucky 13).Also, the time taken to perform the computation for imple-menting different stages of the SSL/TLS state machine mayalso be dependent on the secret key.

We find that our protection mechanism can protect cryp-tographic primitives like hash functions and individual stagesof the SSL/TLS state machine from timing attacks while in-curring minimal overhead.

Experiment setup. We perform all our experiments ona machine with 2.3GHz Intel Xeon CPUs organized in 2sockets each containing 6 physical cores. Each core has a32KB L1 instruction cache, a 32KB L1 data cache, and a256KB L2 cache. Each socket has a 15MB L3 cache. Themachine has a total of 64GB of RAM.

For our experiments, we use OpenSSL version 1.0.1l andJava version BouncyCastle 1.52 (beta). The test machine

0.00

0.05

0.10

0.15

0.20

0.25

0 20 40 60Duration (ns)

Fre

quen

cy

Input 0 1A. Unprotected

0.00

0.05

0.10

0.15

0.20

0.25

0 20 40 60Duration (ns)

Fre

quen

cy

B. With time padding but no randomized noise

0.00

0.05

0.10

0.15

0.20

2390 2400 2410Duration (ns)

Fre

quen

cy

C. Full protection (padding+randomized noise)

0.00

0.04

0.08

0.12

2390 2400 2410Duration (ns)

Fre

quen

cy

Figure 3: Defeated distinguishing attack.

runs Linux kernel version 3.13.11.4 with modifications asdiscussed in Section 5.

Preventing a simple timing attack. To determine theeffectiveness of our safe padding technique, we first testwhether our technique can protect against a large timingchannel that can distinguish between two different inputsof a simple function. To make the attacker’s job easier, wecraft a simple function that has an easily observable tim-ing channel—the function executes a loop for 1 iteration ifthe input is 0 and 11 iterations otherwise. We use the x86loop instruction to implement the loop and just a singlenop instruction as the body of the loop. We assume thatthe attacker calls the protected function directly and mea-sures the value of the timestamp counter immediately beforeand after the call. The goal of the attacker is to distinguishbetween two different inputs (0 and 1) by monitoring theexecution time of the function. Note that these conditionsare extremely favorable for an attacker.

We found that our defense completely defeats such a dis-tinguishing attack despite the highly favorable conditions foran attacker. Figure 3(A) shows the distributions of observedruntimes of the protected function on inputs 0 and 1 withno defense applied. Figure 3(B) shows the runtime distribu-tions where padding is added to reach Tmax = 5000 cycles(≈ 2.17 µs) without the time randomization step. In bothcases, it can be seen that the observed timing distributionsfor the two different inputs are clearly distinguishable. Fig-ure 3(C) shows the same distributions when m = 5 rounds oftiming randomization are applied along with time padding.

−5

−4

−3

−2

−1

0

0 1 2 3 4 5Rounds of noise

log1

0(E

mp.

sta

tistic

al d

ista

nce)

Inputs

0 vs. 1

0 vs. 0

Figure 4: The effect of multiple rounds of randomized noise ad-dition on the timing channel.

In this case, we are no longer able to distinguish the timingdistributions.

We quantify the possibility of success for a distinguishingattack in Figure 4 by plotting the variation of empirical sta-tistical distance between the observed distributions as theamount of padding noise added is changed. The statisticaldistance is computed using the following formula.

d(X,Y ) =1

2

∑i∈Ω

|P [X = i]− P [Y = i]|

We measure the statistical distance over the set of observa-tions that are within the range of 50 cycles on either sideof the median (this contains nearly all observations.) Eachdistribution consist of around 600 million observations.

The dashed line in Figure 4 shows the statistical distancebetween two different instances of the test function with 0as input. The solid line shows the statistical distance whereone instance has 0 as input and the other has 1. We observethat the attack can be completely prevented when at least2 rounds of noise are used.

Timing attack on RSA decryption We next evaluatethe effectiveness of our time padding approach to defeat thetiming attack by Brumley et al. [15] against unblinded RSAimplementations. Blinding is an algorithmic modification toRSA that uses randomness to prevent timing attacks. Toisolate the impact of our specific defense, we apply our de-fense to the RSA implementation in OpenSSL 1.0.1h withsuch constant time defenses disabled. To do so, we configureOpenSSL to disable blinding, use the non-constant time ex-ponentiation implementation, and use the non-word-basedMontgomery reduction implementation. We measure thetime of decrypting 256-byte messages with a random 2048-bit key. We chose messages to have Montgomery represen-tations differing by multiples of 21016. Figure 5A shows theaverage observed running time for such a decryption opera-tion, which is around 4.16 ms. The messages are displayedfrom left to right in sorted order of how many Montgomeryreductions occur during the decryption. Each message wassampled roughly 8000 times and the samples were randomlysplit into 4 sample sets. As observed by Brumley et al. [15],the number of Montgomery reductions can be roughly de-termined from the running time of an unprotected RSA de-

−1.0

−0.5

0.0

0.5

1.0

Message

Dur

atio

n (n

s)

(+~

4.2

5 x

106 )

Trial 1 2 3 4A. Unprotected

−2000

−1000

0

1000

2000

Messages

Dur

atio

n(n

s)

(+~

4.1

6 x

106

)

B. Protected

−1.0

−0.5

0.0

0.5

1.0

Messages

Dur

atio

n(n

s)

(+~

4.2

5 x

106

)

Figure 5: Protecting against timing attacks on unblinded RSA

cryption. Such information can be used to derive full lengthkeys.

We then apply our defense to this decryption using a Tmax

of 9.68× 106 cycles ≈ 4.21 ms. One timer interrupt is guar-anteed to occur during such an operation, as timer inter-rupts occur at a rate of 250/s on our target machine. Under30 million measurements, we observe a multi-modal paddeddistribution with four narrow, disjoint peaks correspondingto the padding algorithm padding to Text preempt for be-tween one and four interrupts. The four peaks represent, re-spectively, 94.0%, 5.8%, 0.6%, and 0.4% of the samples. Wedid not observe that these probabilities vary across differentmessages. Hence, in figure 5B, we show the average ob-served time considering only observations from within thefirst peak. Again, samples are split into 4 random samplesets, each key is sampled around 700 thousand times. Weobserve no message-dependent signal.

6.1 Performance evaluationPerformance costs of individual components. Table 6shows the individual cost of the different components of ourdefense. Our total performance overhead is less than thetotal sum of these components as we do not perform mostof these operations in the critical path. Note that retrievingthe number of times a process was interrupted or determin-ing whether a voluntary context switch occurred during aprotected function’s execution is cheap due to the modifica-tions to the Linux kernel described in Section 5.

Macrobenchmark: protecting the TLS state machine.We applied our implementation to protect the server-sideimplementation of the TLS connection protocol in OpenSSL.The TLS protocol is implemented as a state machine inOpenSSL, and this presented a challenge for applying our so-lution which is defined in terms of protected functions. Ad-ditionally, reading and writing to a socket is interleaved withcryptographic operations in the specification of the TLS pro-tocol, which conflicts with our solution’s requirement thatno blocking I/O may be performed within a protected func-tion.

Component Cost (ns)m = 5 time randomization step, WCET 710Get interrupt counters 16Detect context switch 4Set and restore SCHED FIFO 2,650Set and restore CPU affinity 1,235Flush L1D+L2 cache 23,000Flush BTB cache 7,000

Figure 6: Performance overheads of individual components of ourdefense. WCET, worst-case execution time. Only costs listed inthe upper half of the table are incurred on each call to a protectedfunction.

We addressed both challenges by generalizing the notionof a protected function to that of a protected interval,which is an interval of execution starting with a call tofixed time begin and ending with fixed time end. We thensplit an execution of the TLS protocol into protected inter-vals on boundaries defined by transitions of the TLS statemachine and on low-level socket read and write operations.To achieve this, we first inserted calls to fixed time beginand fixed time end at the start and end of each state withinthe TLS state machine implementation. Next, we modifiedthe low-level socket read and socket write OpenSSL wrap-per functions to end the current interval, communicate withthe socket, and then start a new interval. Thus divided, allcryptographic operations performed inside the TLS imple-mentation are within a protected interval. Each interval isuniquely identifiable by the name of the current TLS stateconcatenated with an integer incremented every time a newinterval is started within the same TLS state (equivalently,the number of socket operations that occurred so far duringthe state.)

The advantage of this strategy is that, unlike any priordefenses, it protects the entire implementation of the TLSstate machine from any form of timing attack. However,such protection schemes may incur additional overheads dueto protecting parts of the protocol that may not be vulner-able to timing attacks because they do not work with secretdata.

We evaluate the performance of the fully protected TLSstate machine as well as an implementation that only pro-tects the public key signing operation in Figure 7. We ob-serve an overhead of less than 8.5% on connection latencyeven when protecting the full TLS protocol.

Microbenchmarks: encryption algorithms in multiplelanguages. We perform a set of microbenchmarks that testthe impact of our solution on individual operations such asRSA and ECDSA signing in the OpenSSL C library andin the BouncyCastle Java library. In order to apply ourdefense to BouncyCastle, we constructed JNI wrapper func-tions that call the fixed time begin and fixed time end func-tions. Since both libraries implement RSA blinding to de-fend against timing attacks, we disable RSA blinding whenapplying our defense.

The results of the microbenchmarks are shown in Figure 8.Note that the delays experienced in any real applicationswill be significantly less than these micro benchmarks asreal applications will also perform some I/O operations thatwill amortize the performance overhead.

Focusing on the BouncyCastle results, we observe a smallimprovement in performance when protecting RSA signing

Connection latency (RSA) Mean (ms) 99% TailStock OpenSSL 6.02 6.99Stock OpenSSL+ Our solution(sign only)

6.16 6.34

Stock OpenSSL+ Our solution 6.50 6.77

Connection latency (ECDSA) Mean (ms) 99% TailStock OpenSSL 5.21 5.44Stock OpenSSL+ Our solution(sign only)

5.27 5.48

Stock OpenSSL+ Our solution 5.66 5.90

Figure 7: The impact on TLS v1.2 connection latency when ap-plying our defense to the OpenSSL server-side TLS implemen-tation. We evaluate the cases where the the server uses anRSA 2048-bit or ECDSA 256-bit signing key with SHA-256 asthe digest function. Latency given in milliseconds and measuresthe end-to-end connection time. The client uses the unmodifiedOpenSSL library attempts. We evaluate our defense when onlyprotecting the signing operation and when protecting all server-side routines performed as part of the TLS connection protocolthat use cryptography. Even when the full TLS protocol is pro-tected, our approach adds an overhead of less than 8.5% to av-erage connection latency. Bold text indicates measurements thatare lower than the baseline.

RSA 2048-bit sign Mean (ms) 99% TailStock OpenSSL 1.95 2.83OpenSSL + our solution 2.03 2.04Stock BouncyCastle 12.78 13.31BouncyCastle + our solution 12.32 12.35ECDSA 256-bit sign Mean (ms) 99% TailStock OpenSSL 0.09 0.10OpenSSL + our solution 0.14 0.15Stock BouncyCastle 0.33 0.83BouncyCastle + our solution 1.38 1.40

Figure 8: Impact on performance of signing a 100 byte messageusing SHA-256 with RSA or ECDSA for the OpenSSL and Boun-cyCastle implementations. Measurements are in milliseconds. Wedisable blinding when applying our defense to the RSA signatureoperation. Bold text indicates a measurement that is lower whenusing our defense than the stock implementation.

(due to disabling of blinding), while the protected ECDSAsigning function costs approximately 1.1 ms longer than thebaseline. We believe that this increase in cost for ECDSA isjustified by the increase in security, as the BouncyCastle im-plementation does not defend against cache timing attacks.

For OpenSSL, our solution adds between 4% (for RSA)and 55% (for ECDSA) to the cost of computing a signatureon average. However, we offer significantly reduced tail la-tency for RSA signatures. This is because OpenSSL reusesblinding factors for 32 calls to the signing function, and thecost of generating these blinding factors is relatively high.

Sensitive data structures. We measured the overheadof applying our approach to protect the lookup operation ofthe C++ STL unordered_map. For this experiment, we pop-ulate the map with 1 million 64-bit integer keys and values.The average cost of performing a lookup of a key presentin the map is 0.173µs without any defense and 2.46µs withour defense applied. The cost of our defense breaks downinto two main components, first the worst-case executiontime of the randomization step, 0.710µs, is always incurredto ensure safe padding and second, the profiled worst-caseexecution time of the lookup when interrupts do not occur is

1.32µs at κ = 10−5. The same worst-case execution estimateincreases to 13.3µs when interrupt cases are not excluded,hence our scheme benefits significantly from adapting to in-terrupts during padding for this example.

7. LIMITATIONSIndirect timing variations in unprotected code. Ourapproach does not currently defend against timing variationsin the execution of code segments outside of protected func-tions. This means that if some of the sensitive accesses madeduring a protected function impacts the runtime of some un-protected code of the same user, an attacker may be able tolearn the secrets by observing the runtime variations of suchunprotected code. We are unaware of any attacks that couldexploit this kind of leakage, however a conservative approachthat guarantees security is to flush all per-cpu resources atthe end of a protected function. The cost of doing this issummarized in Table 6.

Leakage due to fault injection. If an attacker can causea protected function to crash in the middle of executionof a protected function, the attacker can potentially learnsecret information. For example, say the protected functionfirst performs some sensitive operation and then parses someinput message from the user. An attacker can learn theduration of the sensitive operation by providing a bad inputto the parser that makes it crash and time how long it takesthe victim application to crash.

Our solution, in its current form, does not protect againstsuch attacks. One way of overcoming this is to modify theOS to finish off the padding for a protected function evenafter it has crashed as part of the OS’s cleanup process.This can be implemented by calling a kernel module functionat the start of a protected function that informs the OSthat a protected function has begun and that includes thepadding parameters. Whenever the core is tainted as part ofthe protected function ending, the OS could simultaneouslyremove the record that the process is currently executing aprotected function.

8. RELATED WORK

8.1 Defenses against remote timing attacksThe remote attacks described earlier exploit the input-

dependent execution times of cryptographic operations. Thereare three main approaches to make cryptographic operationsconstant time, that is, so that the execution time is inde-pendent of the inputs: static transformation, application-specific changes, and dynamic padding. We describe each ofthem in below.

Application-specific changes. One conceptually simpleway to defend an application against timing attacks is tomodify its sensitive operations such that their timing be-havior is not key-dependent. For example, AES [27, 10] im-plementations can be modified to ensure that their executiontimes are key-independent. Note that, since the cache be-havior impacts running time, achieving secret-independenttiming usually requires rewriting the operation so that itsmemory access pattern is also independent of secrets. Suchmodifications are application specific and very hard to de-sign.

Static transformation. An alternative approach to pre-vent remote attacks is to use static transformations on theimplementation of the cryptographic operation to make itconstant time. One can use a static analyzer to find thelongest possible path through the cryptographic operationand insert padding instructions that have no side-effects(likeNOP) along other paths so that they take the same amountof time as the longest path [17, 19]. While this approach isgeneric and can be applies to any cryptographic operations,it has several drawbacks. In modern architectures like x86,the execution time of several instructions (e.g., the integerdivide instruction and multiple floating-point instructions)depend the value of the input of these instructions. Thismakes it extremely hard and time consuming to staticallyestimate the execution time of these instructions. More-over, it’s very hard to statically predict the changes in theexecution time due to internal cache collisions in the imple-mentation of the cryptographic operation.

Dynamic padding. Dynamic padding techniques add avariable amount of padding to a sensitive computation thatdepends on the observed execution time of the computationin order to mitigate the timing side-channel. Several priorworks [6, 44, 18, 28, 23] have presented ways to pad the exe-cution of a black-box computation to certain predeterminedthresholds and obtain bounded information leakage. Zhanget al. designed a new programming language that, whenused to write sensitive operations, can enforce limits on thetiming information leakage [45]. One of the major draw-back of existing dynamic padding schemes is that the esti-mation of the worst-case execution time tends to be overlypessimistic as it depends on several external parameters likeOS scheduling, cache behavior of the simultaneously run-ning programs etc. For example, Zhang et al. [44] used aworst-case execution time to be 300 seconds for protecting aWiki server. Such overly pessimistic estimates increase theamount of required padding and thus results in significantperformance overheads (90− 400% in [44]).

8.2 Defenses against local attacksLocal attackers can also perform timing attacks, hence

some of the defenses provided in the prior section also de-fend against some local attacks. However, local attackershave access to shared hardware resources that contain in-formation related to the target cryptographic operation andhave access to fine-grained timers. A common attack vectoris to probe a shared hardware resource, and then, using thefine-grained timer, measure how long the probe took to run.Most of the proposed defenses try to remove either access tofine-grained timers or, instead, isolate access to the sharedhardware resources. Still others obfuscate We describe theseapproaches below.

Removing fine-grained timers. Several prior works haveevaluated removing or modifying time measurements takenon the target machine [31, 30, 39]. Such solutions are oftenquite effective at preventing side channel attacks as the un-derlying states of most shared hardware resources can onlybe read by accurately measuring the time taken to performcertain operations (e.g., read a cache line). However, evenremoving access to wall clock time is not sufficient, as an at-tacker using multiple threads can infer time measurementsby observing the scheduling behavior of the threads. Us-ing instruction-based scheduling can eliminate such an at-

tack [35].

Preventing sharing of hardware state across processes.Many proposed defenses prevent an attacker from observingstate changes to shared hardware resources caused by a vic-tim process. We divide the proposed defenses five categoriesand describe them next.

Resource partitioning. Partitioning shared hardware re-sources can defeat local attackers, as they cannot access thesame partition of the resource as a victim. Kim et al. [25]present an efficient management scheme which locks mem-ory regions accessed by sensitive functions into reserved por-tions of the L3 cache. Their approach defeats cache-basedside channel attacks and can be more efficient than pagecoloring. Ristenpart et al. [34] suggest allocating dedicatedhardware to each virtual machine instance to prevent cross-virtual machine attacks.

Limiting concurrent access. Varadarajan et.al. [38] pro-pose using minimum runtime guarantees to ensure that aVM is not preempted too frequently. When gang schedul-ing [25] is used or hyperthreading is disabled, an attackercan only observe per-CPU resources when it has preempteda victim. Hence, reducing the frequency of preemptions re-duces the feasibility of cache-attacks on per-CPU caches.

Custom hardware. Custom hardware can be used toobfuscate and randomize the victim process’s usage of thehardware. For example, Wang et al. [40, 41] proposed newways of designing caches that ensures that no informationabout cache usage is shared across different processes. Ofcourse, such schemes require custom hardware not currentlyavailable in commodity systems.

Flushing state. Another class of defenses ensure that thestate of any per-CPU hardware resources are cleared beforetransferring them from one process to another. Duppel, byZhang et al. [47], attempts to flush per-CPU L1 and (op-tionally) L2 caches periodically, and only when a contextswitch has occurred on a core following a protected oper-ation. Their solution requires that hyperthreading is dis-abled. This is similar to our solution’s technique of flushingper-CPU resources in the OS scheduler whenever a contextswitch to a different user occurs following a protected oper-ation. They report modest overheads of less than 7% evenon workloads that are dominated by protected operations.

Application transformations. Cryptographic operationsin different programs can also be modified to exhibit ei-ther secret-independent or obfuscated hardware access pat-terns. If the access to the hardware is independent of secrets,then an attacker cannot use any of the state leaked throughshared hardware to learn anything meaningful about thecryptographic operations. Several prior works have shownhow to modify AES implementations so that they obfuscatetheir cache access patterns [8, 10, 13, 37, 32]. Another ex-ample is that recent versions of OpenSSL use a modifiedimplementation of RSA specifically with secret-independentcache accesses. These modifications are specific to particu-lar cryptographic operations and are very hard to implementcorrectly. For example, 924 lines of assembly code had to beadded to OpenSSL to implement the Montgomery modularexponentiation used in the modified RSA implementation.

Crane et al. [20] implement a system that dynamicallyapplies cache-access obfuscating transformations to an ap-plication at runtime.

9. CONCLUSIONIn this paper, we have presented a low-overhead, cross-

architecture defense that protects applications against bothlocal and remote timing attacks with minimal applicationcode changes. We have also demonstrated that our defenseis language-independent. We hope that our work will moti-vate application developers to use our techniques to defendagainst a wide variety of timing attacks without incurringany significant performance overhead.

AcknowledgmentsThis work was supported by NSF, DARPA, ONR, and aGoogle PhD Fellowship to Suman Jana. Opinions, findingsand conclusions or recommendations expressed in this mate-rial are those of the author(s) and do not necessarily reflectthe views of DARPA.

References[1] O. Aciicmez. Yet Another MicroArchitectural Attack:: Ex-

ploiting I-Cache. In CSAW, 2007.[2] O. Aciicmez, C. Koc, and J. Seifert. On the power of simple

branch prediction analysis. In ASIACCS, 2007.[3] O. Aciicmez, C. Koc, and J. Seifert. Predicting secret keys

via branch prediction. In CT-RSA, 2007.[4] O. Aciicmez and J. Seifert. Cheap hardware parallelism im-

plies cheap security. In FDTC, 2007.[5] M. Andrysco, D. Kohlbrenner, K. Mowery, R. Jhala,

S. Lerner, and H. Shacham. On subnormal floating pointand abnormal timing. In Proc. of IEEE S&P, 2015.

[6] A. Askarov, D. Zhang, and A. Myers. Predictive black-boxmitigation of timing channels. In CCS, 2010.

[7] G. Barthe, G. Betarte, J. Campo, C. Luna, and D. Pichardie.System-level non-interference for constant-time cryptogra-phy. In Proceedings of the ACM Conference on Computerand Communications Security, pages 1267–1279, 2014.

[8] D. Bernstein. Cache-timing attacks on AES, 2005.[9] D. J. Bernstein. Chacha, a variant of salsa20. http://cr.

yp.to/chacha.html.[10] J. Blomer, J. Guajardo, and V. Krummel. Provably secure

masking of aes. In Selected Areas in Cryptography, pages69–83, 2005.

[11] J. Bonneau and I. Mironov. Cache-collision timing attacksagainst AES. In Cryptographic Hardware and Embedded Sys-tems - CHES 2006, 8th International Workshop, Yokohama,Japan, October 10-13, 2006, Proceedings, pages 201–215,2006.

[12] A. Bortz and D. Boneh. Exposing private information bytiming web applications. In Proceedings of the 16th inter-national conference on World Wide Web, pages 621–628.ACM, 2007.

[13] E. Brickell, G. Graunke, M. Neve, and J. Seifert. Softwaremitigations to hedge AES against cache-based software sidechannel vulnerabilities. IACR Cryptology ePrint Archive,2006.

[14] B. Brumley and N. Tuver. Remote timing attacks are stillpractical. In ESORICS, 2011.

[15] D. Brumley and D. Boneh. Remote Timing Attacks ArePractical. In USENIX Security, 2003.

[16] F. R. K. Chung, P. Diaconis, and R. L. Graham. Randomwalks arising in random number generation. The Annals ofProbability, pages 1148–1165, 1987.

[17] J. Cleemput, B. Coppens, and B. D. Sutter. Compiler miti-gations for time attacks on modern x86 processors. TACO,8(4):23, 2012.

[18] D. Cock, Q. Ge, T. Murray, and G. Heiser. The Last Mile:An Empirical Study of Some Timing Channels on seL4. InCCS, 2014.

[19] B. Coppens, I. Verbauwhede, K. D. Bosschere, and B. D.Sutter. Practical mitigations for timing-based side-channelattacks on modern x86 processors. In S&P, 2009.

[20] S. Crane, A. Homescu, S. Brunthaler, P. Larsen, andM. Franz. Thwarting cache side-channel attacks through dy-namic software diversity. 2015.

[21] S. A. Crosby and D. S. Wallach. Denial of service via algo-rithmic complexity attacks. In Usenix Security, volume 2,2003.

[22] D. Gullasch, E. Bangerter, and S. Krenn. Cache games–bringing access-based cache attacks on aes to practice. InS&P, 2011.

[23] A. Haeberlen, B. C. Pierce, and A. Narayan. Differentialprivacy under fire. In USENIX Security Symposium, 2011.

[24] G. Irazoqui, T. Eisenbarth, and B. Sunar. Jackpot steal-ing information from large caches via huge pages. Cryptol-ogy ePrint Archive, Report 2014/970, 2014. http://eprint.iacr.org/.

[25] T. Kim, M. Peinado, and G. Mainar-Ruiz. Stealthmem:System-level protection against cache-based side channel at-tacks in the cloud. In USENIX Security symposium, pages189–204, 2012.

[26] P. Kocher. Timing attacks on implementations of Diffie-Hellman, RSA, DSS, and other systems. In CRYPTO, 1996.

[27] R. Konighofer. A fast and cache-timing resistant implemen-tation of the AES. In CT-RSA, 2008.

[28] B. Kopf and M. Durmuth. A provably secure and efficientcountermeasure against timing attacks. In CSF, 2009.

[29] A. Langley. Lucky thirteen attack on tls cbc, 2013. www.imperialviolet.org/2013/02/04/luckythirteen.html.

[30] P. Li, D. Gao, and M. Reiter. Mitigating access-driven timingchannels in clouds using StopWatch. In DSN, 2013.

[31] R. Martin, J. Demme, and S. Sethumadhavan. Timewarp:rethinking timekeeping and performance monitoring mecha-nisms to mitigate side-channel attacks. In ISCA, 2012.

[32] D. Osvik, A. Shamir, and E. Tromer. Cache attacks andcountermeasures: the case of AES. In CT-RSA, 2006.

[33] C. Percival. Cache missing for fun and profit, 2005.[34] T. Ristenpart, E. Tromer, H. Shacham, and S. Savage. Hey,

you, get off of my cloud: exploring information leakage inthird-party compute clouds. In CCS, 2009.

[35] D. Stefan, P. Buiras, E. Yang, A. Levy, D. Terei, A. Russo,and D. Mazieres. Eliminating cache-based timing attackswith instruction-based scheduling. In ESORICS, 2013.

[36] K. Suzaki, K. Iijima, T. Yagi, and C. Artho. Memory dedu-plication as a threat to the guest os. In Proceedings ofthe Fourth European Workshop on System Security, page 1.ACM, 2011.

[37] E. Tromer, D. Osvik, and A. Shamir. Efficient cache at-tacks on AES, and countermeasures. Journal of Cryptology,23(1):37–71, 2010.

[38] V. Varadarajan, T. Ristenpart, and M. Swift. Scheduler-based defenses against cross-vm side-channels. In UsenixSecurity, 2014.

[39] B. Vattikonda, S. Das, and H. Shacham. Eliminating finegrained timers in xen. In CCSW, 2011.

[40] Z. Wang and R. Lee. New cache designs for thwarting soft-ware cache-based side channel attacks. In ISCA, 2007.

[41] Z. Wang and R. Lee. A novel cache architecture with en-hanced performance and security. In MICRO, 2008.

[42] Y. Yarom and N. Benger. Recovering OpenSSL ECDSANonces Using the FLUSH+ RELOAD Cache Side-channelAttack. IACR Cryptology ePrint Archive, 2014.

[43] Y. Yarom and K. Falkner. Flush+ Reload: a High Resolu-tion, Low Noise, L3 Cache Side-Channel Attack. In USENIXSecurity, 2014.

[44] D. Zhang, A. Askarov, and A. Myers. Predictive mitigationof timing channels in interactive systems. In CCS, 2011.

[45] D. Zhang, A. Askarov, and A. Myers. Language-based con-trol and mitigation of timing channels. In PLDI, 2012.

[46] Y. Zhang, A. Juels, M. Reiter, and T. Ristenpart. Cross-vmside channels and their use to extract private keys. In CCS,2012.

[47] Y. Zhang and M. Reiter. Duppel: Retrofitting commod-ity operating systems to mitigate cache side channels in thecloud. In CCS, 2013.