Hardware Transactional Persistent MemoryEllis Giles

Rice UniversityHouston, TX 77005, USA

erg@rice.edu

Kshitij DoshiIntel Corporation

Chandler, AZ 85226, USAkshitij.a.doshi@intel.com

Peter VarmanRice University

Houston, TX 77005, USApjv@rice.edu

ABSTRACTEmerging Persistent Memory technologies (also PM, Non-VolatileDIMMs, Storage Class Memory or SCM) hold tremendous promisefor accelerating popular data-management applications like in-memory databases. However, programmers now need to deal withensuring the atomicity of transactions on Persistent Memory resi-dent data and maintaining consistency between the order in whichprocessors perform stores and that in which the updated valuesbecome durable.

�e problem is specially challenging when high-performance iso-lation mechanisms like Hardware Transactional Memory (HTM) areused for concurrency control. �is work shows how HTM transac-tions can be ordered correctly and atomically into PM by the use ofa novel so�ware protocol combined with a Persistent Memory Con-troller, without requiring changes to processor cache hardware orHTM protocols. In contrast, previous approaches require signi�cantchanges to existing processor microarchitectures. Our approach,evaluated using both micro-benchmarks and the STAMP suite com-pares well with standard (volatile) HTM transactions. It also yieldssigni�cant gains in throughput and latency in comparison withpersistent transactional locking.

1 INTRODUCTION�is paper provides a solution to the problems of adding durabilityto concurrent in-memory transactions that use Hardware Trans-actional Memory (HTM) for concurrency control while operatingon data in byte-addressable Non-Volatile Memory or PersistentMemory (PM).

Recent years have witnessed a sharp shi� towards real timedata-driven and high-throughput applications. �is has spurreda broad adoption of in-memory and massively parallelized dataprocessing [1–3] across business, scienti�c, and industrial applica-tion domains [4, 5]. Two emerging hardware developments providefurther impetus to this approach and raise the potential for transfor-mative gains in speed and scalability: (a) the arrival of inexpensive,byte-addressable, and large capacity Persistent Memory devices [6]eliminates the I/O operations and bo�lenecks common tomany datamanagement applications, and, (b) the availability of CPU-basedtransaction support (with HTM) [7, 8]) makes it straight-forward forthreads to work spontaneously in shared memory spaces withouthaving to synchronize explicitly.

To prevent corruption of state upon an untimely machine orso�ware failure, a sequence of store operations to PM in a transac-tional section of code cannot be partially re�ected into PM; nor canit be transmi�ed piecemeal from processor caches into PM withoutsimilarly risking signi�cant loss of data. Operating on PersistentMemory based data thus produces new atomicity and consistencyrequirements. So�ware approaches for ensuring atomic durable

updates share some characteristics withHTM techniques in commer-cial processors – both checkpoint state at some level of granularityand guard against communication of partial updates. However,di�erent mechanisms are at play: while stable stores to persistentmedia are usually obtained by covering updates with logging orversioning, partial updates are prevented from being propagatedbetween threads in HTM transactions by CPUs sheltering themuntil the transaction closes. Once an HTM transaction closes, itsupdates become visible en masse through the cache hierarchy andcan travel in any order to memory DIMMs. However stable storageof the updates into Persistent Memory by the transaction addition-ally requires an ability to reliably delineate its updates from thoseby other overlapping transactions, and to use that delineation torecover from an unanticipated machine restart.

Existing PM programming frameworks separate into categoriesbased on the degree of change they require in the processor ar-chitecture for such problems. Several works [9–13] operate withexisting processor capabilities and write a log (either a write-aheadlog or an undo log) durably to cover data changes arriving via thevolatile cache hierarchy; some [14–16] require signi�cant changesto existing cache hardware and protocols in the processor microar-chitecture, while others [17–19] only require external controllersnot a�ecting the processor core.

Isolation among concurrent transactions in the above worksis achieved either by the use of two-phase locking protocols orprovided within the rubric of an STM [9]. Logging based so�wareapproaches are problematic for HTM transactions (e.g., Intel TSX)which cannot bypass the caches in order to �ush the log recordssynchronously into Persistent Memory ahead of transaction clos-ings. For these, PTM [20] proposes changes to processor cacheswhile adding an on-chip scoreboard and global transaction id regis-ter to couple HTM with PM. Recent work [13, 21, 22] has a�emptedto provide inter-transactional isolation by employing processorbased (HTM) [7, 8] mechanisms. However, these solutions all re-quire changes to the existing HTM semantics and implementations–[21] and [22] propose a new instruction to perform non-abortingcacheline �ush from within an HTM, while [13] proposes allowingnon-aborting concurrent access to designated memory variableswithin an HTM.

Selective and incremental changes to the clean isolation seman-tics of HTM are not to be undertaken lightly; understanding theirimpact on global system correctness and performance typically re-quires long gestation periods before processor manufacturers willembrace them. In this paper we provide a new solution to obtainpersistence consistency in Persistent Memory while using HTMfor concurrency control. �e solution does not alter the processormicroarchitecture, but leverages a very simple, external PersistentMemory controller alongwith a persistence protocol, to supplement

arX

iv:1

806.

0110

8v1

[cs

.DC

] 2

2 M

ay 2

018

the existing HTM semantics and allowing HTM transactions to op-erate at the speed of in-memory volatile transactions. �e solutioni,while it achieves the concurrency bene�ts of HTM for PM-baseddata also applies to non-HTM transactions in a straight-forwardway.

2 OVERVIEW2.1 HTM+PM BasicsHardware Transactional Memory, or HTM, was introduced in [7]as a new, easy-to-use method for lock-free synchronization sup-ported by hardware. �e initial instructions for HTM included loadand store transactional instructions in addition to transactionalmanagement instructions. Most HTM implementations extendan underlying cache-coherency protocol to handle detection oftransactional con�icts during program execution. �e hardwaresystem performs a speculative execution on a demarcated region ofcode similar to an atomic section. Independent transactions (thosethat do not write a shared variable) proceed unrestricted throughtheir HTM sections. Transactions which access common variablesconcurrently in their HTM sections, with at least one transactionperforming a write, are serialized by the HTM. �at is, all but oneof the transactions is aborted; an aborted transaction will restartits HTM code at the beginning. Updates made within the HTMsection are hidden from other transactions and are prevented fromwriting to memory until the transaction successfully completes theHTM section. �is mechanism provides atomicity (all-or-nothing)semantics for individual transactions with respect to visibility byother threads, and serialization of con�icting, dependent transac-tions. However HTM was originally designed for volatile memorysystems (rather than supporting database style ACID transactions)and therefore any power failure leaves main memory in an un-predictable state relative to the actual values of the transactionvariables.

Persistent Memory, or PM, introduces a new method of persis-tence to the processor. PM, in the form of persistent DIMM, resideson the main-memory bus alongside DRAM. So�ware can accesspersistent memory using the usual LOAD and STORE instructionsused for DRAM. Like other memory variables, PM variables aresubject to forced and involuntary cache-evictions and encounterother deferred memory operations done by the processor.

For Intel CPUs, CLWB and CLFLUSHOPT instructions provide theability to �ush modi�ed data (at cacheline granularity) to be evictedfrom the processor cache hierarchy. �ese instructions, however,are weakly ordered with respect to other store operations in theinstruction stream. Intel has extended the semantic for SFENCEto cover such �ushed store operations so that so�ware can issueSFENCE to prevent new stores from executing until previously�ushed data has entered a power-safe domain; i.e., the data isguaranteed by hardware to reach its locations in the PMmedia. �isguarantee also applies to data that is wri�en to PMwith instructionsthat bypass the processor caches. However, when executing withinan HTM transaction, a CPU cannot exercise CLWB, CLFLUSHOPT,non-cacheable stores, and SFENCE instructions since the storesby the CPU are considered speculative until the HTM transactioncompletes successfully.

Even though HTM guarantees that transactional values are onlyvisible on transaction completion, hardware manufacturers cannotsimply utilize a non-volatile processor cache hierarchy or ba�erybacked �ushing of the cache on failures to provide transactionalatomicity. Transactions that do not complete before a so�ware orhardware restart produce partial and therefore inconsistent updatesin non-volatile memory, as there is no guarantee when a machinehalt will occur. �e halt may happen during XEND executionleaving only partial updates in cache or write bu�ers which cancorrupt in-memory data structures.

2.2 Challenges of persistent HTM transactionsConsider transactions A, B and C shown in Listings 1, 2 and 3.Assume that w, x, y, z are persistent variables initialized to zero intheir home locations in Persistent Memory (PM). �e code sectiondemarcated between the instructions XBegin and XEnd will bereferred to as an HTM transaction or simply a transaction. �eHTM mechanism ensures the atomicity of transaction execution.Within anHTM transaction, all updates aremade to private locationsin the cache, and the hardware guarantees that the updates arenot allowed to propagate to their home locations in PM. A�er theXEnd instruction completes, all of the cache lines updated in thetransaction become instantaneously visible in the cache hierarchy.Atomic Persistence: �e �rst challenge is to ensure that the trans-action’s updates that were made atomically in the cache are alsopersisted atomically in PM. Following XEnd, the transaction vari-ables are once again subject to the normal cache operations likeevictions and the use of cache write-back instructions. �ere are noguarantees regarding whether or when the transaction’s updatesactually get wri�en to PM from the cache. �is can create a problemif the machine crashes before all these updates are wri�en backto PM. On a reboot, the values of these variables in PM will beinconsistent with the pre-crash transaction values. �is leads tothe �rst requirement:

• Following crash recovery, ensure that all or none of the up-dates of an HTM transaction are stored in their PM homelocations.

A common solution is to log the transaction updates in a sepa-rate persistent storage area before allowing them to update theirPM home locations. Should a crash interrupt the updating of thehome locations, the saved log can be replayed. When transactionsexecute within an HTM there is a problem with this solution sincethe log cannot be wri�en to PM within the transaction and canbe done only a�er the XEnd. At that time the transaction updatesare also made visible in the cache hierarchy and are susceptible touncontrolled cache evictions into PM. Hence there is no guaranteethat the log has been persisted before transaction updates havepercolated into PM. We describe our solution in Section 3.Persistence Ordering: �e second problem deals with ensuringthat the execution order of dependent HTM transactions is correctlyre�ected in PM following crash recovery. As an example, considerthe dependent transactions A,B,C in Listings 1, 2 & 3. �e HTMwill serialize their execution in some order: say A, B and C . �evalues of the transaction variables following the execution of A aregiven by the vectorV1 = [w, x, y, z] = [1, 1, 0, 0]; a�er the executionof B the vector becomes V2 = [2, 1, 2, 0] and �nally following C it is

2

Listing 1: AA ( ) {XBegin ;w = 1 ;x = w;

XEnd ;}

Listing 2: BB ( ) {XBegin ;w = w+1 ;y = w;

XEnd ;}

Listing 3: CC ( ) {XBegin ;w = w+1 ;z = w;

XEnd ;}

V3 = [3, 1, 2, 3]. Under normal operation the write backs of variablesto PM from di�erent transactions may become arbitrarily inter-leaved. For instance suppose that x is evicted immediately a�er Acompletes,w a�er B completes, and z a�erC completes. �e persis-tent memory state is then [2, 1, 0, 3]; should the machine crash, thePM will contain this meaningless combination of values on reboot.A consistent state would be either the initial vector [0, 0, 0, 0] orone of V1,V2 or V3. �is leads to the second requirement:

• Following crash recovery, ensure that the persistent state ofany sequence of dependent transactions is consistent withtheir execution order.

If individual transactions satisfy atomic persistence, then it issu�cient to ensure that PM is updated in transaction executionorder. With so�ware concurrency control (using an STM or two-phase transaction locking), it is straightforward to correctly orderthe updates simply by saving a transaction’s log before it commitsand releases its locks. In case of a crash, the saved logs are sim-ply replayed in the order they were saved, thereby reconstructingpersistent state to a correctly-ordered pre�x of the executed trans-actions.

When HTM is used for concurrency control the logs can onlybe wri�en to PM a�er the transaction XEnd. At that time otherdependent transactions can execute and race with the completedtransaction, perhaps writing out their logs before the �rst. Solutionslike using an atomic counter within transactions to order themcorrectly are not practical since the shared counter will result inHTM-induced aborts and serialization of all transactions. Somepapers have advocated that processor manufacturers alter HTMsemantics and implementation to allow selective writes to PM fromwithin an HTM [13, 21, 22]. We describe our solution without theneed for such intrusive processor changes in Section 3.

Strict and Relaxed Durability: In traditional ACID databases, acommi�ed transaction is guaranteed to be durable since its log ismade persistent before it commits. We refer to this property asstrict durability. In HTM transactions the log is wri�en to PM a�erthe XEnd instruction some time before the transaction commits. Anatural question is to characterize the time it is safe for a transactionrequiring strict durability to commit.

It is generally not safe to commit a transaction Y at the time itcompletes persisting its log for the same reason that it is di�cultto ensure persistence ordering. Due to races in the code outsidethe HTM it is possible that an earlier transaction X (on which Ydepends) to have completed but not yet persisted its log. Whenrecovering from a crash that occurs at this time, the log of Y shouldnot be replayed since earlier transactionX cannot be replayed. �isleads to the third requirement:

• Following crash recovery, strict durability requires that everycommi�ed transaction is persistent in PM.

We de�ne a new property known as relaxed durability that allowsan individual transaction to opt for an early commit immediatelya�er it persists its log. Requiring relaxed or strict durability isa local choice made individually by a transaction based on theapplication requirements. Transactions choosing relaxed durabilityface a window of vulnerability a�er they commit, during which acrash may result in their transaction updates not being re�ectedin PM a�er recovery. �e gain is potentially reduced transactionlatency. However, irrespective of the choice of the durability modelby individual transactions, the underlying persistent memory willalways recover to a consistent state, re�ecting an ordered atomicpre�x of every sequence of dependent transactions.

3 OUR APPROACHOur approach achieves durability of HTM transactions to PersistentMemory by a cooperative protocol involving three components: aback-end Persistent Memory Controller , transaction execution andlogging, and a failure recovery procedure. �e Persistent MemoryController intercepts dirty cache lines evicted from the last-levelprocessor cache (LLC) on their way to persistent memory. An inter-cepted cache line is held in a FIFO queue within the controller untilit is safe to write it out to PM. All memory variables are subject tothe normal cache operations and are fetched and evicted accord-ing to normal cache protocols. �e only change we introduce isinterposing the external controller between the LLC and memory.Note that the controller does not require changing any of the in-ternal processor behavior. �e controller simply delays the evictedcache lines on their way to memory till it can guarantee safety. Itis pre-programmed with the address range of a region of persistentmemory that is reserved for holding transaction logs. Addresses inthe log region pass through the controller without intervention.

HTM+PM transactions execute independently. Within an outer-envelope that achieves consistency of updates between the volatilecache hierarchy and the durable state in PM, these transactionsuse an unmodi�ed HTM to serialize the computational portions ofcon�icting transactions. A transaction (1) noti�es the controllerwhen it opens and closes, (2) saves start and end timestamps inPM to enable consistent recovery a�er a failure, (3) performs itsHTM operation, and (4) persists a log of its updates in PM beforeclosing. If a transaction requires strict durability it informs thecontroller during its closing step, and then waits for the go-aheadfrom the controller before commi�ing. If it only needs relaxeddurability it can commit immediately a�er its close. �e recoveryroutine is invoked a�er a system crash to restore the PM variablesto a valid state i.e. a state that is consistent with the actual executionorder of every sequence of dependent transactions. �e recoveryprocedure uses the saved logs to recover the values of the updatedvariables, and the saved start and end timestamps to determinewhich logs are eligible for replay and their replay order.

3.1 Transaction LifecycleA transaction can be viewed as progressing through �ve states:OPEN,COMPUTE, LOG,CLOSE andCOMMIT as shown in Listing 4.When a transaction begins, it calls the library function OpenWrapC

3

(see Algorithm 1 in Section 4.2). �is function invokes the Persis-tent Memory Controller with a small unique integer (wrapId) thatidenti�es the transaction. �e controller adds wrapId to a set ofcurrently open transactions (referred to as COT) that it maintains(see Algorithm 2 in Section 4.3). �e transaction then allocates andinitializes space in PM for its log and updates the startTime recordof the log. �e startTime is obtained by reading a system wideplatform timer using the RDTSCP instruction (see Section 4.1). Inaddition to startTime, a log includes a second timestamp persistTimethat will be set just prior to completing the HTM transaction. �ewriteSet is a sequence of (address,value) pairs in the log that will be�lled with the locations and values of the updates made by the HTMtransaction. �e log with its recorded startTime is then persistedusing cache line write-back instructions (clwb) and sfence.

�e transaction then enters the COMPUTE state by executingXBegin and entering the HTM code section. Within the HTM sec-tion, the transaction updates writeSet with the persistent variablesthat it writes. Note that the records in writeSet will be held in cacheduring the COMPUTE state since it occurs within an HTM and can-not be propagated to PM until XEnd completes. Immediately beforeXEnd the transaction obtains a second timestamp persistTime thatwill be used to order the transactions correctly. �is timestamp isalso obtained using the same RDTSCP instruction.

A�er executing XEnd, a transaction next enters the LOG state. It�ushes its log records from cache hierarchy into PM using cacheline write-back instructions (CLWB or CLFLUSHOPT), following thelast of them with an SFENCE. �is ensures that all the log recordshave been persisted. In addition to startTime , a log includes thepersistTime time stamp that was set just prior to completing thetransaction. �ewriteSet records in the log hold (address; value)pairs representing the locations and the values updated by thetransaction. A�er the SFENCE following the log record �ushes, thetransaction enters the CLOSE state.

In the CLOSE state the transaction signals the Persistent MemoryController that its log has been persisted in PM. �e controllerremoves the transaction from its set of currently open transactionsCOT. It also re�ects the closing in the state of evicted cache linesin its FIFO bu�er as described below in Section 3.2. A transactionrequiring strict durability informs the controller at this time; thecontroller will signal the transaction in due course when it is safeto commit i.e. its updates are guaranteed to be durable.

�e transaction is then complete and enters the COMMIT state. Ifit requires strict durability it waits till it is signaled by the controller.Otherwise, it immediately commits and leaves the system.

3.2 Persistent Memory Controller�e Persistent Memory Controller is shown in Figure 1. Whilesuper�cially similar to an earlier design [18, 19, 23], this controllerincludes enhancements to handle the subtleties of using HTM ratherthan locking for concurrency control, and makes signi�cant simpli-�cations by shi�ing some of the responsibility for the maintenanceof PM state to the recovery protocol.

A crucial function of the controller is to prevent any transac-tion’s updates from reaching PM until it is safe for it to do so -without requiring detailed book-keeping about which transactions,

Listing 4: Transaction StructureTr an s a c t i o n Begin−−−−−−−−−− S t a t e : OPEN −−−−−−−−−−−−−−−−1 No t i f yPMCon t ro l l e r ( open ) ;2 A l l o c a t e a Log s t r u c t u r e in PM;3 nowTime = ReadP la t fo rmCounte r ( ) ;4 Log . s t a r t T ime = nowTime ;5 Log . w r i t e S e t = {} ;6 P e r s i s t Log in PM;−−−−−−−−−− S t a t e : COMPUTE −−−−−−−−−−−−−XBegin/ / T r a n s a c t i o n Body o f HTM/ / A l l r e a d s and w r i t e s t o t r a n s a c t i o n/ / v a r i a b l e s a r e p e r f o rmed and a l s o/ / appended t o Log . w r i t e S e t .7 endTime = ReadP la t fo rmCounte r ( ) ;XEnd

−−−−−−−−−− S t a t e : LOG −−−−−−−−−−−−−−−−−8 Log . p e r s i s t T ime = endTime ;9 P e r s i s t Log in PM;

−−−−−−−−−− S t a t e : CLOSE −−−−−−−−−−−−−−−10 No t i f yPMCon t ro l l e r ( c l o s e ) ;

−−−−−−−−−− S t a t e : COMMIT −−−−−−−−−−−−−−11 I f ( s t r i c t d u r a b i l i t y r e qu e s t e d )

Wait for n o t i f i c a t i o n by c o n t r o l l e r ;12 T r an s a c t i on End

Figure 1: PM Controller with Volatile Delay Bu�er, CurrentOpen Transactions Set, and Dependency Wait�eue

currently active or previously completed, generated a speci�c up-date. It does this by enforcing two requirements before allowingan evicted dirty cache line (representing a transaction’s update)from proceeding to PM: (i) ensuring that the log of the transactionhas been made persistent, and (ii) guaranteeing that the saved logwill be replayed during recovery. �e �rst condition is needed (butnot su�cient) for atomic persistence by guarding against a failurethat occurs a�er only a subset of a transaction’s updates have been

4

persisted in PM.�e second requirement arises because transactionlogs are persisted outside the HTM, and there is no relation betweenthe order in which the transactions execute their HTM and the orderin which their logs are persisted. To maintain correct persistenceordering, the recovery routine may not be able to replay a log. Weillustrate the issue in the example below.

Example: Consider two dependent transactions A and B, A: {w =3; x=1;} and B: {y = w+1; z = 1;}. Assume that HTM transaction Aexecutes before B, but that B persists its logs and closes before A.Suppose there is a crash just a�er B closes. �e recovery routinewill not replay either A or B, since the log of the earlier transactionA is not available. �is behavior is correct.Now consider the situation where y (value 4) is evicted from thecache and then wri�en to PM a�er B persists its log. Once again,following a crash at this time, neither log will be replayed. However,atomic persistence is now violated since B’s updates are partiallyre�ected in PM. Note that this violation occurred even though thewrite back of y to PM happened a�er B’s log was persisted. �ePersistent Memory Controller protocol prevents a write back toPM unless it can also guarantee that the log of the transactioncreating the update will be played back on recovery (see LemmasC2 and C4 in Section 3.4).

�e second function of the controller is to track when it is safefor a transaction requiring strict durability to commit. It is notsu�cient to commit when a transaction’s logs are persistent onPM since, as seen in the example above, the recovery routine maynot replay the log if that would violate persistency ordering. �econtroller protocol e�ectively delays a strict durability transac-tion τ from commi�ing until the earliest open transaction has astartTime greater than the persistTime of τ . �is is because therecovery protocol will replay a log (see Section 3.3) if and only ifall transactions with startTime less than its persistTime have closed.

Implementation Overview:�e controller tracks transactions by maintaining a COT (cur-

rently open transactions) set S . When a transaction opens, itsidenti�er is added to COT and when the transaction closes it isremoved. �e write into PM of a cache line C evicted into thePersistent Memory Controller is deferred by placing it at the tailof a FIFO queue maintained by the controller. �e cache line isalso assigned a tag called its dependency set, initialized with S thevalue of COT, at the instant that C entered the Persistent MemoryController.

�e controller holds the evicted instance of C in a FIFO untilall transactions that are in its dependency set (i.e. S) have closed.When a transaction closes it is removed from both the COT and fromthe dependency sets of all the FIFO entries. When the dependencyset of a cache line in the FIFO becomes empty, it is eligible to be�ushed to PM. One can see that the dependency sets will becomeempty in the order in which the cache lines were evicted, since atransaction still in the dependency set of C when a new evictionenters the FIFO will also be in the dependency set of the new entry.�e simple protocol guarantees that all transactions that openedbefore cache line C was evicted into the controller (which mustalso include the transaction that last wroteC) must have closed andpersisted their logs when C becomes eligible to be wri�en to PM.�is also implies that all transactions with startTime less than the

persistTime of the transaction that last wrote C would have closed,satisfying the condition for log replay. Hence the cache line can besafely wri�en to PM without violating atomic persistence.

Note that the evicted cache lines intercepted by the controller donot hold any identifying transaction information and can occur atarbitrary times a�er the transaction leaves the COMPUTE state. �ecache line could hold the update of a currently open transactionor could be from a transaction that has completed or even commit-ted and le� the system. To guarantee safety, the controller mustperforce assume that the �rst situation holds. �e details of thecontroller implementation will be presented in Section 4.3.

3.3 RecoveryEach completed transaction saves its log in PM. �e log holdsrecords startTime and persistTime obtained by reading the platformtimer using RDTSCP instruction. We refer to these as the start andend timestamps of the transaction. �e start timestamp is persistedbefore a transaction enters its HTM. �is allows the recovery routineto identify a transaction that started but had not completed at thetime of a failure. Note even though such a transaction has notcompleted, it could still have �nished its HTM and fed values toa later dependent transaction which has completed and persistedits log. �e end timestamp and the write set of the transaction arepersisted a�er the transaction completes its HTM section, followedby an end of log marker. �ere can be an arbitrary delay betweenthe end of the HTM and the time that its log is �ushed from cachesinto PM and persisted.

�e recovery procedure is invoked on reboot following amachinefailure. �e routine will restore PM values to a consistent state thatsatis�es persistence ordering by copying values from the writeSetsof the logs of qualifying transactions to the speci�ed addresses.A transaction τ quali�es for log replay if and only if all earliertransactions on which it depends (both directly and transitively)are also replayed.

Implementation Overview:�e recovery procedure �rst identi�es the set of incomplete

transactions I, which have started (as indicated by the presence ofa startTime record in their log) but have not completed (indicated bythe lack of a valid end-of-record marker). �e remaining completetransactions (set C) are potential candidates for replay. Denote thesmallest start timestamp of transactions inI byTmin . A transactionin C is valid (quali�es for replay) if its end timestamp (persistTime) isno more thanTmin . All valid transactions are replayed in increasingorder of their end timestamps persistTime.

3.4 Protocol PropertiesWe now summarize the invariants maintained by our protocol.

De�nition: �e precedence set of a transaction T , denoted byprec(T ), is the set of all dependent transactions that executed theirHTM before T . Since the HTM properly orders any two dependenttransactions the set is well de�ned.

Lemma C1: Consider a transaction X with a precedence setprec(X ). For all transactions Y in prec(X ), startTime(Y ) <persistTime(X ).

5

Proof Sketch: Let Y be a transaction in prec(X). First let us con-sider direct precedence, in which a cacheline C modi�ed in Y con-trols the ordering of X with respect to Y . �at is, X either readsor writes the cacheline C . Since Y is in prec(X ), the earliest timethat X accesses C must be no earlier than the latest time that YaccessesC , and thus persistTime(Y ) < persistTime(X ). Next considera chain of direct precedences, Y → Z →W → · · ·X , which putsY in prec(X ); and by transitivity, persistTime(Y ) < persistTime(X ).Since startTime(Y ) < persistTime(Y ) the lemma follows.

Lemma C2: Consider transactions X and Y with startTime(Y ) <persistTime(X ). If a cache line C that is updated by X is wri�ento PM by the controller at time t , then Y must have closed andpersisted its log before t .Proof Sketch: Suppose C was evicted to the controller at timet ′ ≤ t . Now t ′ must be later than the time X completed HTMexecution and set persistTime(X ); by assumption this is a�er Y setits startTime at which time Y must have been registered as an opentransaction by the controller. Now, either Y has closed before t ′ oris still open at that time. In the la�er case, Y will be added to thedependence set of C at t ′. Since C can only be wri�en to PM a�erits dependence set is empty, it follows that Y must have closed andremoved itself from the dependence set of C .

Lemma C3 Any transaction X that writes an update to PM andcloses at time t will be replayed by the recovery routine if there isa crash any time a�er t .Proof �e recovery routine will replay a transaction X if the onlyincomplete transactions (started but not closed) at the time of thecrash started a�er X completed; that is, there is no incompletetransaction Y that has a startTime(Y ) ≤ persistTime(X ). By LemmaC2 such an incomplete transaction cannot exist.

Lemma C4:Consider a transaction X with a precedence set prec(X ). �en

by the time X closes and persists its logs, one of the following musthold: (i) Some update of X has been wri�en back to PM and alltransactions Y in prec(X ) have persisted their logs; (ii) No updateof X has been wri�en to PM and all transactions Y in prec(X ) havepersisted their logs; (iii) No update ofX has been wri�en to PM andsome transactions Y in prec(X ) have not yet persisted their logs.Proof Sketch: From Lemmas C1 and C2, it is not possible to havea transaction in prec(X ) that is still open if an update from X hasbeen wri�en to PM.

4 ALGORITHM AND IMPLEMENTATION�e implementation consists of a user so�ware library backed by asimple Persistent Memory Controller. �e library is used primarilyto coordinate closures of concurrent transactions with the �ow ofany data evicted from processor caches into PM home locations dur-ing those transactions. �e Persistent Memory Controller uses thedependency set concept from [18, 19, 23] to temporarily park anyprocessor cache eviction in a searchable Volatile Delay Bu�er (VDB)so that its e�ective time to reach PM is no earlier than the time thatthe last possible transaction with which the eviction could haveoverlapped has become recoverable. �e Persistent Memory Con-troller in this work improves upon the backend controller of [18] by

Figure 2: Flow of a transaction with implementation usingHTM, cached write sets, timing, and logging.

dispensing with synchronous log replays and victim cache manage-ment. �e library also covers any writes to PM variables by volatilewrite-aside log entries made within the speculative scope of anHTM transaction; and then streaming the transactional log recordinto a non-volatile PM area outside the HTM transaction. �ese logstreaming writes into PM bypass the VDB. A so�ware mechanismmay periodically check the remaining capacity of the PM log areaand initiate a log cleanup if needed; for such occasional cleanups,new transactions are delayed, and, a�er all open transactions haveclosed, processor caches are �ushed (with a closing sfence), the logsare removed.

We refer to our implementation asWrAP, for Write-Aside Per-sistence, and individual transactions as wraps. We �rst describe thetimestamp mechanism, then the user so�ware library, and �nallydescribe the Persistent Memory Controller implementation details.

4.1 System Time StampWe use the recent Intel instruction RDTSCP, or Read Time StampCounter and Processor ID, to obtain the timestamps in listing 4.�e RDTSCP instruction provides access to a global monotonicallyincreasing processor clock across processor sockets [24], whileserializing itself behind the instructions that precede it in programorder. To prevent the reordering of an XEnd before the RDTSCPinstruction, we save the resulting time stamp into a volatile memoryaddress. Since all stores preceding an XEnd become visible a�erXEnd, and the store of the persist timestamp is the last store beforeXEnd, that store gets neither re-ordered before other stores norre-ordered a�er the end of the HTM transaction. We note thatRDTSCP has also been used to order HTM transactions in noveltransaction pro�ling [25] and memory version checkpointing [26]tools.

4.2 So�ware LibraryFor HTM we employ Intel’s implementation of Restricted Trans-actional Memory or RTM, which includes the instructions XBeginand XEnd. Aborting HTM transactions retry with exponentialback-o� a few times, and then are performed under a so�ware lock.Our HTMBegin routine checks the status of the so�ware lock bothbefore and a�er an XBegin, to produce the correct indication ofcon�icts with the non-speculative paths; acquiring the so�warelock non-speculatively a�er having backed o�. HTMBegin andHTMEnd library routines perform the acquire and release of theso�ware lock for the fallback case within themselves. �e remain-ing so�ware library procedures are shown in Algorithm 1. Various

6

Algorithm 1: Concurrent WrAP User So�ware Library

User Level So�ware WrAP Library:

OpenWrapC ()// ———— State: OPEN ————–wrapId = threadId;Notify Controller Open WrapwrapId ;startTime = RDTSCP();Log[wrapId].startTime = startTime;Log[wrapId].writeSet = {};sfence();CLFLUSH Log[wrapId];sfence();// ———— State: COMPUTE ———HTMBegin(); // XBegin

wrapStore (addrVar, Value)Add {addrVar ,Value} to Log[wrapId].writeSet;Normal store of Value to addrVar;

CloseWrapC (strictDurability)Log[wrapId].persistTime = RDTSCP();HTMEnd(); // XEnd// ———— State: LOG ——————CLFLUSH Log[wrapId].persistTime;for num cachelines in Log[wrapId].writeSet

CLFLUSH cacheline;if (strictDurability)

durabilityAddr = 0; // Reset Flag;tAddr = durabilityAddr ;

else tAddr = 0;sfence();// ———— State: CLOSE —————Notify Controller Closed (wrapId , tAddr );// ———— State: COMMIT ————if (strictDurability)

// Wait for durable noti�cation from controllerMonitor durabilityAddr ;

events that arise in the course of a transaction are shown in Figure 2,which depicts the HTM concurrency section with vertical lines andthe logging section with slanted lines.

Not shown in Figure 2, is a per-thread durability address locationthat we call durabilityAddr in Algorithm 1. A so�ware thread mayuse it to setup a Monitor-Mwait coordination to be signaled viamemory by the Persistent Memory Controller (as described shortly)when a transacting thread wants to wait until all updates fromany non-con�icting transactions that may have raced with it arecon�rmed to be recoverable.

�is provision allows for implementing the strict durability forany transaction because the logs of all other transactions that couldpossibly precede it in persistence order are in PM –which guaran-tees the replayability of its log. By contrast, many other transactionsthat only need the consistency guarantee (correct log ordering) maycontinue without waiting (or defer waiting to a di�erent point in

the course of a higher level multi-transaction operation). �e num-ber of active HTM transactions at any given time is bounded by thenumber of CPUs, therefore, we use thread identi�ers as wrapIds. InOpenWrapC we notify the Persistent Memory Controller that awrap has started. We then read the start time with RDTSCP and saveit and an empty write set into its log persistently. �e transactionis then started with the HTMBegin routine.

During the course of a transactional computation, the storesare performed using the wrapStore function. �e stores are justthe ordinary (speculatively performed) store instructions, but areaccompanied by (speculative) recording of the updates into the loglocations, each capturing the address, value pair for each update, tobe commi�ed into PM later during the logging phase (a�er XEnd).

In CloseWrapC we obtain and record the ending timestamp foran HTM transaction into the persistTime variable in its log. Itsconcurrency section is then terminated with the HTMEnd routine.At this point, the cached write set for the log and ending persistenttimestamp are instantly visible in the cache. Next, we �ush transac-tional values and the persist timestamp to the log area followed by apersistent memory fence. �e transaction closure is then noti�ed tothe Persistent Memory Controller with the wrapId, and along withit, the durabilityAddr, if the thread has requested strict durability(by passing a �ag to CloseWrapC) – for which, we use the e�-cient Monitor-Mwait construct to receive memory based signalingfrom the Persistent Memory Controller. If strict durability is notrequested, then CloseWrapC can return immediately and let thethread proceed immediately with relaxed durability. In many casesa thread performing a series of transactions may choose relaxeddurability over all but the last and then request strict durabilityover the entire set by waiting for only the last one to be strictlydurable.

4.3 PM Controller Implementation�e Persistent Memory Controller provides for two needs: (1) hold-ing back modi�ed PM cachelines that fall into it at any time Tfrom the processor caches, from �owing into PM until at leasta time when all successful transactions that were active at timeT are recoverable, and (2) tracking the ordering of dependenciesamong transactions so that only those that need strict durabilityguarantees need to be delayed pending the completion of the logphases of those with which they overlap. It implements a VDB(volatile data bu�er) as means for the transient storage for the �rstneed, implements a durability wait queue (DWQ) for the secondneed, and implements a dependency set (DS) tracking logic acrossthe open/close noti�cations to control the VDB and the DWQ, asdescribed next.

Volatile Delay Bu�er (VDB): �e VDB is comprised of a FIFOqueue and hash table that points to entries in the FIFO queue. Eachentry in the FIFO queue contains a tuple of PM address, data, anddependency set. On a PM write, resulting from a cache evictionor streaming store, to a memory address not in the log area orpass-through area, the PM address and data are added to the FIFOqueue and tagged with a dependency set initialized to the COT.Additionally, the PM address is inserted into the hash table with apointer to the FIFO queue entry. If the address already exists in thehash table, then it is updated to point to the new queue entry. On

7

Algorithm 2: Hardware WrAP Implementation

Persistent Memory Controller :

Open Wrap Notification (wrapId)AddwrapId to Current Open Transactions COT;

Memory Write (memoryAddr, data)// Triggered from cache evict or stream store.if ((memoryAddr not in Pass-�rough Log Area) and(Current Open Transactions COT != {}))

Add (memoryAddr , data, COT ) to VDB;else

// Normal memory writeMemory[memoryAddr ] = data;

Memory Read (memoryAddr)if (memoryAddr in Volatile Delay Bu�er)

return latest cacheline data from VDB;return Memory[memoryAddr ];

Close WrAP Notification (wrapId, durabilityAddr)RemovewrapId from Current Open Transactions COT;if (durabilityAddr )

durabilityDS = COT;// Add pair to Durability Wait �eue DWQ;Add (durabilityDS , durabilityAddr ) to DWQ;

RemovewrapId from Volatile Delay Bu�er elementsif earliest VDB elements have empty DS

// Write back entries to memory in FIFO orderMemory[memoryAddr ] = data;

RemovewrapId from Durability Wait �eue elementsif earliest DWQ elements have empty DS

// Notify waiting thread of durabilityMemory[durabilityAddr ] = 1;

a memory read, the hash table is �rst consulted. If an entry is inthe hash table, then the pointer is to the latest memory value forthe address, and the data is retrieved from the queue. On a hashtable miss, PM is read and data is returned. As wraps close, thedependency set in each entry in the queue is updated to removethe dependency on the wrap.

Dependency sets become empty in FIFO order, and as they be-come empty, we perform three actions. First, we write back thedata to the PM address. Next, we consult the hash table. If the hashtable entry points to the current FIFO queue entry, we remove theentry in the hash table, since we know there are no later entriesfor the same memory address in the queue. Finally, we remove theentry from the FIFO queue.

On inserting an entry into the back of the queue, we can alsoconsult the head of the FIFO queue to check to see if the dependencyset is empty. If the head has an empty dependency set, we canperform the same actions, allowing for O(1) VDB management.

Dependency Wait�eue (DWQ): Strict durability is handledby the Persistent Memory Controller using the Dependency Wait�eue or DWQ, which is used to track transactions waiting onothers to complete and notify the transaction that it is safe to

proceed. �e DWQ is a FIFO queue similar to the VDB with entriescontaining pairs of the dependency set and a durability address.

When a thread noti�es the Persistent Memory Controller that itis closing a transaction (see steps below), it can request strict dura-bility by passing a durability address . Dependencies on closingwraps are also removed from the dependency set for each entry inthe DWQ. When the dependency set becomes empty, the controllerwrites to the durability address and removes the entry from thequeue. �reads waiting on a write to the address can then proceed.

Opening and Closing WrAPs: As outlined in Algorithm 2,the controller supports two interfaces from so�ware, namely thosefor Open Wrap and Close Wrap noti�cations exercised from theuser library as shown in Algorithm 1. (Implementations of thesenoti�cation can vary: for example, one possible mechanism mayconsist of so�ware writing to a designated set of control addressesfor these noti�cations). It also implements hardware operationsagainst the VDB from the processor caches: Memory Write, forhandling modi�ed cachelines evicted from the processor caches ornon-temporal stores from CPUs and Memory Read, for handlingreads from PM from the processor caches.

�e Open Wrap noti�cation simply adds the passed (wrapId)to a bit vector of open transactions. We call this bit vector ofopen transactions the Current Open Transactions COT. When thecontroller receives a Memory Write (i.e., a processor cache evictionor a non-temporal/streaming/uncached write) it checks the COT: ifthe COT is empty, writes can �ow into the PM. Writes that targetthe log range in PM can also �ow into PM irrespective of the COT.For the non-log writes, if the COT is nonempty cache line is taggedwith the COT and placed into the VDB.

�e Close Wrap controller noti�cation receives the wrapId anddurability address, durabilityAddr . �e controller removes thewrapId from the Current Open Transactions COT bit mask. If thetransaction requires strict durability, we save the durabilityDSand COT as a pair in the DWQ. �e controller then removes thewrapId from all entries in the VDB and DWQ. �is is performedby simply draining the bit on the dependency set bit mask for theentire FIFO VDB. If the earliest entries in the queue result in anempty dependency set, the cache line data is wri�en back in FIFOorder. Similarly, the controller removes thewrapId from all entriesin the Durability Wait �eue DWQ.

So�ware Based Strict Durability Alternative: As an alter-native for implementing strict durability in the controller, strictdurability may be implemented entirely in the so�ware library; wemodify the so�ware algorithm as follows. On a transaction start,threads save the start time and an open �ag in a dedicated cacheline for the thread. On transaction close, to ensure strict durability,it saves its end time in the same cache line with the start time andclears the open �ag. It then waits until all prior open transactionshave closed. It scans the set of all thread cache lines and comparesany open transaction end times and start times to its end time. �ethread may only continue, with ensured durability, once all otherthreads are either not in an open transaction or have a start orpersist time greater than its persist time.

8

(a) Example Transactions T1-T4

Time Only Start TimestampHas Been Persisted

Order of PersistTimestamps

t1 T1t2 T1, T2t3 T1, T2, T3t4 T1, T3 T2t5 T1, T3, T4 T2t6 T1, T4 T2, T3t7 T1, T4 T2, T3t8 T1, T4 T2, T3t9 T4 T2 , T3, T1t10 T2 , T3 , T4, T1t11 T2 , T3 , T4, T1t12 T2 , T3 , T4 , T1

(b) Transaction Recovery Order on Failure

Time t1 - t3 t4 t5 t6 t7 t8 t9 t10 t11 t12Event T1,T2,T3 Start Evict X T4 Starts Evict Y T3 Ends T2 Ends Evict Z Evict X T1 Ends T4 EndsCOT {1,1,1,0} {1,1,1,0} {1,1,1,1} {1,1,1,1} {1,1,0,1} {1,0,0,1} {1,0,0,1} {1,0,0,1} {0,0,0,1} {0,0,0,0}↓

VolatileDelayBu�er↓

-

↓X: {1,1,1,0} X: {1,1,1,0}

↓Y: {1,1,1,1}X: {1,1,1,0}

Y: {1,1,0,1}X: (1,1,0,0}

Y: {1,0,0,1}X: {1,0,0,0}

↓Z: {1,0,0,1}Y: {1,0,0,1}X: {1,0,0,0}

↓X: {1,0,0,1}Z: {1,0,0,1}Y: {1,0,0,1}X: {1,0,0,0}

X: {0,0,0,1}Z: {0,0,0,1}Y: {0,0,0,1}X: {0,0,0,0}↓

X: {0,0,0,0}Z: {0,0,0,0}Y: {0,0,0,0}↓

(c) Contents of the Persistent Memory Controller Current Open Transactions Dependency Set and FIFO�eue

Figure 3: Example Transaction Sequence with Contents of Persistent Memory Controller and Recovery Algorithm

4.4 ExampleFigure 3a shows an example set of four transactions, T1-T4, happen-ing concurrently. In this example, we show T1-T4 split into states,speci�cally, concurrency (COMPUTE), shown in vertical lines, andLOG, depicted with slanted lines. At certain time steps we showthe contents of the Persistent Memory Controller’s Volatile De-lay Bu�er, or VDB which is a FIFO �eue, and the Current OpenTransactions or COT in Figure 3c. �e recovery algorithm is shownin Figure 3b with the contents of the log. Either the start times-tamp only or the persist timestamp order is shown; where boldand underline indicate the log is wri�en, and a circled transactionindicates that it is recoverable.

First, at time t1, T1 opens, notifying the controller, and recordsits start timestamp safely in its log. �e controller adds T1 to thebitmap COT of open transactions. At times t2 and t3, transactions T2and T3 also open, notify the controller, and safely read and persisttheir start timestamps. At this point in time the Persistent MemoryController has a COT of {1,1,1,0} and only start timestamps havebeen recorded in the log. T2 then completes its concurrency sectionand persists its persist timestamp at time t4 and begins writing itslog. In Figure 3c, we also show a random cache eviction of a cacheline X that is tagged with the COT of {1,1,1,0}.

Transaction T4 starts at time t5 persisting its start timestamp andis added to the set of open transactions on the Persistent MemoryController, now {1,1,1,1}. At time t6 we illustrate several events. T3completes its concurrency section and persists its persist timestamp.

At this time, as shown in 3b, T3 is now ordered a�er T2 for recovery,however neither have completed persisting their logs. Also, weshow a random cache eviction of cache line Y , and it is placed atthe back of the VDB on the Persistent Memory Controller as shownin 3c.

At time t7, transaction T3 has completed writing its logs and ismarked completed and is removed from the dependency set in thecontroller and cache line dependencies for X and Y . However, asshown in 3b, T3 is not recoverable at this point since it is not �rstin the persist timestamp order. T3 is behind T2 and T3 also has asmaller persist timestamp than the start timestamp of T1, whichhasn’t wri�en its persist timestamp yet. When T2 completes writingof logs at time t8, it is removed from the current and dependencysets in the queue of the Persistent Memory Controller as shownin 3c. Note that cache line X is still not able to be wri�en backto Persistent Memory as it is still tagged as being dependent onT1, and Y on both T1 and T4. At this time, T2 and T3 are also notrecoverable as shown in 3b as T2 has a persist timestamp that isgreater than the start timestamp of T1, as it would be unknown bya recovery process if T1 had simply had a delay in persisting its logand T2 had transactional values dependent on T1.

We illustrate two events at time t9. T1 �nally completes itsconcurrency section and writes its persist timestamp at t9. Since thepersist timestamp of T1 is safely persisted and known at recoverytime, transaction T2 is now fully recoverable as shown circled in 3b.However, at this time, T3 is not fully recoverable since it is waiting

9

on T4, which started before T3 completed its concurrency sectionand T4 hasn’t yet wri�en the persist timestamp. Also at t9, in 3cwe illustrate the eviction of cache line Z which is tagged with theset of open transactions COT {1,0,0,1}.

At time t10, T4 writes its persist timestamp and its order is nowknown to a recovery routine to be behind T3, which is now fullyrecoverable as shown with the circle in 3b. Note that T4 has apersist time before T1. In 3c, we also illustrate the eviction of cacheline X again into the VDB of the Persistent Memory Controller andtagged with the set of the two open transactions, T1 and T4. Notethat there are two copies of cache line X in the controller. �e oneat the head of the queue has fewer dependencies (only dependenton T1) than the recent eviction. Any subsequent read for cache lineX returns the most recent copy, the last entry in the VDB. Note howcache lines at the back of the queue have dependency set sizes thatare greater than or equal to entries earlier in the queue.

T1 completes log writing at t11, but is behind T4, which hasn’tyet �nished writing its logs, so neither are yet recoverable. �ePM controller also removes T1 from its dependency set and of thosein the VDB. �e �rst copy of X now has no dependencies in thequeue and is safely wri�en back to PM as shown in 3c.

At time t12, T4 completes writing its logs and both T4 and T1are recoverable. Also, T4 is removed from the dependency sets inthe controller, which allows for Y , Z , and X to �ow to PM .

Strict Durability: Suppose a transaction requires strict dura-bility during its Commit Stage, ensuring that once complete, thetransactional writes will be re�ected in PM if a failure were to occur.If T4 requires strict durability, it is simply durable at the end asthere are no open transactions when it completes. However, T1,T2, and T3, have other constraints. A transaction requiring strictdurability is only durable when it is fully recoverable. Table 3b illus-trates transaction durability when it is circled. T1 must wait untilstep t12 if it requires strict durability as it might have dependencieson T4. T2 is fully durable at time t9 when T1, which started earlier,writes its persist timestamp. At time t10, T3 is fully durable whenT4, which started before T3 completed its concurrency section andcould have introduced transactional dependencies, writes its persisttimestamp which indicates T4 started HTM section later.

5 EVALUATIONWe evaluated our method using benchmarks directly running onhardware and through simulation analysis (described in Section 5.4).Our simulation evaluates the length of the FIFO bu�er and perfor-mance against various Persistent Memory write times. In the directhardware evaluation described next, we employed Intel(R) Xeon(R)E5-2650 v4 series processors, 12 cores per processor, running at2.20 GHz, with Red Hat Enterprise Linux 7.2. HTM transactionswere implemented with Intel Transactional Synchronization Exten-sions (TSX) [27] using a global fallback lock. We built our so�wareusing g++ 4.8.5. Each measurement re�ects an average over twentyrepeats with small variation among the repeats.

Using micro-benchmarks and SSCA2 [28] and Vacation, from theSTAMP [29] benchmark suite, we compared the following methods:

• HTM Only: Hardware Transactional Memory with IntelTSX, without any logging or persistence. �is method pro-vides a baseline for transaction performance in cache mem-ory without any persistence guarantees. If a power failureoccurs a�er a transaction, writes to memory locations maybe le� in the cache, or wri�en back to memory in an out-of-order subset of the transactional updates.

• WrAP: Our method. �e volatile delay bu�er in the con-troller is assumed to be able to keep up with back pres-sure from the cache, as shown in Section 3. We thereforeperform all other aspects of the protocol such as logging,reading timestamps, HTM and fall-back locking, etc.

• WrAP-Strict: Same as above, but we implement the so�-ware strict durability method as described in Section 4.�reads wait until all prior-open transactions have closedbefore proceeding.

• PTL-Eager: (Persistent Transactional Locking). In thismethod, we added persistence to Transactional Locking(TL-Eager) [30–32] by persisting the undo log at the timethat a TL transaction performs its sequence of writes. �eundo-log entries are wri�en with write-through stores andSFENCEs, and once the transaction commits and the newdata values are �ushed into PM, the undo-log entries areremoved.

5.1 Benchmarks�e Scalable Synthetic Compact Applications for benchmarkingHigh Productivity Computing Systems [28], SSCA2, is part of theStanford Transactional Applications for Multi-Processing [29], orSTAMP, benchmark suite. SSCA2 uses a large memory area and hasmultiple kernels that construct a graph and perform operations onthe graph. We executed the SSCA2 benchmark with scale 20, whichgenerates a graph with over 45 million edges. We increased thenumber of threads from 1 to 16 in powers of two and recorded theexecution time for the kernel for each method.

Figure 4: SSCA2 Benchmark Compute Graph Kernel Execu-tion Time as a Function of the Number of Parallel Execution�reads

10

Figure 5: SSCA2BenchmarkComputeGraphKernel Speedupas a Function of the Number of Parallel Execution�reads

Figure 6: SSCA2 Benchmark Compute Graph Kernel HTMAborts as a Function of the Number of Parallel Execution�reads

Figure 4 shows the execution time for each method for the Com-pute Kernel in the SSCA2 benchmark as a function of the number ofthreads. Each method reduces the execution time with increasingnumbers of threads. Our WrAP approach has similar executiontime to HTM in the cache hierarchy with no persistence and is over2.25 times faster than a persistent PTL-Eager method to PM.

Figure 5 shows the speedup for each method as a function ofthe number of threads when compared to a single threaded undolog for the persistence methods and speedup versus no persistencefor the in cache method HTM only. Even though the HTM (cache-only) method does be�er in absolute terms as we saw in Figure 4, itproceeds from a higher baseline for single-threaded execution. PTL-Eager yields a signi�cantly weaker scalability due to the inherentcosts of having to perform persistent �ushes within its concurrentregion.

Figure 7: VacationBenchmarkExecutionTime as a Functionof the Number of Parallel Execution�reads

Figure 8: Vacation Benchmark Execution Timewith VariousPersistent Memory Write Times for Four �reads

Figure 6 shows the number of hardware aborts for both ourWrAP approach and cache-only HTM. Our approach introducesextra writes to log the write-set, and, along with reading the systemtime stamp, extends the transaction time. However, as shown in theFigure, this only slightly increases the number of hardware aborts.

We also evaluated the Vacation benchmark which is part of theSTAMP benchmark suite. �e Vacation benchmark emulates data-base transactions for a travel reservation system. We executed thebenchmark with the low option for lower contention emulation.Figure 7 shows the execution time for each method for the Va-cation benchmark as a function of the number of threads. Eachmethod reduces the execution time with increasing numbers ofthreads. �e WrAP approach follows the trends similar to HTMin the cache hierarchy with no persistence, with both approaches�a�ening execution time a�er 4 threads. We examine the e�ectof strict durability, WrAP-Strict in the �gure, and show that strict

11

durability only introduces a small amount of overhead. For justa single thread, it has the same performance as WrAP relaxed asa thread doesn’t need to wait on other threads, as it is durable assoon as the transaction completes.

Additionally, we examined the e�ect of increased PersistentMemorywrite times on the benchmark. Byte-addressable PersistentMemory can have longer write times. To emulate the longer writetimes for PM, we insert a delay a�er non-temporal stores whenwriting to new cache lines and a delay a�er cache line �ushes. �ewrite delay can be tuned to emulate the e�ect of longer write timestypical of PM. Figure 8 shows the Vacation benchmark executiontime for various PM write times.

�eWrAP approach is less a�ected by increasing PM write timesthan the PTL-Eager approach due to several factors. WrAP performswrite-combining for log entries on the foreground path for eachthread, so writes to several transaction variables may be combinedinto a fewer writes. Also, PTL-Eager transactionally persists an undolog on writes causing a foreground delay.

5.2 Hash TableOur next series of experiments show transaction sizes and highmemory tra�c a�ect overall performance. We create a 64 MBHash Table Array of elements in main memory and transactionallyperform a number of element updates. For each transaction, wegenerate a set of random numbers of a con�gurable size, computetheir hash, and write the value into the Hash Table Array.

Figure 9: Millions of Transactions per Second for Hash Ta-ble Updates of 10 Elements versus Concurrent Number of�reads

First, we create transactions consisting of 10 atomic updates andvary the number of concurrent threads and measure the maximumthroughput. We perform 1 million updates and record the averagethroughput and plot the results in Figure 9. Our approach achievesroughly 3x throughput over PTL-Eager. Figure 10 shows increasingthe write set to 20 atomic updates has similar performance. In both�gures, adding strict durability only slightly decreases the overallperformance; threads wait additional time for the dependency onother transactions to clear before continuing to another transaction.

Figure 10: Millions of Transactions per Second for Hash Ta-ble Updates of 20 Elements versus Concurrent Number of�reads

Figure 11: Average Txps for Increasing Transaction Sizes ofAtomic Hash Table Updates with 6 Concurrent �reads

�e transaction write set was then varied from 2 to 30 elementswith 6 concurrent threads. �e average throughput was recordedand is shown in Figure 11. Even with adding strict durability, WrAPperforms roughly three times faster than PTL-Eager.

A transaction size of ten elements was then varied with a write toread ratio with 6 concurrent threads. �e average throughput wasrecorded and is shown in Figure 12. Unlike transactional memoryapproaches, our approach does not require instrumenting readaccesses and can therefore execute reads at cache speeds.

5.3 Red-Black TreeWe use the transactional Red-Black tree from STAMP [29] initial-ized with 1 million elements. We then perform insert operationson the Red-Black tree and record average transaction times andthroughput over 200k additional inserts. Each transaction inserts

12

Figure 12: Average Txps for Write / Read Percentage ofAtomic Hash Table Updates with 6 Concurrent �reads

Figure 13: Millions of Transactions per Second for AtomicRed-Black Tree Element Inserts versus Number of Concur-rent �reads

an additional element into the Red-Black tree. Inserting an ele-ment into a Red-Black tree �rst requires �nding the insertion pointwhich can take many read operations and can trigger many writesthrough a rebalance. In our experiments, we averaged 63 reads and11 writes per transactional insert of one element into the Red-Blacktree.

We record themaximum throughput of inserts into the Red-Blacktree per second for a varying number of threads in Figure 13. Ascan be seen in the Figure, WrAP has almost 9x higher throughputover PTL-Eager, and with strict durability almost 6x faster. Ourmethod can perform reads at the speed of the hardware, whilePTL-Eager requires instrumenting reads through so�ware to trackdependencies on other concurrent transactions.

Figure 14: Average Atomic Hash Table 10 Element Updatewith Various Persistent Memory Write Times with 8 Con-current �reads

Figure 15: Maximum FIFO �eue Length for Atomic HashTable 10 Element Update with 4 Concurrent �reads

5.4 Persistent Memory Controller AnalysisWe investigated the required length of our FIFO in the Volatile DelayBu�er and performance with respect to Persistent Memory writetimes using an approach similar to [14]. In the absence of readilyavailable memory controllers, we modi�ed the McSimA+ simula-tor [33]. McSimA+ is a PIN [34] based simulator that decouplesexecution from simulation and tightly models out-of-order proces-sor micro-architecture at the cycle level. We extended the simulatorto support the noti�cations for opening and closing WrAPs alongwith extended support for memory reads and writes. We addedsupport for DRAMSim2 [35], a cycle-accurate memory system andDRAM memory controller model library. Write-combining andstore bu�ers were then added with multiple con�guration optionsto allow �ne tuning to match the system to be modeled.

13

Figure 16: Average B-Tree Atomic Element Insert with Var-ious Persistent Memory Write Times with 8 Concurrent�reads

Figure 17: Maximum FIFO �eue Length for B-Tree AtomicElement Insert with 8 Concurrent �reads

To stress the Persistent Memory Controller, we executed anatomic hash table update without any thread contention by havingeach thread update elements on a separate portion of the table. Inthe simulation, we �ll the cache with dirty cache lines so that eachwrite by a thread in a transaction generates write-backs to mainPersistent Memory. For 8 threads, we recorded the average atomichash table update time for 10 elements in each transaction. We thenvary the Persistent Memory write time as a multiple of DRAM writetime. As shown in Figure 14, WrAP is less a�ected by increasingwrite times when compared to PTL-Eager.

Additionally, we record the maximum FIFO bu�er size for variousPersistent Memory write times and 4 concurrent threads, shownin Figure 15. Initially, the bu�er size decreases for an increasingPM write time, due to slower transaction throughput and less cache

evictions into the bu�er. As the write time increases, the bu�erlength increases, but is still less than 1k cache lines or 64KB.

We performed a similar analysis using a B-Tree, where eachthread atomically inserts elements on its own copy of a B-Tree.Each insert into the tree required, on average, over 5 times as manyreads as writes. As shown in Figure 16, our method is less a�ectedby increasing PM write times, due to PTL-Eager instrumenting thelarge portion of the read operations. In this experiment, we useeight concurrent threads each atomically inserting elements intoan initialized B-Tree of 128 elements.

As more reads than writes are generated for each atomic inserttransaction, the FIFO bu�er length remains small. We also exam-ined the FIFO bu�er length in the VDB with 8 concurrent threads.Figure 17 shows the length was less than about 100 elements foreach write speed due to the large proportion of reads.

6 RELATEDWORKRelated Persistence Work: Analysis of consistency models forpersistent memory was considered in [36]. Changes to the front-end cache for ordering cache evictions were proposed in [14, 15,37, 38]. BPFS [37] proposed epoch barriers to control eviction order,while [38] proposed a �ush so�ware primitive to control of updateorder. Snapsho�ing the entire micro architectural state at the pointof a failure is proposed in [39]. A non-volatile victim cache to pro-vide transactional bu�ering was proposed in [14], with the addedproperty of not requiring logging, but requires changes to the front-end cache controller to track pre- and post- transactional statesfor cache lines in both volatile and persistent caches, atomicallymoving them to durable state on transaction commits.

Memory controller support for transaction atomicity in Persis-tent Memory have been proposed in [17–19, 23, 40–42]. Addinga small DRAM bu�er in front of Persistent Memory to improvelatency and to coalesce writes was proposed in [40]. �e use ofa volatile victim cache to prevent uncontrolled cache evictionsfrom reaching PM was described in [17–19], but requires so�warelocking for concurrency control. FIRM [42] describes techniquesto di�erentiate persistent and non-persistent memory tra�c, andpresents scheduling algorithms to maximize system throughputand fairness. Low-level memory scheduling to improve e�ciencyof persistent memory access was studied in [41]. Except for [17–19],none of these works deal with the issues of atomicity or durabilityof write sequences. Our approach e�ectively uses HTM for concur-rency control and does not require changes to the font-end cachecontroller or use logs for replaying transactions to PM.

Related Concurrency Work: Existing non-HTM solutions [9,11, 12] tightly couple concurrency control with durable writes ofeither write-ahead logs or data updates into Persistent Memory tomaintain persistence consistency. So�ware that employs these ap-proaches generally means they must extend the duration for whichthey remain in critical sections, leading to longer times to hold locks,which reduces concurrency and expands transactional duration.Other work [10, 13] decouples concurrency control so that posttransactional values may �ow through cache hierarchy and reachPM asynchronously; however, the write ahead log for an updatingtransaction has to get commi�ed into PM synchronously beforethe transaction can close so that the integrity of the foreground

14

value �ow is preserved across machine restarts. Another hardware-assisted mechanism proposes hardware changes to allow a dual-scheme checkpointing that writes previous check-pointed valuesin the background while collecting current transaction writes [43].

Recent work [13, 21, 22] aims to exploit processor-supportedHTM mechanisms for concurrency control instead of traditionallocking or STM-based approaches. However, all of these solutionsrequire making signi�cant changes to the existing HTM semanticsand implementations. For instance, PHTM [21] and PHyTM [22],propose a new instruction called TransparentFlush which can beused to �ush a cache line from within a transaction to persistentmemory without causing any transaction to abort. �ey also pro-pose a change to the xend instruction that ends an atomic HTMregion, so that it atomically updates a bit in persistent memory aspart of its execution. Similarly, for DUDETM [13] to use HTM, itrequires that designated memory variables within a transaction beallowed to be updated globally and concurrently without causingan abort. Other work [26] utilizes HTM for concurrency control,but requires aliasing all read and write accesses while concurrentlymaintaining log ordering and and replaying logs for retirement.

7 SUMMARYIn this paper we presented an approach that uni�es HTM and PM tocreate durable, concurrent transactions. Our approach works withexisting HTM and cache coherency mechanisms, and does not re-quire changes to existing processor caches or store instructions,avoids synchronous cache-line write-backs on completions, andonly utilizes logs for recovery. �e solution correctly orders HTMtransactions and atomically commits them to Persistent Memory bythe use of a novel so�ware protocol combined with a back-end Per-sistent Memory Controller.

Our approach, evaluated using both micro-benchmarks and theSTAMP suite compares well with standard (volatile) HTM transac-tions. In comparison with persistent transactional locking, ourapproach performs 3x faster on standard benchmarks and almost9x faster on a Red-Black Tree data structure.

REFERENCES[1] F. Farber, S. K. Cha, J. Primsch, C. Bornhovd, S. Sigg, and W. Lehner, “SAP HANA

Database: Data management for modern business applications,” SIGMOD Rec.,vol. 40, no. 4, pp. 45–51, Jan. 2012.

[2] V. Raman, G. A�aluri, R. Barber, N. Chainani, D. Kalmuk, V. KulandaiSamy,J. Leenstra, S. Lightstone, S. Liu, G. M. Lohman et al., “Db2 with blu acceleration:So much more than just a column store,” Proceedings of the VLDB Endowment,vol. 6, no. 11, pp. 1080–1091, 2013.

[3] R. Palamu�am, R. M. Mogrovejo, C. Ma�mann, B.Wilson, K.Whitehall, R. Verma,L. McGibbney, and P. Ramirez, “Scispark: Applying in-memory distributedcomputing to weather event detection and tracking,” in Big Data (Big Data),2015 IEEE International Conference on. IEEE, 2015, pp. 2020–2026.

[4] G. Team, “Gridgain: In-memory computing platform,” 2007.[5] X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman,

D. Tsai, M. Amde, S. Owen et al., “MLlib: Machine learning in apache spark,”Journal of Machine Learning Research, vol. 17, no. 34, pp. 1–7, 2016.

[6] I. Corporation. (2015, July) Intel and Micron Produce Breakthrough MemoryTechnology. [Online]. Available: h�ps://newsroom.intel.com/news-releases/intel-and-micron-produce-breakthrough-memory-technology/

[7] M. Herlihy and J. E. B. Moss, Transactional Memory: Architectural support forlock-free data structures. ACM, 1993, vol. 21,2.

[8] R. Rajwar and J. R. Goodman, “Speculative lock elision: Enabling highly con-current multithreaded execution,” in Proceedings of the 34th annual ACM/IEEEinternational symposium on Microarchitecture. IEEE Computer Society, 2001,pp. 294–305.

[9] H. Volos, A. J. Tack, and M. Swi�, “Mnemosyne: Lightweight persistent mem-ory,” in Proceedings of 16th International Conference on Architectural Support forProgramming Languages and Operating Systems. ACM Press, 2011, pp. 91–104.

[10] E. Giles, K. Doshi, and P. Varman, “So�WrAP: A lightweight framework fortransactional support of storage class memory,” in Mass Storage Systems andTechnologies (MSST), 2015 31st Symposium on, May 2015, pp. 1–14.

[11] D. R. Chakrabarti, H.-J. Boehm, and K. Bhandari, “Atlas: Leveraging locks fornon-volatile memory consistency,” in Proceedings of the 2014 ACM InternationalConference on Object Oriented Programming Systems Languages & Applications,ser. OOPSLA ’14. New York, NY, USA: ACM, 2014, pp. 433–452. [Online].Available: h�p://doi.acm.org/10.1145/2660193.2660224

[12] A. Chatzistergiou, M. Cintra, and S. D. Viglas, “Rewind: Recovery write-aheadsystem for in-memory non-volatile data-structures,” Proceedings of the VLDBEndowment, vol. 8, no. 5, pp. 497–508, 2015.

[13] M. Liu, M. Zhang, K. Chen, X. Qian, Y. Wu, W. Zheng, and J. Ren, “DudeTM:Building Durable Transactions with Decoupling for Persistent Memory,” inProceedings of the Twenty-Second International Conference on ArchitecturalSupport for Programming Languages and Operating Systems, ser. ASPLOS’17. New York, NY, USA: ACM, 2017, pp. 329–343. [Online]. Available:h�p://doi.acm.org/10.1145/3037697.3037714

[14] J. Zhao, S. Li, D. H. Yoon, Y. Xie, and N. P. Jouppi, “Kiln: Closing the performancegap between systems with and without persistence support,” in Proceedings ofthe 46th Annual IEEE/ACM International Symposium on Microarchitecture, ser.MICRO-46. New York, NY, USA: ACM, 2013, pp. 421–432. [Online]. Available:h�p://doi.acm.org/10.1145/2540708.2540744

[15] A. Joshi, V. Nagarajan, S. Viglas, and M. Cintra, “ATOM: Atomic durability innon-volatile memory through hardware logging,” in 2017 IEEE InternationalSymposium on High Performance Computer Architecture (HPCA), 2017.

[16] S. Li, P. Wang, N. Xiao, G. Sun, and F. Liu, “SPMS: Strand based persistentmemory system,” in Proceedings of the Conference on Design, Automation &Test in Europe, ser. DATE ’17. 3001 Leuven, Belgium, Belgium: EuropeanDesign and Automation Association, 2017, pp. 622–625. [Online]. Available:h�p://dl.acm.org/citation.cfm?id=3130379.3130529

[17] E. Giles, K. Doshi, and P. Varman, “Bridging the programming gap between per-sistent and volatile memory usingWrAP,” in Proceedings of the ACM InternationalConference on Computing Frontiers. ACM, 2013, p. 30.

[18] L. Pu, K. Doshi, E. Giles, and P. Varman, “Non-Intrusive Persistence with aBackend NVM Controller,” IEEE Computer Architecture Le�ers, vol. 15, no. 1, pp.29–32, Jan 2016.

[19] K. Doshi, E. Giles, and P. Varman, “Atomic Persistence for SCM with a Non-intrusive Backend Controller,” in �e 22nd International Symposium on High-Performance Computer Architecture (HPCA). IEEE, March 2016.

[20] Z. Wang, H. Yi, R. Liu, M. Dong, and H. Chen, “Persistent transactional memory,”IEEE Computer Architecture Le�ers, vol. 14, no. 1, pp. 58–61, Jan 2015.

[21] H. Avni, E. Levy, and A. Mendelson, “Hardware transactions in nonvolatilememory,” in Proceedings of the 29th International Symposium on DistributedComputing - Volume 9363, ser. DISC 2015. New York, NY, USA: Springer-VerlagNew York, Inc., 2015, pp. 617–630.

[22] H. Avni and T. Brown, “PHyTM: Persistent hybrid transactional memory,” Pro-ceedings of the VLDB Endowment, vol. 10, no. 4, pp. 409–420, 2016.

[23] E. Giles, K. Doshi, and P. Varman, “Brief announcement: Hardware transactionalstorage class memory,” in Proceedings of the 29th ACM Symposium on Parallelismin Algorithms and Architectures, ser. SPAA ’17. New York, NY, USA: ACM, 2017,pp. 375–378. [Online]. Available: h�p://doi.acm.org/10.1145/3087556.3087589

[24] W. Ruan, Y. Liu, and M. Spear, “Boosting timestamp-based transactional memoryby exploiting hardware cycle counters,” ACM Transactions on Architecture andCode Optimization (TACO), vol. 10, no. 4, p. 40, 2013.

[25] Y. Liu, J. Go�schlich, G. Pokam, and M. Spear, “Tsxprof: Pro�ling hardwaretransactions,” in Parallel Architecture and Compilation (PACT ), 2015 InternationalConference on. IEEE, 2015, pp. 75–86.

[26] E. Giles, K. Doshi, and P. Varman, “Continuous Checkpointing of HTMTransactions in NVM,” in Proceedings of the 2017 ACM SIGPLAN InternationalSymposium onMemory Management, ser. ISMM 2017. New York, NY, USA: ACM,2017, pp. 70–81. [Online]. Available: h�p://doi.acm.org/10.1145/3092255.3092270

[27] Intel Corporation, “Intel Transactional Synchronization Extensions,” in IntelArchitecture Instruction Set Extensions Programming Reference, February 2012,ch. 8, h�p://so�ware.intel.com/.

[28] D. A. Bader and K. Madduri, “Design and implementation of the HPCS graphanalysis benchmark on symmetric multiprocessors,” in International Conferenceon High-Performance Computing. Springer, 2005, pp. 465–476.

[29] C. C. Minh, J. Chung, C. Kozyrakis, and K. Olukotun, “STAMP: Stanford trans-actional applications for multi-processing,” inWorkload Characterization, 2008.IISWC 2008. IEEE International Symposium on. IEEE, 2008, pp. 35–46.

[30] D. Dice and N. Shavit, “Understanding tradeo�s in so�ware transactional mem-ory,” in Code Generation and Optimization, 2007. CGO’07. International Symposiumon. IEEE, 2007, pp. 21–33.

[31] D. Dice, O. Shalev, and N. Shavit, “Transactional Locking II,” in DistributedComputing. Springer, 2006, pp. 194–208.

15

[32] C. C. Minh, “TL2-x86, a port of tl2 to x86 architecture,” in On GitHub, ccaominh,tl2-x86. Stanford, 2015. [Online]. Available: h�ps://github.com/ccaominh/tl2-x86

[33] J. H. Ahn, S. Li, O. Seongil, and N. P. Jouppi, “Mcsima+: A manycore simulatorwith application-level+ simulation and detailed microarchitecture modeling,” inPerformance Analysis of Systems and So�ware (ISPASS), 2013 IEEE InternationalSymposium on. IEEE, 2013, pp. 74–85.

[34] C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J.Reddi, and K. Hazelwood, “Pin: building customized program analysis tools withdynamic instrumentation,” in ACM Sigplan Notices, vol. 40, no. 6. ACM, 2005,pp. 190–200.

[35] P. Rosenfeld, E. Cooper-Balis, and B. Jacob, “Dramsim2: A cycle accurate memorysystem simulator,” Computer Architecture Le�ers, vol. 10, no. 1, pp. 16–19, 2011.

[36] S. Pelley, P. M. Chen, and T. F. Wenisch, “Memory persistency,” in ComputerArchitecture (ISCA), 2014 ACM/IEEE 41st International Symposium on. IEEE,2014, pp. 265–276.

[37] J. Condit, E. B. Nightingale, C. Frost, E. Ipek, B. Lee, D. Burger, and D. Coetzee,“Be�er I/O through byte-addressable, persistent memory,” in Proceedings ofthe ACM SIGOPS 22Nd Symposium on Operating Systems Principles, ser. SOSP’09. New York, NY, USA: ACM, 2009, pp. 133–146. [Online]. Available:h�p://doi.acm.org/10.1145/1629575.1629589

[38] S. Venkatraman, N. Tolia, P. Ranganathan, and R. H. Campbell, “Consistent anddurable data structures for non-volatile byte addressable memory,” in Proceedingsof 9th Usenix Conference on File and Storage Technologies. ACM Press, 2011, pp.61–76.

[39] D. Narayanan and O. Hodson, “Whole-system persistence,” in Proceedings of 17thInternational Conference on Architectural Support for Programming Languagesand Operating Systems. ACM Press, 2012, pp. 401–410.

[40] M. K. �reshi, V. Srinivasan, and J. A. Rivers, “Scalable high performancemain memory system using phase-change memory technology,” in Proceedingsof the 36th Annual International Symposium on Computer Architecture, ser.ISCA ’09. New York, NY, USA: ACM, 2009, pp. 24–33. [Online]. Available:h�p://doi.acm.org/10.1145/1555754.1555760

[41] P. Zhou, Y. Du, Y. Zhang, and J. Yang, “Fine-grained QoS scheduling for PCM-based main memory systems,” in Parallel & Distributed Processing (IPDPS), 2010IEEE International Symposium on. IEEE, 2010, pp. 1–12.

[42] J. Zhao, O. Mutlu, and Y. Xie, “Firm: Fair and high-performance memory controlfor persistent memory systems,” in Microarchitecture (MICRO), 2014 47th AnnualIEEE/ACM International Symposium on. IEEE, 2014, pp. 153–165.

[43] J. Ren, J. Zhao, S. Khan, J. Choi, Y. Wu, and O. Mutlu, “�yNVM: Enablingso�ware-transparent crash consistency in persistent memory systems,” in Pro-ceedings of the 48th International Symposium on Microarchitecture. ACM, 2015,pp. 672–685.

16