On Convergence of Model Parallel Proximal Gradient Algorithm forStale Synchronous Parallel System

Yi Zhou∗ Yaoliang Yu† Wei Dai† Yingbin Liang∗ Eric P. Xing†

∗Syracuse University †Carnegie Mellon University

Abstract

With ever growing data volume and modelsize, an error-tolerant, communication effi-cient, yet versatile parallel algorithm has be-come a vital part for the success of manylarge-scale applications. In this work we pro-pose msPG, an extension of the flexible proxi-mal gradient algorithm to the model paralleland stale synchronous setting. The workermachines of msPG operate asynchronously aslong as they are not too far apart, and theycommunicate efficiently through a dedicatedparameter server. Theoretically, we providea rigorous analysis of the various convergenceproperties of msPG, and a salient feature ofour analysis is its seamless generality that al-lows both nonsmooth and nonconvex func-tions. Under mild conditions, we prove thewhole iterate sequence of msPG converges toa critical point (which is optimal under con-vexity assumptions). We further provide aneconomical implementation of msPG, com-pletely bypassing the need of keeping a localfull model. We confirm our theoretical find-ings through numerical experiments.

1 Introduction

Many machine learning and statistics problems fit intothe general composite minimization framework:

minx∈Rd1n

∑ni=1 fi(x) + g(x), (1)

where the first term is typically a smooth empiricalrisk over n training samples and the second term gis a nonsmooth regularizer that promotes structures.

Appearing in Proceedings of the 19th International Con-ference on Artificial Intelligence and Statistics (AISTATS)2016, Cadiz, Spain. JMLR: W&CP volume 51. Copyright2016 by the authors.

Popular examples under this framework include theLasso (least squares loss fi and various sparse regular-izers g), logistic regression (logistic loss fi), boosting(exponential loss fi), support vector machines (hingeloss fi and RKHS norm regularizer g), matrix com-pletion (least squares loss f and trace norm regular-izer g), etc. There is also rising interest in usingnonconvex losses (mainly for robustness to outliers)and nonconvex regularizers (mainly for smaller biasin feature selection and support estimation), see e.g.[15, 26, 31, 35, 36, 38–40].

Due to the apparent importance of the compositeminimization framework and the rapidly growing sizein both model dimension and sample volume, thereis a strong need to develop a practical parallel sys-tem that can solve the problem in (1) efficiently andin a scale that is impossible for a single machine[2, 8, 14, 16, 19, 21, 25, 37]. Existing systems canroughly be divided into three categories: bulk syn-chronous [14, 34, 37], (totally) asynchronous [8, 25],and partially asynchronous (also called stale syn-chronous in this work) [2, 8, 16, 19, 21, 32]. The bulksynchronous parallel mechanism (BSP) forces synchro-nization barriers so that the worker machines can stayon the same page to ensure correctness. However, in areal deployed parallel system BSP usually suffers fromthe straggler problem, that is, the performance of thewhole system is bottlenecked by the slowest workermachine. On the other hand, asynchronous systemsachieve much greater throughputs, although at the ex-pense of potentially losing the correctness of the algo-rithm. The stale synchronous parallel (SSP) mecha-nism is a compromise between the previous two mech-anisms: it allows the worker machines to operate asyn-chronously, as long as they are not too far apart. SSPis particularly suitable for machine learning applica-tions, where iterative algorithms robust to small errorsare usually used to find an appropriate model. Thisview is also practiced by many recent works buildingon the SSP mechanism [2, 16, 19, 21, 24, 27].

Existing parallel systems can also be divided intodata parallel and model parallel. In the former case,

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

one usually distributes the computation involving eachcomponent function fi in (1) into different worker ma-chines. This is suitable when n d, i.e. large datavolume but moderate model size. A popular algorithmfor this case is the stochastic gradient algorithm and itsproximal versions [2, 16, 19, 21], under the SSP mech-anism. In contrast, model parallel refers to the regimewhere d n, i.e. large model size but moderate datavolume. This is the case for many computational biol-ogy and health care problems, where collecting manysamples can be very expensive but for each sample wecan relatively cheaply take a large number of measure-ments (features). As a result we need to partition themodel x into different (disjoint) blocks and distributethem among many worker machines. The proximalgradient [11, 18] or its accelerated version [6] is againa natural candidate algorithm due to its nice abilityof handling nonsmooth regularizers. However, suchproximal gradient algorithm has not been investigatedunder SSP mechanism for model parallelism, althoughsome other types of asynchronous algorithms are stud-ied before (see Section 7). The main goal of this workis to fill in this important gap.

More specifically, we make the following contributions:1). We propose msPG, an extension of the proximalgradient algorithm to the new model parallel and stalesynchronous setting. 2). We provide a rigorous anal-ysis of the convergence properties of msPG. Under avery general condition that allows both nonsmooth andnonconvex functions we prove in Theorem 1 that anylimit point of the sequence generated by msPG is acritical point. Then, inspired by the recent Kurdyka- Lojasiewicz (K L) inequality [3, 5, 9, 11, 20], we furtherprove in Theorem 2 that the whole sequence of msPGin fact converges to a critical point, under mild techni-cal assumptions that we verify for many familiar exam-ples. Lastly, relating msPG to recent works on inexactproximal gradient (on a single machine), we provide,under the new model parallel and SSP setting, a simpleproof of the usual sublinear O(1/t) rate of convergence(assuming convexity). We remark our technical con-tributions with comparison to related work after eachmain results. 3). Building on the recent parameterserver framework [19, 21], we give an economical im-plementation of msPG that completely avoids storinglocal full models in each worker machine. The resultingimplementation only requires storing the partitioneddata (with size O(ndi) for di assigned parameters) andcommunicating a vector of length n in each iteration.4). We corroborate our theoretical findings with con-trolled numerical experiments.

This paper proceeds as follows: We first recall somedefinitions in §2, followed by the proposed algorithmmsPG in §3. Theoretical findings are reported in §4

(with all proofs deferred to the appendix). An eco-nomical implementation of msPG is detailed in §5 andexperimentally verified in §6. Finally, we discuss somerelated works in §7 and conclude in §8.

2 Preliminaries

We collect here some useful definitions that will beneeded in our later analysis.

Since we consider a proper and closed1 function h :Rd → (−∞,+∞] that may not be smooth or convex,we need a generalized notion of “derivative”.

Definition 1 (Subdifferential and critical point, [29]).

The Frechet subdifferential ∂h of h at x ∈ domh is theset of u such that

lim infz6=x,z→x

h(z)−h(x)−u>(z−x)‖z−x‖ ≥ 0, (2)

while the (limiting) subdifferential ∂h at x ∈ domh is

the graphical closure of ∂h:

u : ∃xk → x, h(xk)→ h(x),uk ∈ ∂h(xk)→ u. (3)

The critical points of h are crith := x : 0 ∈ ∂h(x).

Pleasantly, when h is continuously differentiable orconvex, the subdifferential ∂h and critical points crithcoincide with the usual notions.

Definition 2 (Distance and projection). The distancefunction to a closed set Ω ⊆ Rd is defined as:

distΩ(x) := miny∈Ω ‖y − x‖, (4)

and the metric projection onto Ω is:

projΩ(x) := argminy∈Ω ‖y − x‖. (5)

Note that projΩ is always a singleton iff Ω is convex.

Definition 3 (Proximal map, e.g. [29]). A naturalgeneralization of the metric projection using a closedand proper function h is (with parameter η > 0):

proxηh(x) := argminz h(z) + 12η‖z− x‖2, (6)

where ‖ · ‖ is the usual Euclidean norm.

If h decreases slower than a quadratic function (in par-ticular, when h is bounded below), its proximal map iswell-defined for all (small) η. For convex h, the prox-imal map is always a singleton while for nonconvexh, the proximal map can be set-valued. In the lattercase we also abuse the notation proxηh(x) for an ar-bitrary element from that set. The proximal map isthe key component of the popular proximal gradientalgorithms [6, 11, 18].

1An extended real-valued function h is proper if its do-main domh := x : h(x) <∞ is nonempty; it is closed iffits sublevel sets x : h(x) ≤ α is closed for all α ∈ R.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Definition 4 (K L function, [10, 20]). A function h iscalled K L if for all x ∈ dom ∂h there exist λ > 0 anda neighborhood X of x such that for all x ∈ X ∩ [x :h(x) < h(x) < h(x) + λ] the following inequality holds

ϕ′ (h(x)− h(x)) · dist∂h(x)(0) ≥ 1, (7)

where the function ϕ : [0, λ)→ R+, 0 7→ 0, is continu-ous, concave, and has continuous and positive deriva-tive ϕ′ on (0, λ).

The KL inequality (7) is an important tool to boundthe trajectory length of a dynamical system (see[10, 20] and the references therein for some historicdevelopments). It has recently been used to ana-lyze discrete-time algorithms in [1] and proximal al-gorithms in [3, 4, 11]. As we shall see, the functionϕ will serve as a Lyapunov function. Quite conve-niently, most practical functions, in particular, “de-finable” functions and convex functions under certaingrowth condition, are K L. For a more detailed discus-sion of K L functions, including many familiar exam-ples, see [11, Section 5] and [4, Section 4].

3 Problem Formulation

We consider the composite minimization problem:

minx∈Rd F (x), where F (x) = f(x) + g(x). (P)

Usually f is a smooth loss function and g is a reg-ularizer that promotes structure. We consider themodel parallel scenario, that is, we decompose thed model parameters into p disjoint groups, and des-ignate one worker machine for each group. Formally,consider the decomposition Rd = Rd1×Rd2×· · ·×Rdp ,x = (x1, x2, . . . , xp), and let ∇if : Rd → Rdi be thepartial gradient of f on the i-th factor space (ma-chine). Clearly, xi,∇if(x) ∈ Rdi and

∑pi=1 di = d.

The i-th machine is responsible for the i-th factorxi ∈ Rdi , however, we also allow machine i to keepa local copy xi ∈ Rd of the full model parameter.This is for the convenience of evaluating the partialgradient ∇if : Rd → Rdi , and we will discuss in Sec-tion 5 how to implement this in an economical way.Note that unlike the data parallel setting, we do notconsider explicitly distributing the computation of thegradient ∇if .

We extend the proximal gradient algorithm [11, 18] tosolve the composite problem (P) under the new modelparallel setting, and we require the following standardassumptions for our convergence analysis.

Assumption 1. Regarding the functions f, g in (P):

1. They are bounded from below;

2. The function f is differentiable and the gradients∇f , ∇if are Lipschitz continuous with constant Lfand Li, respectively. Set L =

∑pi=1 Li;

3. The function g is closed, and separable, i.e., g(x) =∑pi=1 gi(xi).

The first two assumptions are needed to analyze theproximal gradient algorithm even in the convex andnon-distributed setting, and the third assumption iswhat makes model parallelism interesting (and feasi-ble). We remark that the differentiability assumptionon f can be easily relaxed by smoothing, and the sepa-rability assumption on g can also be relaxed using theproximal average idea in [36]. For brevity we do notpursue these extensions here. One salient feature ofour analysis is that we do not assume convexity on ei-ther f or g (although our conclusions are considerablystronger under convexity).

The separability assumption above on g implies that

proxηg(x) =(proxηg1(x1), . . . , proxηgp(xp)

). (8)

Let us introduce the update operator (on machine i):

Ui(xi)=Ui(x

i, xi) :=proxηgi(xi − η∇if(xi))− xi, (9)

i.e. machine i computes the i-th part of the gradientusing its local model xi, updates its parameter xi incharge using step size η, and finally applies the proxi-mal map of the component function gi. In a real largescale parallel system, the communication delay amongmachines and the unexpected shut down of machinesare practical issues that bottlenecks the performance ofthe system, and hence a more relaxed synchronizationprotocol than full synchronization is needed. Considera global clock shared by all machines and denote Tithe set of active clocks when machine i takes an up-date, and It∈Ti as the indicator function of the eventt ∈ Ti. Formally, the t-th iteration on machine i canbe written as:

msPG

∀i, xi(t+ 1) = xi(t) + It∈TiUi(x

i(t)),

(local) xi(t) =(x1(τ i1(t)), . . . , xp(τ

ip(t))

),

(global) x(t) =(x1(t), . . . , xp(t)

),

where 0 ≤ τ ij(t) ≤ t models the delay among ma-

chines: when machine i conducts its t-th update it onlyhas access to xj(τ

ij(t)), a delayed version of the factor

xj(t) on the j-th machine. In the above, we collectthe fresh parameters of all machines to form a globalmodel x(t), which brings convenience for our analysis.We will refer to the above updates as msPG (for modelparallel, stale synchronous, Proximal Gradient). Fig-ure 1 illustrates the main idea of msPG.

Obviously, to establish convergence for msPG we needsome control over the delay τ ij(t) and the active

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Figure 1: The algorithm msPG under model parallelism and stale synchronism. Machine i keeps a local modelxi(t) that contains stale parameters of other machines (due to communication delay and network latency). Theselocal models are used to compute the partial gradient ∇if(xi) which is then used to update the parameters xi(t)in each machine. See Section 5 for an economical implementation of msPG.

clocks Ti, for otherwise some machines may not makeprogress at all. We first impose the following assump-tions on the delay and active clocks, and follow bysome explanations.

Assumption 2. The delay and skip frequency satisfy:

1. ∀i, ∀j,∀t, 0 ≤ t− τ ij(t) ≤ s;2. ∀i, ∀t, τ ii (t) = t;3. ∀i, ∀t, Ti ∩ t, t+ 1, · · · , t+ s 6= ∅.

The first assumption basically says that if machinei conducts its t-th update, then the information itgathered from other machines cannot be too obso-lete (bounded by s iterations). The second assump-tion simply says that machine i always has the lat-est information on itself. The third assumption re-quires each machine to update at least once in ev-ery s iterations. These assumptions are very natu-ral and have been widely adopted in previous works[2, 8, 16, 19, 22, 24, 32]. They are also in some senseunavoidable: one can construct instances such thatmsPG do not converge if these assumptions are vio-lated. Clearly, when s = 0 (no delay) our frameworkreduces to the bulk synchronous proximal gradient al-gorithm.

4 Convergence Analysis

In this section, we conduct detailed analysis of themodel parallel stale synchronous proximal gradient al-gorithm msPG. Our first result is as follows:

Theorem 1 (Asymptotic consistency). Let Assump-tion 1 and 2 hold, and apply msPG to problem (P). Ifthe step size η < (Lf + 2Ls)−1, then the global modeland local models satisfy:

1.∑∞t=0 ‖x(t+ 1)− x(t)‖2 <∞;

2. limt→∞

‖x(t+ 1)− x(t)‖ = 0, limt→∞

‖x(t)− xi(t)‖ = 0;

3. The limit points ω(x(t)) = ω(xi(t)) ⊆ critF .

The proof is a bit involved and can be foundin Appendix A. The first assertion says that theglobal sequence x(t) defined in msPG has squaresummable successive differences, which we will signif-icantly strengthen below. The second assertion im-plies that the successive difference of the global se-quence diminishes, and the inconsistency between thelocal sequences and the global sequence also vanishes.These two conclusions provide some stability guaran-tee about our algorithm msPG. The third assertion fur-ther justifies msPG by showing that any limit pointit produces is necessarily a critical point. Of course,when F is convex, any critical point is globally opti-mal.

The closest result to Theorem 1 we are aware of is [8,Proposition 7.5.3], where essentially the same conclu-sion was reached but under the much more simplifiedassumption that g is an indicator function of a con-vex set. Thus, our Theorem 1 is new even when g isa convex function such as the popular `1 norm. Fur-thermore, we allow g to be nonconvex, which is partic-ularly interesting due to the rising interest in noncon-vex penalties in machine learning and statistics (seee.g. [15, 26, 31, 35, 36, 38–40]). We also note that theproof of Theorem 1 (for nonconvex g) involves signifi-cantly new ideas beyond those of [8].

Interesting as it is, Theorem 1 has one significant de-ficiency, though: it does not tell us when there existsa limit point, and it does not guarantee the whole se-quence to converge to the limit point. In fact, in themodel parallel setting with delays and skips, it is evena nontrivial task to argue that the objective valuesF (x(t)) do not diverge to infinity. This is in sharpcontrast with the bulk synchronous setting where itis trivial to guarantee the objective values to decrease(by using a sufficiently small step size). This is wherewe need some further assumptions.

Assumption 3 (Sufficient Decrease). There existsα > 0 such that the global model x(t) generated by

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

msPG (for problem (P)) satisfies: for all large t,

F (x(t+ 1)) ≤ F (x(t))− α‖x(t+ 1)− x(t)‖2. (10)

We will provide below a sufficient condition inLemma 1 and many interesting examples in Example 1to ensure Assumption 3 to hold.

Assumption 4. F is a K L function.

As we mentioned in the end of Section 2, most practicalfunctions (including all functions in this work) are K L.Hence, this is a very mild assumption.

We are now ready to state the much more strengthenedconvergence guarantee for msPG:

Theorem 2 (Finite Length). Let Assumption 1, 2,3 and 4 hold, and apply msPG to problem (P). If thestep size η < (Lf+2Ls)−1 and x(t) is bounded, then∑∞

t=0 ‖x(t+ 1)− x(t)‖ <∞, (11)

∀i = 1, . . . , p,∑∞t=0 ‖xi(t+ 1)− xi(t)‖ <∞. (12)

Furthermore, x(t) and xi(t), i = 1, ..., p, convergeto the same critical point of F .

The proof, in Appendix B, borrowed some ideas fromthe recent works [3, 4, 11], which, however, did not con-sider the parallel and asynchronous setting. Comparedwith the first assertion in Theorem 1, we now have thesuccessive differences to be absolutely summable (in-stead of square summable). The former property isusually called finite length in dynamical systems. Itis a significantly stronger property as it immediatelyimplies that the whole sequence is Cauchy hence con-vergent, whereas we cannot get the same conclusionfrom the square summable property in Theorem 1.2

We note that local maxima are excluded from beingthe limit in Theorem 2, thanks to Assumption 3. Also,the boundedness assumption on x(t) is easy to sat-isfy, for instance, when F has bounded sublevel sets.We refer to [4, Remark 3.3] for more conditions thatguarantee the boundedness. Needless to say, if F isconvex, then the whole sequence in Theorem 2 con-verges to a global minimizer.

The closest result to Theorem 2, to our best knowl-edge, is [32], which proved an s-step linear rate whenf satisfies a certain error bound condition and g is theindicator function of a convex set. In contrast, we al-low any convex or nonconvex function g (as long asit is K L). Our proof, inspired by [3, 4, 11], differssubstantially from [32].

2A simple example would be the sequence x(t) =∑tk=1

1k

, whose successive difference is square summablebut clearly x(t) does not converge. Consequently, x(t) isnot absolutely summable.

We now provide some justifications on Assumption 3by considering the simplified case where ∀t, i, t ∈ Ti,i.e., machines do not skip updates. The general casecan also be dealt with but the analysis is much morecomplicated. The following condition truns out to bea good justification for Assumption 3.

Assumption 5 (Eventual Lipschitz). The update op-erators Ui, i = 1, ..., p are eventually Lipschitz contin-uous, i.e., for all large t and small learning rate η > 0:

‖Ui(xi(t+1))−Ui(xi(t))‖≤Ciη‖xi(t+1)−xi(t)‖, (13)

where Ci ≥ Li, i = 1, ..., p, are positive constants.

Note that when g ≡ 0, Ui = −η∇if is Lipschitz con-tinuous (due to Assumption 1.2), thus Assumption 5is a natural generalization to an arbitrary regularizerg. Equipped with this assumption, we can now jus-tify Assumption 3. (Proof is in Appendix C.) In thesequel, we denote C =

∑pi=1 Ci ≥ L.

Lemma 1. Assume ∀t, i, t ∈ Ti. Let the step size

η < ρ−14Cρ

√ρ−1

√ρs+1−1

for any ρ > 1 and all Ui, i = 1, ..., p

be eventually Lipschitz continuous, then the sequencesx(t) and xi(t), i = 1, ..., p, have finite length.

Hence it is sufficient to further characterize Assump-tion 5, which turns out to be a mild condition. In-stead of giving a very technical justification, we givehere some popular examples where Assumption 5 holds(proof in Appendix D). Some of these will also betested in our experiments.

Example 1. Assume ∀t, i, t ∈ Ti, then Assumption 5holds for the following cases (modulo a technical con-dition on the 1-norm):

• g ≡ 0 (no regularization), g = ‖ · ‖0 (nonconvex0-norm), g = ‖ · ‖1 (1-norm), g = ‖ · ‖2 (squared2-norm);

• elastic net g = ‖ · ‖1 + ‖ · ‖2 and its nonconvex vari-ation g = ‖ · ‖0 + ‖ · ‖2;

• non-overlapping group norms g = ‖ · ‖0,2 and g =‖ · ‖0,2 + ‖ · ‖2.

Further for this non-skip case (∀t, i, t ∈ Ti), msPGcan be cast as an inexact proximal gradient algorithm(IPGA), which, together with the finite length prop-erty in Theorem 2, provide new insights on the natureof staleness in real parallel systems. It also allows usto easily obtain the usual O(1/t) rate of convergenceof the objective value.

Let us introduce an error term e(t) =(e1(t), . . . , ep(t)

), with which we can rewrite

the global sequence of msPG as:

∀i, xi(t+1)= proxηgi (xi(t)−η∇if(x(t))+ei(t)) , (14)

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

where ei(t) = η[∇if(x(t)) − ∇if(xi(t))]. This alter-native representation of msPG falls under the IPGAin [30], where e(t) is the (gradient) error at iterationt. Note, however, that the error e(t) is not caused bycomputation but by communication delay and networklatency, which only presents itself in a real stale syn-chronous parallel system. For convex functions F , [30]showed that the convergence rate of the objective valueof IPGA can be controlled by the summability of theerror magnitude ‖e(t)‖. Interestingly, our next resultproves that the finite length property in Theorem 2 im-mediately implies the summability of the errors ‖e‖,even for nonconvex functions f and g. Moreover, forconvex F , this also leads to the usual O(1/t) rate ofconvergence in terms of the objective value.

Theorem 3 (Global rate of convergence). If the finitelength property in Theorem 2 holds, then

1.∑∞t=0 ‖e(t)‖ <∞;

2. F ( 1t

∑tk=1 x(k))− inf F ≤ O

(t−1).

The proof is in Appendix E. Intuitively, if the error e(t)decreases (slightly) faster than O(1/t), then the rateof convergence of msPG is not affected even under themodel parallel and stale synchronous setting (providedF is convex). To the best of our knowledge, this isthe first deterministic rate of convergence result in themodel parallel and stale synchronous setting.

5 Economical Implementation

In this section, we show how to economically imple-ment msPG for the widely used linear models:

minx∈Rd f(Ax) + g(x), (15)

where A ∈ Rn×d. Typically f : Rn → R is the like-lihood function and g : Rd → R is the regularizer(we absorb the regularization constant into g). Eachrow of A corresponds to a sample point and we havesuppressed the labels in classification or responses inregression. Support vector machines, Lasso, logisticregression, boosting, etc. all fit under this framework.Our interest here is when the model dimension d ismuch higher than the number of samples n (d can beup to hundreds of millions and n can be up to mil-lions). This is the usual setup in many computationalbiology and health care problems.

A naive implementation of msPG might be inefficientin terms of both network communication and param-eter storage. First, Each machine needs to communi-cate with every other machine, to exchange the latestblock of parameters. If using a peer-to-peer networktopology, the resulting connections will be too denseand crowded when the system holds hundreds of ma-chines. We resolve this issue by adopting the parame-

ter server system advocated in previous works [19, 21],that is, we dedicate a specific server (which can span aset of machines if needed) to store the key parameters(will be specified later) and let each worker machine tocommunicate only with the server. There is a secondadvantage for this master-slave network topology, aswe shall see momentarily.

Second, each machine needs to keep a local copy ofthe full model (i.e. xi(t) in msPG), which can incura very expensive storage cost when the dimension ishigh. This is where the linear model structure in (15)comes into help. Note that the local models xi(t) arekept solely for the convenience of evaluating the partialgradient ∇if : Rd → Rdi . For some problems such asthe Lasso, a seemingly workaround is to pre-computethe Hessian H = A>A and distribute the correspond-ing row blocks of H to each worker machine. Thisscheme, however, is problematic in the high dimen-sional setting: the pre-computation of the Hessian canbe very costly, and each row block of H has a verylarge size (di × d).

Instead, we use the column partition scheme [e.g. 12,28], namely, we partition the matrix A into p columnblocks A = [A1, . . . , Ap] and distribute the block Ai ∈Rn×di to machine i. Now the local update computedby machine i at the t-th iteration can be rewritten as

Ui(xi(t))=proxηgi

(xi(t)−ηA>i f ′(Axi(t))

)−xi(t) (16)

Since machine i is in charge of updating the i-th blockxi(t) of the global model, to compute the local update(16) it is sufficient to have the matrix-vector productAxi(t). For simplicity we initialize ∀i, xi(0) ≡ 0, thenwe have the following cumulative form:

Axi(t) =

p∑j=1

Aj [xi(t)]j =

p∑j=1

τ ij (t)∑k=0

AjIk∈TjUj(xj(k))︸ ︷︷ ︸

∆j(k)

,

where recall that when machine i conducts its t-th it-eration it only has access to a delayed copy xj(τ

ij(t)) of

the parameters in machine j. Since this matrix-vectorproduct is needed by every machine to conduct theirlocal updates in (16), we aggregate ∆j(t) ∈ Rn on theparameter server whenever it is generated and sentby the worker machines. In details, the worker ma-chines first pull this aggregated matrix-vector product(denoted as N) from the server to conduct the localcomputation (16) in an economical way (by replacingAxi(t) in (16) with N). Then machine i performs thesimple update:

xi(t+ 1) = xi(t) + Ui(xi(t)). (17)

Note that machine i does not maintain or update otherblocks of parameters xj(t), j 6= i. Lastly, machine i

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Algorithm 1 Economic Implementation of msPG

1: For the server:2: while recieves update ∆i from machine i do3: N← N + ∆i

4: end while5: while machine i sends a pull request do6: send N to machine i7: end while8: For machine i at active clock t ∈ Ti:9: pull N from the server

10: Ui ← proxηgi(xi − ηA>i f ′(N)

)− xi

11: send ∆i = AiUi to the server12: update xi ← xi + Ui

computes and sends the vector ∆i(t) = AiUi(xi(t)) ∈

Rn to the server, and the server immediately performsthe aggregation:

N← N + ∆i(t). (18)

We summarize the above economical implementationin Algorithm 1, where N denotes the aggregation of in-dividual matrix-vector products ∆ on the server. Thestorage cost for each worker machine is O(ndi) (forstoring Ai). Each iteration requires two matrix-vectorproducts that cost O(ndi) in the dense case, and thecommunication of a length n vector between the serverand the worker machines.

6 Experiments

We first test the convergence properties of msPG via anon-convex Lasso problem with the group regularizer‖ · ‖0,2, which takes the form

minx12‖Ax− b‖2 + λ

∑20i=1 I(‖xi‖ = 0), (19)

where A ∈ R1000×2000 and we refer to Appendix F forthe specifications of data generation. We use 4 ma-chines (cores) with each handling five groups of coor-dinates, and consider staleness s = 0, 10, 20, 30, respec-tively. To better demonstrate the effect of staleness,we let machines only communicate when exceed themaximum staleness. This can be viewed as the worstcase communication scheme and a larger s brings morestaleness into the system. We set the learning rate tohave the form η(αs) = 1/(Lf +2Lαs), α > 0, that is, alinear dependency on staleness s as suggested by The-orem 1. Then we run Algorithm 1 with different stal-eness and use η(0), η(10), η∗(αs), respectively, whereη∗(αs) is the largest step size we tuned for each s thatachieves a stable convergence. We track the globalmodel x(t) and plot the results in Figure 2. Note thatwith the large step size η(0) all instances (with nonzero

staleness) diverge hence are not presented. With η(10)(Figure 2, left), the staleness does not substantiallyaffect the convergence in terms of the objective value.We note that the objective curves converge to slightlydifferent minimal values due to the non-convexity ofproblem (19). With η∗(αs) (Figure 2, middle), it canbe observed that adding a slight penalty αs on thelearning rate suffices to achieve a stable convergence,and the penalty grows as s increases, which is intuitivesince a larger staleness requires a smaller step size tocancel the inconsistency. In particular, for s = 10 thebest convergence is comparable to the bulk synchro-nized case s = 0. (Figure 2, right) further shows theasymptotic convergence behavior of the global modelx(t) under the step size η∗(αs). It is clear that a linearconvergence is eventually attained, which confirms thefinite length property in Theorem 2.

Next, we verify the time and communication efficiencyof msPG via an l1 norm Lasso problem with very highdimensions, taking the form

minx12‖Ax− b‖22 + λ‖x‖1. (20)

We generate A ∈ Rn×d of size n = 1Million andd = 100Millions. See Appendix F for the specificationsof data generation. We implement Algorithm 1 onPetuum [13, 19] — a stale synchronous parallel systemwhich eagerly updates the local parameter caches viastale synchronous communications. The system con-tains 100 computing nodes and each is equipped with16 AMD Opteron processors and 16GB RAM linkedby 1Gbps ethernet. We fix the learning rate η = 10−3

and consider maximum staleness s = 0, 1, 3, 5, 7, re-spectively. (Figure 3, left) shows that per-iterationprogress is virtually indistinguishable among variousstaleness settings, which is consistent with our previ-ous experiment. (Figure 3, middle) shows that systemthroughput is significantly higher when we introducestaleness. This is due to lower synchronization over-heads, which offsets any potential loss due to stalenessin progress per iteration. We also track the distribu-tions of staleness during the experiments, where werecord in N the clocks of the freshest updates thataccumulate from all the machines. Then whenever amachine pulls N from the server, it compares its lo-cal clock with these clocks and records the clock dif-ferences. (Figure 3, right) shows the distributions ofstaleness under different maximal staleness settings.Observe that bulk synchronous (s = 0) peaks at stale-ness 0 by design, and the distribution concentrates insmall staleness area due to the eager communicationmechanism of Petuum. It can be seen that a smallamount of staleness is sufficient to relax the communi-cation bottlenecks without affecting the iterative con-vergence rate much.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

0 100 200 300 400 500 600 700 800 900 10000

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

number of iterations

objectivevalue

learning rate = η(10)

s=0s=10s=20s=30

0 50 100 150 200 250 300 350 400 450 500 5500

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

number of iterations

objectivevalue

learning rate = η∗(αs)

s=0, αs=0s=10, αs=0.094s=20, αs=0.25s=30, αs=0.45

0 500 1000 1500 2000 2500 3000−14

−12

−10

−8

−6

−4

−2

0

number of iterations

log(‖xc−x∗‖)

learning rate η∗(αs)

s=0 αs=0s=10 αs=0.094s=20 αs=0.25s=30 αs=0.45

Figure 2: Convergence curves of msPG under different staleness parameter s and step size η.

0 5 10 15 20 25 30 352

4

6

8

10

12

14x 10

4

number of iterations

objectivevalue

Objective vs Iteration

s=0

s=1

s=3

s=5

s=7

100 200 300 400 500 600 700 800 900 10003

4

5

6

7

8

9

10

11x 10

4

seconds

objectivevalue

Objective vs Seconds

s=0s=1s=3s=5s=7

0 1 2−3 4−5 6−70

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

staleness

percentage

Staleness Distributions

s = 0s = 1s = 3s = 5s = 7

Figure 3: Efficiency of msPG on a large scale Lasso problem.

7 Related Work

The stale synchronous parallel system dates back to[7, 8, 32, 33], where it is also refered to as partiallyasyncrhonous system. These work consider using stalesynchronous systems to solving different kinds of op-timizations problems with allowing machines to skipupdates during the process. Asymptotic convergenceof partially asynchronous gradient descent algorithmfor solving unconstraint smooth optimizations is es-tablished in [8], with its stochastic version being an-alyzed in [33]. Asymptotic convergence of partiallyasynchronous gradient projection algorithm for solv-ing smooth optimizations with convex constraint is es-tablished in [8], and a “B-step” linear convergence isfurther established in [32] with an error bound con-dition. Linear convergence of partially asynchronousalgorithm for finding the fixed point of maximum normcontraction mappings is established in [7, 17].

Another series of work focus on SSP systems wheremachines are not allowed to skip updates [21–23]. Intheir settings, The system imposes an upper boundon the maximum clock difference between machines.Asymptotic convergence is established for proximalgradient algorithm for data parallelism [23] and forblock coordinate descent [22] with a smooth objec-tive and convex regularizer. Other works considerstochastic algorithms on stale synchronous system.[19] proposes an SSP system for stochastic gradientdescent, and establishes O(1/

√k) regret bound under

bounded diameter and bounded sub-gradient assump-

tion. [16, 27] consider a delayed stochastic gradientdescent algorithm. Linear convergence to a neighbor-hood of optimum is established with strong convexityassumption in [16] and with additional bounded gra-dient assumption in [27]. [2] proposes a distributeddelayed dual averaging and mirror descent algorithm,and establishes O(1/

√k) regret bound under standard

stochastic assumptions.

8 Conclusion

We have proposed msPG as an extension of the proxi-mal gradient algorithm to the model parallel and stalesynchronous setting. msPG allows worker machines tooperate asynchronously as long as they are not too farapart, hence greatly improves the system throughput.Theoretically, we provide a rigorous analysis of msPGthat simultaneously covers nonsmooth and nonconvexfunctions. In particular, under mild conditions, thewhole iterate sequence generated by msPG convergesto a critical point. We implement msPG using the pa-rameter server platform, and completely bypass theneed of keeping a local full model. Preliminary nu-merical experiments confirm the effectiveness of msPGon solving very high dimensional problems.

Acknowledgement

We thank the reviewers for their valuable com-ments. This work is supported by NIH R01GM114311,DARPA FA87501220324 and NSF IIS1447676.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

References

[1] P.-A. Absil, R. Mahony, and B. Andrews. Conver-gence of the iterates of descent methods for ana-lytic cost functions. SIAM Journal on Optimization,16(2):531–547, 2005.

[2] A. Agarwal and J. C. Duchi. Distributed delayedstochastic optimization. In Advances in Neural Infor-mation Processing Systems 24, pages 873–881. 2011.

[3] H. Attouch and J. Bolte. On the convergence of theproximal algorithm for nonsmooth functions involvinganalytic features. Mathematical Programming, 116(1-2):5–16, 2009.

[4] H. Attouch, J. Bolte, P. Redont, and A. Soubeyran.Proximal alternating minimization and projectionmethods for nonconvex problems: An approach basedon the Kurdyka- Lojasiewicz inequality. Mathematicsof Operations Research, 35(2):438–457, 2010.

[5] H. Attouch, J. Bolte, and B. Svaiter. Convergenceof descent methods for semi-algebraic and tame prob-lems: proximal algorithms, forward-backward split-ting, and regularized Gauss-Seidel methods. Mathe-matical Programming, 137(1-2):91–129, 2013.

[6] A. Beck and M. Teboulle. A fast iterative shrinkage-thresholding algorithm for linear inverse problems.SIAM J. Img. Sci., 2(1):183–202, 2009.

[7] D. P. Bertsekas and J. N. Tsitsiklis. Convergence rateand termination of asynchronous iterative algorithms.In Proceedings of the 3rd International Conference onSupercomputing, pages 461–470, 1989.

[8] D. P. Bertsekas and J. N. Tsitsiklis. Parallel and Dis-tributed Computation: Numerical Methods. Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1989.

[9] J. Bolte, A. Daniilidis, and A. Lewis. The Lojasiewiczinequality for nonsmooth subanalytic functions withapplications to subgradient dynamical systems. SIAMJournal on Optimization, 17:1205–1223, 2007.

[10] J. Bolte, A. Danilidis, O. Ley, and L. Mazet.Characterizations of Lojasiewicz inequalities and ap-plications: Subgradient flows, talweg, convexity.Transactions of the American Mathematical Society,362(6):3319–3363, 2010.

[11] J. Bolte, S. Sabach, and M. Teboulle. Proximal al-ternating linearized minimization for nonconvex andnonsmooth problems. Mathematical Programming,146(1-2):459–494, 2014.

[12] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eck-stein. Distributed optimization and statistical learn-ing via the alternating direction method of multipliers.Foundations and Trends in Machine Learning, 3(1):1–122, 2010.

[13] W. Dai, A. Kumar, J. Wei, Q. Ho, G. Gibson, andE. P. Xing. High-performance distributed ml atscale through parameterserver consistency models. InAAAI, 2014.

[14] J. Dean and S. Ghemawat. Mapreduce: Simplifieddata processing on large clusters. Communications ofACM, 51(1):107–113, 2008.

[15] J. Fan and R. Li. Variable selection via nonconcavepenalized likelihood and its oracle properties. Journalof the American Statistical Association, 96(456):1348–1360, 2001.

[16] H. Feyzmahdavian, A. Aytekin, and M. Johansson. Adelayed proximal gradient method with linear conver-gence rate. In 2014 IEEE International Workshop onMachine Learning for Signal Processing.

[17] H. Feyzmahdavian and M. Johansson. On the con-vergence rates of asynchronous iterations. In 2014IEEE 53rd Annual Conference on Decision and Con-trol, pages 153–159, 2014.

[18] M. Fukushima and H. Mine. A generalized proximalpoint algorithm for certain non-convex minimizationproblems. International Journal of Systems Science,12(8):989–1000, 1981.

[19] Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B.Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing.More effective distributed ml via a stale synchronousparallel parameter server. In Advances in Neural In-formation Processing Systems 26, pages 1223–1231.2013.

[20] K. Kurdyka. On gradients of functions definable ino-minimal structures. Annales de l’institut Fourier,48(3):769–783, 1998.

[21] M. Li, D. G. Andersen, J. W. Park, A. J. Smola,A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, andB.-Y. Su. Scaling distributed machine learning withthe parameter server. In 11th USENIX Symposium onOperating Systems Design and Implementation (OSDI14), pages 583–598, 2014.

[22] M. Li, D. G. Andersen, and A. Smola. Distributed de-layed proximal gradient methods. Big Learning NIPSWorkshop, 2013.

[23] M. Li, L. Zhou, Z. Yang, A. Li, F. Xia, D. G. An-dersen, and A. Smola. Parameter server for dis-tributed machine learning. Big Learning NIPS Work-shop, 2013.

[24] J. Liu and S. J. Wright. Asynchronous stochastic co-ordinate descent: Parallelism and convergence prop-erties. SIAM Journal on Optimization, 25(1):351–376,2015.

[25] Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Ky-rola, and J. M. Hellerstein. Distributed graphlab: Aframework for machine learning and data mining inthe cloud. Proc. VLDB Endow., 5(8):716–727, 2012.

[26] R. Mazumder, J. H. Friedman, and T. Hastie.Sparsenet: Coordinate descent with nonconvex penal-ties. Journal of the American Statistical Association,106(495):1125–1138, 2011.

[27] B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: Alock-free approach to parallelizing stochastic gradientdescent. In Advances in Neural Information Process-ing Systems 24, pages 693–701. 2011.

[28] P. Richtarik and M. Takac. Distributed CoordinateDescent Method for Learning with Big Data. arXiv,2013.

[29] R. Rockafellar and R. Wets. Variational Analysis.Springer, 1997.

[30] M. Schmidt, N. L. Roux, and F. R. Bach. Conver-gence rates of inexact proximal-gradient methods forconvex optimization. In Advances in Neural Informa-tion Processing Systems 24, pages 1458–1466. 2011.

[31] Y. She. Thresholding-based iterative selection proce-dures for model selection and shrinkage. ElectronicJournal of Statistics, 3:384–415, 2009.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

[32] P. Tseng. On the rate of convergence of a partiallyasynchronous gradient projection algorithm. SIAMJournal on Optimization, 1(4):603–619, 1991.

[33] J. Tsitsiklis, D. Bertsekas, and M. Athans. Distributedasynchronous deterministic and stochastic gradientoptimization algorithms. IEEE Transactions on Au-tomatic Control, 31(9):803–812, 1986.

[34] L. G. Valiant. A bridging model for parallel computa-tion. Communications of ACM, 33(8):103–111, 1990.

[35] Z. Wang, H. Liu, and T. Zhang. Optimal computa-tional and statistical rates of convergence for sparsenonconvex learning problems. The Annals of Statis-tics, 42(6):2164–2201, 2014.

[36] Y. Yu, X. Zheng, M. Marchetti-Bowick, and E. P.Xing. Minimizing nonconvex non-separable functions.In The 17th International Conference on Artificial In-telligence and Statistics (AISTATS), 2015.

[37] M. Zaharia, M. Chowdhury, M. J. Franklin,S. Shenker, and I. Stoica. Spark: Cluster computingwith working sets. pages 10–10, 2010.

[38] C.-H. Zhang. Nearly unbaised variable selection un-der minimax concave penalty. Annals of Statistics,38(2):894–942, 2010.

[39] C.-H. Zhang and T. Zhang. A general theory of con-cave regularization for high-dimensional sparse esti-mation problems. Statistical Science, 27(4):576–593,2012.

[40] Y. Zhu, X. Shen, and W. Pan. Simultaneous group-ing pursuit and feature selection over an undirectedgraph. Journal of the American Statistical Associa-tion, 108(502):713–725, 2013.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Supplementary Materials

A Proof of Theorem 1

Theorem 1 (Asymptotic consistency). Let Assumption 1 and 2 hold, and apply msPG to problem (P). If thestep size η < (Lf + 2Ls)−1, then the global model and local models satisfy:

1.∑∞t=0 ‖x(t+ 1)− x(t)‖2 <∞;

2. limt→∞

‖x(t+ 1)− x(t)‖ = 0, limt→∞

‖x(t)− xi(t)‖ = 0;

3. The limit points ω(x(t)) = ω(xi(t)) ⊆ critF .

Proof. We start from bounding the difference between the global model x and the local model xi (on any machinei). Indeed, at iteration t, by the definition of the global and local models in msPG:

‖x(t)− xi(t)‖ =

√√√√ p∑j=1

‖xj(t)− xj(τ ij(t))‖2 (21)

( triangle inequality ) ≤

√√√√√ p∑j=1

t−1∑k=τ i

j (t)

‖xj(k + 1)− xj(k)‖

2

(22)

( Assumption 2.1 ) ≤

√√√√√ p∑j=1

t−1∑k=(t−s)+

‖xj(k + 1)− xj(k)‖

2

(23)

=

∥∥∥∥∥∥( t−1∑k=(t−s)+

‖x1(k + 1)− x1(k)‖, · · · ,t−1∑

k=(t−s)+

‖xp(k + 1)− xp(k)‖)∥∥∥∥∥∥ (24)

=

∥∥∥∥∥∥t−1∑

k=(t−s)+

(‖x1(k + 1)− x1(k)‖, · · · , ‖xp(k + 1)− xp(k)‖

)∥∥∥∥∥∥ (25)

( triangle inequality ) ≤t−1∑

k=(t−s)+

∥∥∥(‖x1(k + 1)− x1(k)‖, · · · , ‖xp(k + 1)− xp(k)‖)∥∥∥ (26)

=

t−1∑k=(t−s)+

‖x(k + 1)− x(k)‖, (27)

where in the last equality we used the following property of the Euclidean norm:

‖x‖ = ‖(x1, . . . , xp)‖ = ‖(‖x1‖, . . . , ‖xp‖)‖. (28)

Equation (27) bounds the inconsistency between the global model and the local models. We will repeatedly useit in the following, as it provides a bridge to jump from the global model and the local models back and forth.

Next we bound the progress of the global model x(t). If t ∈ Ti (i.e., machine i updates at iteration t), then usingthe definition of the update operator Ui(x

i(t)) in Equation (9) we can rewrite xi(t+ 1) as

xi(t+ 1) = proxηgi(xi(t)− η∇if(xi(t))

), (29)

where we recall the proximal map proxηgi from Definition 3. Thus, for all z ∈ Rdi :

gi(xi(t+ 1)

)+

1

2η

∥∥xi(t+ 1)− xi(t) + η∇if(xi(t)

)∥∥2 ≤ gi(z)

+1

2η

∥∥z − xi(t) + η∇if(xi(t)

)∥∥2. (30)

Substituting with z = xi(t) and simplifying yields

gi(xi(t+ 1)

)− gi

(xi(t)

)≤ − 1

2η‖xi(t+ 1)− xi(t)‖2 −

⟨∇if

(xi(t)

), xi(t+ 1)− xi(t)

⟩. (31)

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

(If gi is convex, we can replace 12η with 1

η .) Note that if t 6∈ Ti, then xi(t+ 1) = xi(t) hence Equation (31) stilltrivially holds. On the other hand, Assumption 1.2 implies

f(x(t+ 1)

)− f

(x(t)

)≤ 〈x(t+ 1)− x(t),∇f

(x(t)

)〉+

Lf2‖x(t+ 1)− x(t)‖2. (32)

Adding Equation (32) and Equation (31) (for all i) and recalling F (x) = f(x) +∑i gi(xi), we have

F(x(t+ 1)

)− F

(x(t)

)≤ 1

2 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 +

p∑i=1

⟨xi(t+ 1)− xi(t),∇if(x(t))−∇if

(xi(t)

)⟩(33)

( Cauchy-Schwarz ) ≤ 12 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 +

p∑i=1

‖xi(t+ 1)− xi(t)‖ · ‖∇if(x(t))−∇if(xi(t)

)‖

(34)

( Assumption 1.2 ) ≤ 12 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 +

p∑i=1

‖xi(t+ 1)− xi(t)‖ · Li‖x(t)− xi(t)‖ (35)

( Equation (27) ) ≤ 12 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 +

p∑i=1

Li‖xi(t+ 1)− xi(t)‖ ·t−1∑

k=(t−s)+

‖x(k + 1)− x(k)‖

(36)

( Assumption 1.2 ) ≤ 12 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 + L‖x(t+ 1)− x(t)‖ ·

t−1∑k=(t−s)+

‖x(k + 1)− x(k)‖

(37)

( ab ≤ a2+b2

2 ) ≤ 12 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 +

L

2

t−1∑k=(t−s)+

[‖x(k + 1)− x(k)‖2 + ‖x(t+ 1)− x(t)‖2

](38)

≤ 12 (Lf + Ls− 1/η)‖x(t+ 1)− x(t)‖2 +

L

2

t−1∑k=(t−s)+

‖x(k + 1)− x(k)‖2. (39)

Summing the above inequality from m to n− 1 we have

F(x(n)

)− F

(x(m)

)≤ 1

2 (Lf + Ls− 1/η)

n−1∑t=m

‖x(t+ 1)− x(t)‖2 +L

2

n−1∑t=m

t−1∑k=(t−s)+

‖x(k + 1)− x(k)‖2 (40)

≤ 12 (Lf + 2Ls− 1/η)

n−1∑t=m

‖x(t+ 1)− x(t)‖2. (41)

Therefore, as long as η < 1/(Lf + 2Ls), letting m = 0 we deduce

n−1∑t=0

‖x(t+ 1)− x(t)‖2 ≤ 2

1/η − Lf − 2Ls[F(x(0)

)− F

(x(n)

)] ≤ 2

1/η − Lf − 2Ls[F(x(0)

)− inf

zF (z)]. (42)

By Assumption 1.1, F is bounded from below hence the right-hand side is finite and independent of n. Lettingn goes to infinity completes the proof of item 1.

Item 2 follows immediately from item 1 and (27), whence it is clear that for all i the limit points satisfyω(x(k)) = ω(xi(k)).

To prove item 3, let x∗ be a limit point of x(t)t, i.e., there exists a subsequence x(tm) → x∗. Since theobjective function F is closed we know x∗ ∈ domF . To show x∗ ∈ critF we need to exhibit a sequence sayx(km + 1) such that3

x(km + 1)→ x∗, F (x(km + 1))→ F (x∗), 0← u(km + 1) ∈ ∂F (x(km + 1)). (43)

3Technically, from Definition 1 we should have the Frechet subdifferential ∂F in Equation (43), however, a usualdiagonal argument allows us to use the more convenient (limiting) subdifferential.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

This is the most difficult part of the proof, and the argument differs substantially from previous work (e.g. [8]).

We first prove the subdifferential goes to zero. Observe from Assumption 2.3 that the iterations t, t ∈ Ti whenmachine i updates is infinite. Let t ∈ Ti and by the optimality condition of xi(t+ 1) in Equation (29):

− 1η

[xi(t+ 1)− xi(t) + η∇if

(xi(t)

)]∈ ∂gi(xi(t+ 1)), (44)

i.e. there exists ui(t+ 1) ∈ ∂gi(xi(t+ 1)) such that

‖ui(t+ 1) +∇if(x(t+ 1))‖ ≤ ‖ui(t+ 1) +∇if(x(t))‖+ ‖∇if(x(t+ 1))−∇if(x(t))‖ (45)

( Equation (44), Assumption 1.2 ) ≤∥∥∥ 1η (xi(t+ 1)− xi(t) +∇if

(xi(t)

)−∇if

(x(t)

)∥∥∥+ Li‖x(t+ 1)− x(t)‖(46)

( triangle inequality, Assumption 1.2 ) ≤ 1η‖xi(t+ 1)− xi(t)‖+ Li‖xi(t)− x(t)‖+ Li‖x(t+ 1)− x(t)‖ (47)

( Equation (27) ) ≤ 1η‖xi(t+ 1)− xi(t)‖+ Li

t∑k=(t−s)+

‖x(k + 1)− x(k)‖. (48)

We now use a chaining argument to remove the condition t ∈ Ti above. For each t 6∈ Ti let ti be the largest elementin k ≤ t : k ∈ Ti. Thanks to Assumption 2.3 ti always exists and t − ti ≤ s. Therefore, for any t 6∈ Ti, sincexi(t+1) = xi(ti+1) we can certainly choose ui(t+1) ∈ ∂gi(xi(t+1)) to coincide with ui(ti+1) ∈ ∂gi(xi(ti+1)).Then:

‖ui(t+ 1) +∇if(x(t+ 1))− ui(ti + 1)−∇if(x(ti + 1))‖ = ‖∇if(x(t+ 1))−∇if(x(ti + 1))‖ (49)

( triangle inequality ) ≤t∑

k=ti+1

‖∇if(x(k + 1))−∇if(x(k))‖ (50)

( Assumption 2.3 ) ≤t∑

k=(t−s+1)+

‖∇if(x(k + 1))−∇if(x(k))‖ (51)

( Assumption 1.2 ) ≤ Lit∑

k=(t−s+1)+

‖x(k + 1)− x(k)‖. (52)

Combining the two separate cases in Equation (48) and Equation (52) above we have for all t:

‖u(t+ 1) +∇f(x(t+ 1))‖ ≤ (√p/η + 2L)

t∑k=(t−2s)+

‖x(k + 1)− x(k)‖, (53)

where of course u(t + 1) =(u1(t + 1), . . . , up(t + 1)

)∈ ∂g(x(t + 1)) and we artificially introduce

√p for

convenience of subsequent proof. Therefore, from item 2 we deduce

limt→∞

dist∂F (x(t+1))(0)→ 0. (54)

Next we deal with the function value convergence in Equation (43). For any ti ∈ Ti, using Equation (30) withz = x∗i we have

gi(xi(ti + 1)) +1

2η

∥∥xi(ti + 1)− xi(ti) + η∇if(xi(ti)

)∥∥2 ≤ gi(x∗i)

+1

2η

∥∥x∗i − xi(ti) + η∇if(xi(ti)

)∥∥2, (55)

which, after rearrangement, yields

gi(xi(ti + 1)) ≤ gi(x∗i)

+1

2η‖x∗i − xi(ti)‖2 −

1

2η‖xi(ti + 1)− xi(ti)‖2 + 〈x∗i − xi(ti + 1),∇if(xi(ti))〉 (56)

= gi(x∗i)

+1

2η‖x∗i − xi(ti)‖2 −

1

2η‖xi(ti + 1)− xi(ti)‖2 + 〈x∗i − xi(ti + 1),∇if(x∗)〉

+ 〈x∗i − xi(ti + 1),∇if(xi(ti))−∇if(x∗)〉. (57)

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

We wish to deduce from the above inequality that gi(xi(ti + 1)) → g(x∗i ), but we need a uniformization deviceto remove the dependence on i (hence removing the condition ti ∈ Ti). Observing from item 2 that

limm→∞

maxt∈[tm−s,tm+s]

‖x(t+ 1)− x∗‖ → 0, limm→∞

maxt∈[tm−s,tm+s]

‖xi(t+ 1)− x∗‖ → 0. (58)

By Assumption 2.3, [tm − s, tm + s] ∩ Ti 6= ∅ for all i, using item 2 again and the Lipschitz continuity of ∇f , wededuce from Equation (57) that

lim supm→∞

maxt∈[tm−s,tm+s]∩Ti

gi(xi(t+ 1)) ≤ gi(x∗i ). (59)

Since each machine must update at least once on the intervals [tm − s, tm] and [tm, tm + s], let tim be the largestelement of [tm − s, tm] ∩ Ti. Then from the previous inequality we have

lim supm→∞

maxt∈[tim,tm+s]∩Ti

gi(xi(t+ 1)) ≤ gi(x∗i ). (60)

Since gi(xi(t+ 1)) = gi(xi(t)) if t 6∈ Ti and tim ∈ Ti, it follows that

maxt∈[tm,tm+s]

gi(xi(t+ 1)) ≤ maxt∈[tim,tm+s]∩Ti

gi(xi(t+ 1)), (61)

hence

lim supm→∞

maxt∈[tm,tm+s]

gi(xi(t+ 1)) ≤ gi(x∗i ). (62)

Choose any sequence km such that km ∈ [tm, tm + s]. Since x(tm)→ x∗, from item 2 it is clear that

x(km + 1)→ x∗. (63)

From Equation (62) we know for all i, lim supm→∞ gi(xi(km+1)) ≤ gi(x∗i ) while using closedness of the functiongi we have lim infm→∞ gi(xi(km + 1)) ≥ gi(x

∗i ), thus in fact limm→∞ gi(xi(km + 1)) = gi(x

∗i ). Since f is

continuous, we know

limm→∞

F (x(km + 1)) = F (x∗). (64)

Lastly, combining Equation (54), Equation (63) and Equation (64), it follows from Definition 1 that x∗ ∈critF .

B Proof of Theorem 2

Theorem 2 (Finite Length). Let Assumption 1, 2, 3 and 4 hold, and apply msPG to problem (P). If the stepsize η < (Lf + 2Ls)−1 and x(t) is bounded, then∑∞

t=0 ‖x(t+ 1)− x(t)‖ <∞, (11)

∀i = 1, . . . , p,∑∞t=0 ‖xi(t+ 1)− xi(t)‖ <∞. (12)

Furthermore, x(t) and xi(t), i = 1, ..., p, converge to the same critical point of F .

Our proof requires the following simple uniformization of the K L inequality in Definition 4:

Lemma 2 (Uniformized K L inequality, [11, Lemma 6]). Let h be a K L function and Ω ⊂ domh be a compactset. If h is constant on Ω, then there exist ε, λ > 0 and a function ϕ as in Definition 4, such that for all x ∈ Ωand all x ∈ x ∈ Rd : distΩ(x) < ε ∩ [x : h(x) < h(x) < h(x) + λ], one has

ϕ′(h(x)− h(x)) · dist∂h(x)(0) ≥ 1.

The proof of this lemma is the usual covering argument.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Proof of Theorem 2. We first show that if the global sequence has finite length (i.e. (11)) then the local sequencesalso have finite lenggth (i.e. (12)). Indeed,

‖xi(t+ 1)− xi(t)‖ ≤ ‖xi(t+ 1)− x(t+ 1)‖+ ‖x(t+ 1)− x(t)‖+ ‖x(t)− xi(t)‖ (65)

( Equation (27) ) ≤ ‖x(t+ 1)− x(t)‖ +

t∑k=(t+1−s)+

‖x(k + 1)− x(k)‖ +

t−1∑k=(t−s)+

‖x(k + 1)− x(k)‖. (66)

Therefore, summing from t = 0 to t = n:

n∑t=0

‖xi(t+ 1)− xi(t)‖ ≤n∑t=0

t∑k=(t+1−s)+

‖x(k + 1)− x(k)‖ +

t∑k=(t−s)+

‖x(k + 1)− x(k)‖

(67)

≤ (2s+ 1)

n∑t=0

‖x(t+ 1)− x(t)‖. (68)

Letting n tend to infinity we have (11) =⇒ (12).

From Theorem 1 we know the limit points of x(t) and xi(t), i = 1, . . . , p, coincide, and they are criticalpoints of F .

The only thing left to prove is the finite length property of the global sequence x(t). If for all large t wehave x(t + 1) = x(t) then the conclusion is trivial. On the other hand, we can remove all iterations t withx(t + 1) = x(t), without affecting the length of the trajectory. Thus, in the following we assume for all (large)t we have x(t + 1) 6= x(t). Thanks to Assumption 3 and Assumption 1.1, it is then clear that the objectivevalue F (x(t)) is strictly decreasing to a limit F ∗. Since x(t) is assumed to be bounded, the limit point setΩ := ω(x(t)) is nonempty and compact. Obviously for any x∗ ∈ Ω we have F (x∗) = F ∗. Fix any ε > 0,clearly for t sufficiently large we have4 distΩ(x(t)) ≤ ε. We now have all ingredients to apply the uniformizedK L inequality in Lemma 2, which implies that for all sufficiently large t, there exists a continuous and concavefunction ϕ (with additional properties listed in Definition 4) such that

ϕ′(F (x(t))− F ∗

)· dist∂F (x(t))(0) ≥ 1. (69)

Since ϕ is concave, we obtain

∆t,t+1 := ϕ(F (x(t))− F ∗

)− ϕ

(F (x(t+ 1))− F ∗

)≥ ϕ′

(F (x(t))− F ∗

)(F (x(t))− F (x(t+ 1))

)(70)

( Assumption 3 and Equation (69) ) ≥ α‖x(t+ 1)− x(t)‖2

dist∂F (x(t))(0). (71)

It is clear that the function ϕ (composed with F ) serves as a Lyapunov function. To proceed, we need to upperbound the subdifferential ∂F (x(t)), which has been painstakingly dealt with in the proof of Theorem 1.

Using the inequality 2√ab ≤ a+ b for positive numbers we obtain from Equation (71): for t sufficiently large,

2‖x(t+ 1)− x(t)‖ ≤ δα∆t,t+1 + 1

δdist∂F (x(t))(0), (72)

4This is true for any bounded sequence, and we provide a proof for completeness: Suppose not, then there exists ε > 0such that for all n there exists a t ≥ n such that distΩ(x(t)) > ε. Thus, we can extract a subsequence x(tm) such thatdistΩ(x(tm)) > ε. However, since x(t) is bounded, we can extract a further subsequence, say x(tmn), that converges,i.e. distΩ(x(tmn))→ 0, contradiction.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

where δ > 0 will be fixed later. Summing the above inequality over t from m (sufficiently large) to n:

2

n∑t=m

‖x(t+ 1)− x(t)‖ ≤n∑

t=m

δ

α∆t,t+1 +

n∑t=m

1

δdist∂F (x(t))(0) (73)

( telescoping and Equation (53) ) ≤ δ

αϕ(F (x(m))− F ∗

)+

n∑t=m

√p/η + 2L

δ

t∑k=(t−2s)+

‖x(k + 1)− x(k)‖ (74)

≤ δ

αϕ(F (x(m))− F ∗

)+

(2s+ 1)(√p/η + 2L)

δ

m−1∑k=(m−2s)+

‖x(k + 1)− x(k)‖

+(2s+ 1)(

√p/η + 2L)

δ

n∑t=m

‖x(t+ 1)− x(t)‖. (75)

Setting δ = (2s+ 1)(√p/η + 2L) and rearranging:

n∑t=m

‖x(t+ 1)− x(t)‖ ≤(2s+ 1)(

√p/η + 2L)

αϕ(F (x(m))− F ∗

)+

m−1∑k=(m−2s)+

‖x(k + 1)− x(k)‖ (76)

Since the right-hand side is finite and does not depend on n, letting n tend to infinity completes our proof forEquation (11).

C Proof of Lemma 1

Lemma 1. Assume ∀t, i, t ∈ Ti. Let the step size η < ρ−14Cρ

√ρ−1

√ρs+1−1

for any ρ > 1 and all Ui, i = 1, ..., p be

eventually Lipschitz continuous, then the sequences x(t) and xi(t), i = 1, ..., p, have finite length.

Follow the same argument of equation (27), one can bound ‖xi(t)− xi(t+ 1)‖ similarly as

‖xi(t)− xi(t+ 1)‖ ≤t∑

k=(t−s)+

‖x(k + 1)− x(k)‖. (77)

For simplicity we omit the details.

Since Assumption 5 holds for all t > tL, we prove the lemma by considering two complementary cases.

Case 1: There exists a t > tL such that

t∑k=(t−s)+

‖x(k + 1)− x(k)‖ ≤√ρs+1 − 1√ρ− 1

‖x(t+ 1)− x(t)‖

Case 2: For all t > tL case 1 fails.

We will show that case 1 leads to the sufficient decrease property in Assumption 3 for all large t, case 2 leads tothe finite length of the models.

Case 1: t exists.

We start by proving the following lemma.

Lemma 3. With Assumption 5 and the existence of t. Set η−1 > 4Cρρ−1

√ρs+1−1√ρ−1 , then it holds for all t > t that

‖x(t+ 1)− x(t)‖ ≤ √ρ‖x(t+ 2)− x(t+ 1)‖.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Proof. Using the inequality ‖a‖22 − ‖b‖22 ≤ 2‖a‖‖a− b‖, we have for all t > t > tL

‖x(t+ 1)− x(t)‖22 − ‖x(t+ 2)− x(t+ 1)‖22 ≤ 2 ‖x(t+ 1)− x(t)‖ ‖(x(t+ 1)− x(t))− (x(t+ 2)− x(t+ 1))‖

(no skip of update) = 2 ‖x(t+ 1)− x(t)‖

∥∥∥∥∥p∑i=1

Ui(xi(t))−

p∑i=1

Ui(xi(t+ 1))

∥∥∥∥∥≤ 2 ‖x(t+ 1)− x(t)‖

p∑i=1

∥∥Ui(xi(t))− Ui(xi(t+ 1))∥∥

(Assumption 5) ≤ 2 ‖x(t+ 1)− x(t)‖

(p∑i=1

Ciη‖xi(t)− xi(t+ 1)‖

)

(equation (77)) ≤ 2 ‖x(t+ 1)− x(t)‖

p∑i=1

Ciη

t∑k=(t−s)+

‖x(k + 1)− x(k)‖

= 2Cη ‖x(t+ 1)− x(t)‖

t∑k=(t−s)+

‖x(k + 1)− x(k)‖

. (78)

Now we use an induction argument. Since there exists t > tL such that∑tk=(t−s)+ ‖x(k + 1) − x(k)‖ ≤

√ρs+1−1√ρ−1 ‖x(t+ 1)− x(t)‖, then set t = t in the above inequality, we obtain

∥∥x(t+ 1)− x(t)∥∥2

2−∥∥x(t+ 2)− x(t+ 1)

∥∥2

2≤ 2Cη

√ρs+1 − 1√ρ− 1

∥∥x(t+ 1)− x(t)∥∥2

2

(choice of η) ≤(

1− 1

ρ

)∥∥x(t+ 1)− x(t)∥∥2. (79)

After rearranging terms we conclude ‖x(t+1)−x(t)‖ ≤ √ρ‖x(t+2)−x(t+1)‖. Now we assume this relationship

holds up to t (t > t), then (78) becomes

‖x(t+ 1)− x(t)‖22 − ‖x(t+ 2)− x(t+ 1)‖22 ≤ 2Cη

√ρs+1 − 1√ρ− 1

‖x(t+ 1)− x(t)‖22

(choice of η) ≤(

1− 1

ρ

)‖x(t+ 1)− x(t)‖22

we obtain ‖x(t+ 1)− x(t)‖ ≤ √ρ‖x(t+ 2)− x(t+ 1)‖. This completes the lemma.

With this bound, inequality (37) can be further bounded for t > t as

F(x(t+ 1)

)− F

(x(t)

)≤ 1

2 (Lf − 1/η)‖x(t+ 1)− x(t)‖2 + L‖x(t+ 1)− x(t)‖ ·t−1∑

k=(t−s)+

‖x(k + 1)− x(k)‖.

≤ −α‖x(t+ 1)− x(t)‖2, (80)

where

α ≥ η−1 − Lf2

−L√ρ(1−√ρs)1−√ρ

(C > L, ρ > 1) ≥ 2Lρ

ρ− 1

√ρs+1 − 1√ρ− 1

−L(√ρs+1 − 1)√ρ− 1

− Lf2

≥L(√ρs+1 − 1)√ρ− 1

(2ρ

ρ− 1− 1)− Lf

2

(L > Lf , ρ > 1) > 0,

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

This proves the sufficient decrease for all t > t of the objective value. Hence, the finite length property of themodels follows from Theorem 2.

Case 2: t does not exist

In this case we have for all t > tL it holds that∑tk=(t−s)+ ‖x(k + 1) − x(k)‖ ≥

√ρs+1−1√ρ−1 ‖x(t + 1) − x(t)‖. Set

D =√ρs+1−1√ρ−1 and sum the inequality over t from tL to n yields

n∑k=tL

‖x(k + 1)− x(k)‖ < 1

D

n∑t=tL

t∑k=(t−s)+

‖x(k + 1)− x(k)‖

<s+ 1

D

n∑t=(tL−s)+

‖x(t+ 1)− x(t)‖ ,

which after rearranging terms becomes

(1− s+ 1

D)

n∑t=tL

‖x(t+ 1)− x(t)‖ ≤ s+ 1

D

tL−1∑t=(tL−s)+

‖x(t+ 1)− x(t)‖ .

Since D =√ρs+1−1√ρ−1 > s + 1 for ρ > 1 and tL is finite, the right hand side of the above inequality is finite, and

the left hand side has positive coefficient. Thus the above inequality implies

n∑t=0

‖x(t+ 1)− x(t)‖ < +∞.

Enlarge n→∞ gives the finite length property of the global model. By the proof of Appendix B, we know thefinite length of global model implies the finite length of all local models.

D Proof of Example 1

We proof case by case, and the scaled version γg(x), γ > 0 trivially follows from the same argument.

Cases g = 0, g = 12‖ · ‖

2:

When g = 0, the update operator in eq. (9) becomes Ui(xi(t)) = −η∇if(xi(t)), which is ηLi Lipschitz due to

Assumption 1.2 .

When g = 12‖ · ‖

2, the update operator becomes for i = 1, ..., p

Ui(xi(t)) = proxη1

2‖·‖22(xi(t)− η∇if(xi(t)))− xi(t) = − 1

1 + η−1

(xi(t) +∇if(xi(t))

).

With which we have

‖Ui(xi(t+ 1))− Ui(xi(t))‖ ≤1

1 + η−1‖(xi(t+ 1)− xi(t)) + (∇if(xi(t+ 1))−∇if(xi(t)))‖

≤ η(1 + Li)‖xi(t+ 1)− xi(t)‖

Cases g = ‖ · ‖0, ‖ · ‖0 + ‖ · ‖2, ‖ · ‖0,2, ‖ · ‖0,2 + ‖ · ‖2: For the non-overlapping group norms, we assign eachmachine a subset of groups of coordinates.

Consider g = ‖ · ‖0, its proximal map on i-th coordinate can be expressed as

proxηgi(zi) =

zi, if |zi| >

√2η

0, otherwise

The mapping contains a hard threshold, i.e., it filters out those coordinates with magnitude less than√

2η. Thisimplies that any change of the support set of proxηgi(zi) will induce a jump of magnitude of at least

√2η. On

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

the other hand, the second assertion of Theorem 1 imply that limt→∞

‖xi(t + 2) − xi(t + 1)‖ = 0, which by local

update can be expressed as

limt→∞

‖proxηgi(xi(t+ 1)− η∇if(xi(t+ 1)))− proxηgi(xi(t)− η∇if(xi(t)))‖ = 0.

Hence by the above equation and the jump of proximal map, the support Ω of proxηgi(xi(t)− η∇if(xi(t))) (i.e.,xi(t+1)) must remain stable for all t sufficiently large. Moreover, the proximal map reduces to identity operatoron the support set Ω. Thus, for all t sufficiently large we have

‖Ui(xi(t+ 1))− Ui(xi(t))‖ = ‖proxηgi(xi(t+ 1)− η∇if(xi(t+ 1)))− xi(t+ 1)− proxηgi(xi(t)− η∇if(xi(t)))− xi(t)‖(support on Ω) = ‖[proxηgi(xi(t+ 1)− η∇if(xi(t+ 1)))− xi(t+ 1)− proxηgi(xi(t)− η∇if(xi(t)))− xi(t)]Ω‖

(proxηg is identity on Ω) ≤ ‖η∇if(xi(t))− η∇if(xi(t+ 1))‖≤ ηLi‖xi(t+ 1)− xi(t)‖.

Hence the operator is eventually O(η) Lipschitz.

Next we consider (without loss of generality) g = ‖ · ‖0 + λ2 ‖ · ‖

2 where λ > 0. The proximal map on i-thcoordinate is

proxηgi(zi) =

zi, if |zi| >

√2(η + η2λ)

0, otherwise

Thus, the mapping also contains a hard threshold. Following similar argument as previous case, we concludethat the support Ω of proxηgi(xi(t)− η∇if(xi(t))) (i.e., xi(t + 1)) must remain stable for all t sufficiently large,

and the proximal map reduces to identity operator on Ω. Consequently, the operator Ui(xi(t)) is O(η) Lipschitz

for all t large.

The proof of group norms g = ‖ · ‖0,2, ‖ · ‖0,2 + ‖ · ‖2 then follows by realizing that the proximal maps have hardthreshold on group support.

Cases g = ‖ · ‖1, ‖ · ‖1 + ‖ · ‖2 with eventual stable support set of x(t):

For these two cases we assume that the support set of x(t) remains unchanged for all large t.

We just need to consider g = ‖ · ‖1 + λ2 ‖ · ‖

2, λ ≥ 0. Its proximal map on vector zi has the form

proxηg(zi) = 11+ηλ sgn(zi) (|zi| − η)+ .

Since the support set Ω of x(t) (i.e. proxηg(xi(t)− η∇if(xi(t))

)) is assumed to be stable after some tL, the

above soft-thresholding operator ensures that |xi(t)− η∇if(xi(t))|Ω > η for all large t, and we obtain

Ui(xi(t)) = [xi(t+ 1)− xi(t)]Ω = [proxηg

(xi(t)− η∇if(xi(t))

)− xi(t)]Ω

= (1 + ηλ)−1[−η∇if(xi(t))− ηsgn

(xi(t)− η∇if(xi(t))

)]Ω− ηλ

1 + ηλ[xi(t)]Ω (81)

On the other hand, Theorem 1.2 and the Lispchitz gradient of f implies

limt→∞

∥∥[xi(t+ 1)− η∇if(xi(t+ 1))]−[xi(t)− η∇if(xi(t))

]∥∥ = 0.

Then [sgn(xi(t) − η∇if(xi(t))]Ω must eventually remain constant, since otherwise the condition |xi(t) −η∇if(xi(t))|Ω > η will induce a change of |xi(t)−η∇if(xi(t)| to be at least 2η and violate the above asymptoticcondition. In summary, for all large t we have

Ui(xi(t)) = (1 + ηλ)−1

[−η∇if(xi(t))− Const

]Ω− ηλ

1 + ηλ[xi(t)]Ω

which further implies that

‖Ui(xi(t+ 1))− Ui(xi(t))‖ ≤ ‖(1 + ηλ)−1[η∇if(xi(t+ 1))− η∇if(xi(t))

]Ω

+ηλ

1 + ηλ[xi(t+ 1)− xi(t)]Ω‖

≤ ηLi‖xi(t+ 1)− xi(t)‖+ ηλ‖xi(t+ 1)− xi(t)]‖≤ η(Li + λ)‖xi(t+ 1)− xi(t)‖.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

E Proof of Theorem 3

Theorem 3 (Global rate of convergence). If the finite length property in Theorem 2 holds, then

1.∑∞t=0 ‖e(t)‖ <∞;

2. F ( 1t

∑tk=1 x(k))− inf F ≤ O

(t−1).

Proof. For the first assertion, note that for any n:

n∑t=0

‖e(t)‖ = η

n∑t=0

∥∥(∇1f(x1(t))−∇1f(x(t)), . . . , ∇pf(xp(t))−∇pf(x(t)))∥∥(82)

( triangle inequality, Assumption 1.2 ) ≤ ηn∑t=0

p∑i=1

Li‖x(t)− xi(t)‖ (83)

( Equation (27) ) ≤ ηn∑t=0

(p∑i=1

Li

)t−1∑

k=(t−s)+

‖x(k + 1)− x(k)‖

= Lη

n∑t=0

t−1∑k=(t−s)+

‖x(k + 1)− x(k)‖ (84)

≤ Lsηn−1∑t=0

‖x(t+ 1)− x(t)‖ . (85)

Letting n tend to infinity we obtain

∞∑t=0

‖e(t)‖ ≤ Lsη∞∑t=0

‖x(t+ 1)− x(t)‖ <∞. (86)

For the second assertion, we first recall from [30] that the inexact proximal gradient algorithm in Equation (14)has the following bound, provided that F is convex:

F

(1

t

t∑k=1

x(k)

)− F ∗ ≤ (‖x(0)− x∗‖+ 2At)

2

2tη, where At =

t∑k=0

η‖e(k)‖. (87)

The second assertion thus follows from the first one (assuming convexity).

F Experiments Specifications

Specifications for ‖ · ‖0,2 Lasso:

minx∈Rd

1

2‖Ax− b‖22 +

20∑i=1

γiI(‖xi‖).

Here A ∈ R1000×2000,b ∈ R1000, and x ∈ R2000 is divided into 20 equal groups of features. Matrix A isgenerated from N (0, 1) with normalized columns. We set b = Ax + ε, where ε is generated from N (0, 10−2) andx is a normalized vector with 8 non-zero groups of features generated from N (0, 1). For the non-zero groups ofx, we set the corresponding γi = 10−4, and for the remaining groups we set γi = 10−2.

We implement msPG on four cores with each core assigned five group of features. Each core stores the corre-sponding column blocks of A.

Specifications for ‖ · ‖1 Lasso:

minx1

2‖Ax− b‖22 + γ‖x‖1.

Yi Zhou∗, Yaoliang Yu†, Wei Dai†, Yingbin Liang∗, Eric P. Xing†

Data Generation

We generate the data column-wise. Starting from first column, we randomly pick 104 samples to have non-zeroin column 1 and sample each value from Uniform(−1, 1). We normalize it such that the `2-norm of the columnis 1. We denote these values as v1 ∈ Rn. To generate column i, with probability 0.5 we randomly pick a newset of samples to have non-zero values at column i (otherwise we use the same samples from column i− 1). Thissimulates the correlations between each column. Once the samples are chosen, we assign values from Unif(−1, 1).vi is again normalized. We generate ground truth regressor β ∈ Rd from N (0, 1) with 1% non-zero entries, andobtain the regressed value from b = Aβ where A is the design matrix.