Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf ·...
Transcript of Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf ·...
![Page 1: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/1.jpg)
Lorenzo Orecchia Department of Computer Science
Challenges in Optimization for Data Science
Combinatorial meets Convex Leveraging combinatorial structure to obtain faster algorithms, provably
6/30/15 1
![Page 2: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/2.jpg)
A Tale of Two Disciplines
![Page 3: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/3.jpg)
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
![Page 4: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/4.jpg)
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
Numerical and Convex Optimization !!
•• Asks about matrices, convex programs (e.g. LPs, SDPs)
••Questions and solution techniques tend to be continuous/numerical
![Page 5: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/5.jpg)
NEW INSIGHT: Deep connections between core concepts Use techniques from one to help the other
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
Numerical and Convex Optimization !!
•• Asks about matrices, convex programs (e.g. LPs, SDPs)
••Questions and solution techniques tend to be continuous/numerical
![Page 6: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/6.jpg)
NEW INSIGHT: Deep connections between core concepts Use techniques from one to help the other
A Tale of Two Disciplines
Combinatorial Optimization !!
••Asks about paths, flows, cuts, clustering, routing !
••Questions and solutions techniques tend to be discrete/combinatorial
Numerical and Convex Optimization !!
•• Asks about matrices, convex programs (e.g. LPs, SDPs)
••Questions and solution techniques tend to be continuous/numerical
Fast Algorithms for Fundamental Graph Problems
![Page 7: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/7.jpg)
Why Graph Algorithms?
Computer Networks
Transportation Networks
Social Networks
Representations of Physical Objects
![Page 8: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/8.jpg)
Why Fast Graph Algorithms• Classical Algorithms (1970s-‐1990s): Standard notion of efficiency is polynomial running time • Today: Graphs of interest are getting larger and larger … Even quadratic running time is unfeasibly large for many instances
!!!!
![Page 9: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/9.jpg)
Why Fast Graph Algorithms
![Page 10: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/10.jpg)
Why Fast Graph Algorithms• Classical Algorithms (1970s-‐1990s): Standard notion of efficiency is polynomial running time • Today: Graphs of interest are getting larger and larger … Even quadratic running time is unfeasibly large for many instances
!• New Efficiency Requirement: as close as possible to linear
NEARLY-‐LINEAR RUNNING TIME
!!!
Input Size Running Time
![Page 11: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/11.jpg)
Why Fast Graph Algorithms• Classical Algorithms (1970s-‐1990s): Standard notion of efficiency is polynomial running time • Today: Graphs of interest are getting larger and larger … Even quadratic running time is unfeasibly large for many instances
!• New Efficiency Requirement: as close as possible to linear
NEARLY-‐LINEAR RUNNING TIME
!!!
Input Size Running Time
![Page 12: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/12.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
![Page 13: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/13.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
REACHABILITY PROBLEM: Is there a path between vertices u and v in the graph G?
u v
![Page 14: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/14.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability
REACHABILITY PROBLEM: Is there a path between vertices u and v in the graph G?
u v
YES, by depth-‐first search
![Page 15: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/15.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability
YES, by depth-‐first search
SHORTEST PATH PROBLEM: What is the shortest path from u to v?
u
v
![Page 16: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/16.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
ReachabilityShortest Path
YES, by depth-‐first search
SHORTEST PATH PROBLEM: What is the shortest path from u to v?
u
v
YES, by breadth-‐first search
![Page 17: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/17.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
ReachabilityShortest PathConnectivity
Minimum Cost Spanning Tree…
![Page 18: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/18.jpg)
GOAL: Build library of primitives running in almost-‐linear time • What can you do to a graph in nearly-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
ReachabilityShortest PathConnectivity
Minimum Cost Spanning Tree…
Probe Graph Structure in Simple Way
![Page 19: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/19.jpg)
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
ELECTRICAL FLOW PROBLEM: What is the current flow when one unit of current enters s and one leaves t?
1 1
![Page 20: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/20.jpg)
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
Laplacian Systems of Linear Equations [Spielman, Teng’04]
ELECTRICAL FLOW PROBLEM: What is the current flow when one unit of current enters s and one leaves t?
1 1
![Page 21: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/21.jpg)
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
Laplacian Systems of Linear Equations [Spielman, Teng’04]
![Page 22: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/22.jpg)
• What can you do to a graph in almost-‐linear time?
Nearly-‐Linear-‐Time Algorithms
Nearly-‐Linear Time
Reachability, Shortest Path, Connectivity Minimum Cost Spanning Tree, …
Laplacian Systems of Linear Equations [Spielman, Teng’04]
Undirected Flow Problems • s-‐t Maximum Flow [Kelner, Orecchia, Sidford, Lee ‘13][Sherman’13] • Concurrent Multi-‐commodity Flow [Kelner, Orecchia, Sidford, Lee ’13] • Oblivious Routing [Kelner, Orecchia, Sidford, Lee ’13] !… and Undirected Cut Problems • Minimum s-‐t cut [Kelner, Orecchia, Sidford, Lee ‘13][Sherman’13] • Approximate Sparsest Cut [Kelner, Orecchia, Sidford, Lee ’13] [Sherman’13] • Approximate Minimum Conductance Cut [Orecchia, Sachdeva, Vishnoi ’12]
![Page 23: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/23.jpg)
A Faster, Simpler Laplacian Solver
+8 +8
![Page 24: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/24.jpg)
A Faster, Simpler Laplacian Solver
+8 +8
INITIALIZATION:• Choose a spanning tree T of G.
![Page 25: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/25.jpg)
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION:• Choose a spanning tree T of G.• Route flow along T to obtain initial flow f0.
88
8
![Page 26: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/26.jpg)
A Faster, Simpler Laplacian Solver
+8 +88
INITIALIZATION:• Choose a spanning tree T of G.• Route flow along T to obtain initial flow f0.
88
8
0
0
0
0 0
0
![Page 27: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/27.jpg)
+8 +88
88
8
0
0
0
0 0
0
A Faster, Simpler Laplacian Solver
![Page 28: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/28.jpg)
+8 +88
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
!!
88
8
0
0
0
0 0
0
A Faster, Simpler Laplacian Solver
![Page 29: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/29.jpg)
+8 +88
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
!!
88
8
0
81624
32
0
0
0
0 0
0
A Faster, Simpler Laplacian Solver
![Page 30: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/30.jpg)
+8 +88
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
!!
88
8
0
81624
32
24 24
24
0
0
0
0 0
032
A Faster, Simpler Laplacian Solver
![Page 31: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/31.jpg)
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
• Check if voltages and flows obey Ohm’s Law on off-‐tree edges. !!
![Page 32: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/32.jpg)
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
• Check if voltages and flows obey Ohm’s Law on off-‐tree edges. !!
![Page 33: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/33.jpg)
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
NOTE: Flow f0 is electrical flow if and only if Ohm’s Law holds for all edges e=(b,a) !• Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. !
• Check if voltages and flows obey Ohm’s Law on off-‐tree edges. !!
FAIL
FAIL
![Page 34: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/34.jpg)
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032
![Page 35: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/35.jpg)
Our Fastest, Simplest Laplacian Solver
+8 +88
88
8
0
81624
32
24 24
24
0
0
0
0 0
032INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e
![Page 36: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/36.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e
Our Fastest, Simplest Laplacian Solver
+8 +8
8
88
8
0
0
0
0 0
0
![Page 37: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/37.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e
Our Fastest, Simplest Laplacian Solver
+8 +8
8
88
8
0
0
0
0 0
0
![Page 38: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/38.jpg)
Our Fastest, Simplest Laplacian Solver
+8 +8
8
88
8
0
0
0
0 0
0
![Page 39: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/39.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
Our Fastest, Simplest Laplacian Solver
+8 +8
88
8
0
20/3
4/3
4/3 4/3
-‐4/3
-‐4/3
![Page 40: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/40.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
![Page 41: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/41.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
![Page 42: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/42.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
![Page 43: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/43.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
OK
FAIL
![Page 44: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/44.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
0
81624
92/3
84/3 80/388/3
76/3
OK
FAIL
![Page 45: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/45.jpg)
INITIALIZATION: • Choose a spanning tree T of G. • Route flow along T to obtain initial flow f0. !
MAIN LOOP (CYCLE FIXING): • Apply Ohm’s Law to the spanning-‐tree edges to deduce voltages. • Check if voltages and flows obey Ohm’s Law on off-‐tree edges. • Consider cycle corresponding to failing off-‐tree edge e • Send flow around cycle until Ohm’s Law is satisfied on e • Repeat
4/3
-‐4/3
Our Fastest, Simplest Laplacian Solver
+8
+8
88
8
0
20/3
4/3
4/3
-‐4/3
![Page 46: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/46.jpg)
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
![Page 47: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/47.jpg)
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
NB: Our iterative solutions never leave this subspace thanks to cycle updates !
f1
Cycle update on C1
![Page 48: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/48.jpg)
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
NB: Our iterative solutions never leave this subspace thanks to cycle updates !
f1
f2
Cycle update on C2
![Page 49: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/49.jpg)
Working in the Space of Cycles
16
f0
f*
Subspace of flows routing required current input/output
START: Geometric interpretation of electrical flow algorithm
Initial Flow
Electrical Flow
NB: Our iterative solutions never leave this subspace thanks to cycle updates !
f1
f2
Cycle update on C2
Linear combination of cycles
![Page 50: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/50.jpg)
Coordinate Descent in the Space of Cycles
17
GOAL:
f0 f*
Electrical Flow
![Page 51: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/51.jpg)
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles
f*
Electrical Flow
![Page 52: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/52.jpg)
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles• Fix a coordinate at the time, i.e., coordinate descent
f*
Electrical Flow
f1
![Page 53: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/53.jpg)
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles• Fix a coordinate at the time, i.e., coordinate descent
f*
Electrical Flow
f1
f2
![Page 54: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/54.jpg)
Coordinate Descent in the Space of Cycles
17
GOAL:
f0
• Pick a basis of the space of cycles• Fix a coordinate at the time, i.e., coordinate descent
f*
Electrical Flow
f1
f2
![Page 55: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/55.jpg)
Coordinate Descent in the Space of Cycles
18
GOAL:
f0
EXAMPLE: Ill-‐conditioned basis yields slow convergence
f*
Electrical Flow
![Page 56: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/56.jpg)
Coordinate Descent in the Space of Cycles
18
GOAL:
f0
EXAMPLE: Ill-‐conditioned basis yields slow convergence
f*
Electrical Flow
![Page 57: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/57.jpg)
Low-‐Stretch Spanning TreesG
![Page 58: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/58.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
![Page 59: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/59.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
e
![Page 60: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/60.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
e
Stretch of e =
Length of cycle Ce
st (e) = 5
![Page 61: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/61.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
e
Stretch of e =
Length of cycle Ce
![Page 62: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/62.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
e
Stretch of e =
Length of cycle Ce
![Page 63: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/63.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
e
Average Stretch:
Stretch of e =
Length of cycle Ce
![Page 64: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/64.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
Fact [Abraham, Neiman ’12]: It is possible to compute a spanning tree with average stretch O(log n loglog n) in almost-‐linear time.
Average Stretch:
Stretch of e =
Length of cycle Ce
![Page 65: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/65.jpg)
Low-‐Stretch Spanning TreesGspanning tree T
Fact [Abraham, Neiman ’12]: It is possible to compute a spanning tree with average stretch O(log n loglog n) in almost-‐linear time.
Average Stretch:
Stretch of e =
Length of cycle Ce
![Page 66: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/66.jpg)
Electrical Flow: Algorithmic Components
24
Continuous Optimization:
Randomized Coordinate Descent ITERATIVE METHOD
![Page 67: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/67.jpg)
Electrical Flow: Algorithmic Components
24
Continuous Optimization:
Randomized Coordinate Descent
PROBLEM REPRESENTATION
ITERATIVE METHOD
Combinatorial Optimization:
• Space of cycles
• Basis given by Low-‐Stretch Spanning Tree
![Page 68: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/68.jpg)
Electrical Flow: Algorithmic Components
24
Continuous Optimization:
Randomized Coordinate Descent
PROBLEM REPRESENTATION
ITERATIVE METHOD
Combinatorial Optimization:
• Space of cycles
• Basis given by Low-‐Stretch Spanning Tree
Joint Design of Components
![Page 69: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/69.jpg)
A Framework for Algorithmic Design
Leverage Continuous Optimization Ideas Fast convergence requires smooth problem
ITERATIVE METHOD
PROBLEM REPRESENTATION
Not all representations are created equal: Use combinatorial techniques to produce a smooth representation
![Page 70: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/70.jpg)
A Framework for Algorithmic Design
Leverage Continuous Optimization Ideas Fast convergence requires smooth problem
ITERATIVE METHOD
PROBLEM REPRESENTATION
Not all representations are created equal: Use combinatorial techniques to produce a smooth representation
Efficiency and simplicity rely on combining these two components in the right way
![Page 71: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/71.jpg)
Connection with Electrical Flow
312
2
43
2
1
2
1
s-‐t Maximum Flow
Congestion Minimization
![Page 72: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/72.jpg)
Connection with Electrical Flow
312
2
43
2
1
2
1
Electrical Flow s-‐t Maximum Flow
Energy Minimization Congestion Minimization
![Page 73: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/73.jpg)
Connection with Electrical Flow
312
2
43
2
1
2
1
Electrical Flow s-‐t Maximum Flow
Set resistances as:
![Page 74: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/74.jpg)
An Extra DifficultyObjective function:
![Page 75: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/75.jpg)
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
![Page 76: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/76.jpg)
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
![Page 77: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/77.jpg)
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
![Page 78: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/78.jpg)
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
![Page 79: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/79.jpg)
An Extra Difficulty
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
Objective function:
![Page 80: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/80.jpg)
An Extra Step: RegularizationObjective function:
![Page 81: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/81.jpg)
An Extra Step: Regularization
SOLUTION: Change objective Find function that is close to objective but somewhat smooth
Objective function:
![Page 82: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/82.jpg)
An Extra Step: Regularization
SOLUTION: Change objective Find function that is close to objective but somewhat smooth
Objective function:
![Page 83: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/83.jpg)
An Extra Step: Regularization
SOLUTION: Change objective Find function that is close to objective but somewhat smooth
Objective function:
PROBLEM: OBJECTIVE IS EXTREMELY NON-‐SMOOTH
![Page 84: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/84.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed Regularize to softmax
![Page 85: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/85.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
![Page 86: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/86.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Which basis to use?Little interference in
![Page 87: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/87.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Which basis to use?Little interference in
Surprising equivalence: Basis is
OBLIVIOUS ROUTING SCHEME
![Page 88: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/88.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed
Use basis given byLow-‐stretch spanning tree
Regularize to softmax
Which basis to use?Little interference in
Surprising equivalence: Basis is
OBLIVIOUS ROUTING SCHEME
![Page 89: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/89.jpg)
Oblivious Routing
![Page 90: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/90.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
![Page 91: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/91.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
![Page 92: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/92.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
![Page 93: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/93.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
![Page 94: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/94.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
![Page 95: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/95.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
![Page 96: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/96.jpg)
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
Oblivious Routing
Routing = Probability Distribution over Paths = Flow
![Page 97: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/97.jpg)
Oblivious RoutingGOAL: Route traffic between many pairs of users on the Internet
Minimize maximum congestion of a link DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
![Page 98: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/98.jpg)
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
![Page 99: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/99.jpg)
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
![Page 100: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/100.jpg)
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
PRE-‐PREPROCESSING: Routes are pre-‐computed
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
![Page 101: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/101.jpg)
Oblivious Routing
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests
PRE-‐PREPROCESSING: Routes are pre-‐computed
MEASURE OF PERFORMANCE: Worst-‐case ratio between congestion
of oblivious-‐routing and optimal a posteriori routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
COMPETITIVE RATIO
![Page 102: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/102.jpg)
Oblivious Routing: A New Scheme
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests !!PRE-‐PREPROCESSING: Routes are pre-‐computed !!MEASURE OF PERFORMANCE:
Worst-‐case ratio between congestion of oblivious-‐routing and optimal a posteriori routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
COMPETITIVE RATIO
![Page 103: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/103.jpg)
Oblivious Routing: A New Scheme
SOLUTION: Oblivious Routing: Every request is routed obliviously of the other requests !!PRE-‐PREPROCESSING: Routes are pre-‐computed !!MEASURE OF PERFORMANCE:
Worst-‐case ratio between congestion of oblivious-‐routing and optimal a posteriori routing
GOAL: Route traffic between many pairs of users on the Internet Minimize maximum congestion of a link
DIFFICULTY: Requests arrive online in arbitrary order How to avoid global flow computation at every new arrival
SUBLINEAR COMPETITIVE RATIO
NEARLY-‐LINEAR RUNNING TIME
![Page 104: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/104.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Non-‐Euclidean Gradient Descent
![Page 105: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/105.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Coordinate Descent ?Non-‐Euclidean Gradient Descent
![Page 106: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/106.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 107: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/107.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 108: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/108.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 109: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/109.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 110: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/110.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 111: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/111.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 112: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/112.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 113: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/113.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 114: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/114.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 115: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/115.jpg)
Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 116: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/116.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 117: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/117.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 118: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/118.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 119: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/119.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 120: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/120.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 121: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/121.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 122: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/122.jpg)
Non-‐Euclidean Gradient Descent
Contour map of over feasible subspace
![Page 123: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/123.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Coordinate Descent ?Non-‐Euclidean Gradient Descent
![Page 124: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/124.jpg)
Applying the Framework: Comparison
ITERATIVE METHOD
Electrical Flow s-‐t Maximum Flow
OBJECTIVE
PROBLEM REPRESENTATION
No regularization needed !Use basis given by low-‐stretch spanning tree
Regularize to softmax !Use basis given by oblivious routing scheme
Coordinate Descent ?Non-‐Euclidean Gradient Descent
NEARLY-‐LINEAR-‐TIME FOR BOTH PROBLEMS
![Page 125: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/125.jpg)
Where Do We Go From Here? Future Directions and Open Problems
![Page 126: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/126.jpg)
A New Algorithmic Approach• A novel design framework
for fast graph algorithms !
• Incorporates and leverages idea from multiple fields !
• Based on radically different approach !
• Has yielded conceptually simple, powerful algorithms !
• Combinatorial insight plays a crucial role -‐ Low-‐stretch spanning trees -‐ Oblivious routings !
• Numerous potential applications in Algorithms and other fields
Combinatorial Optimization !!
Numerical Scientific Computing !!ITERATIVE METHOD
PROBLEM REPRESENTATION
![Page 127: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/127.jpg)
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
![Page 128: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/128.jpg)
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
![Page 129: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/129.jpg)
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
?
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
![Page 130: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/130.jpg)
Limits of Nearly-‐Linear Time? Almost-‐Linear Time Super-‐linear Time
Reachability Shortest Path Connectivity
Minimum Cost Spanning Tree …
Directed Flow Problems Directed Cut Problems
!All-‐pair Shortest Path Network Design
…
?
Laplacian Systems of Linear Equations
Undirected Flow Problems: • s-‐t Maximum Flow • Concurrent Multi-‐commodity Flow • Oblivious Routing !
Undirected Cut Problems: • Minimum s-‐t cut • Approximate Sparsest Cut • Approximate Minimum Conductance Cut
Recent Partial Progress: -‐ Improved running time for directed flow problems [Madry’13] !
-‐ Conditional lowerbounds for All-‐Pair Shortest Path [Williams’13]
![Page 131: Combinatorial’meets’Convex - sorbonne-universiteplc/data2015/orecchia2015.pdf · LorenzoOrecchia’ Department’of’Computer’Science’ Challenges)in)Optimization)for)DataScience](https://reader033.fdocuments.us/reader033/viewer/2022050411/5f87c91a784ace1e4700afcc/html5/thumbnails/131.jpg)
Challenges in Optimization for Data Science
Recent Developments Beyond Graph Algorithms
• Interpretation of Nesterov’s Acceleration as Gradient + Mirror Descent [Allen-‐Zhu, O ’14]
APPLICATIONS: Faster Approximate Solver for Packing Linear Programs [Allen-‐Zhu, O ’15] Faster Parallel Algorithms for Positive Linear Programs [Allen-‐Zhu, O ’14] !• Coordinate Descent on Non-‐smooth Objectives to Construct Sparse Objects APPLICATIONS: Fast Graph and Matrix Sparsification [Allen-‐Zhu, Liao, O ’15] !!• Many more from computational complexity to quantum computing
7/2/15 42