Apache SystemML - Declarative Large-Scale Machine Learning
-
Upload
romeo-kienzler -
Category
Technology
-
view
82 -
download
0
Transcript of Apache SystemML - Declarative Large-Scale Machine Learning
![Page 1: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/1.jpg)
Apache SystemML - Declarative Large-Scale
Machine LearningRomeo Kienzler (IBM Waston IoT)
Berthold Reinwald (IBM Almaden Research Center) Frederick R. Reiss (IBM Almaden Research Center)
Matthias Rieke (IBM Analytics)
Swiss Data Science Conference 16 - ZHAW - Winterthur
![Page 2: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/2.jpg)
–Assembler vs. Python?
“High-level programming”
![Page 3: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/3.jpg)
Why another lib?• Custom machine learning algorithms
• Declarative ML
• Transparent distribution on data-parallel framework
• Scale-up
• Scale-out
• Cost-based optimiser generates low level execution plans
![Page 4: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/4.jpg)
Why on Spark?• Unification of SQL, Graph, Stream, ML
• Common RDD structure
• General DAG execution engine
• lazy evaluation
• distributed in-memory caching
![Page 5: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/5.jpg)
2009 2008 2007
2007-2008: Multiple projects at IBM Research – Almaden involving machine learning on Hadoop.
2010
2009-2010: Through engagements with customers, we observe how data scientists create ML solutions.
2009: We form a dedicated team for scalable ML
![Page 6: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/6.jpg)
2014 2013 2012 2011
Research
![Page 7: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/7.jpg)
2016 2015
June 2015: IBM Announces open-source SystemML
September 2015: Code available on Github
November 2015: SystemML enters Apache incubation
June 2016: Second Apache release (0.10)
February 2016: First release (0.9) of Apache SystemML
![Page 8: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/8.jpg)
SystemML at
Moved from Hadoop MapReduce to Spark SystemML supports both frameworks Exact same code 300X faster on 1/40th as many nodes
![Page 9: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/9.jpg)
R or Python
Data Scientist
Results
Systems Programmer
Scala
![Page 10: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/10.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Alternating Least Squares
![Page 11: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/11.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Alternating Least Squares
![Page 12: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/12.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Alternating Least Squares
![Page 13: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/13.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Products Factor
Cus
tom
ers
Fact
or
![Page 14: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/14.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Products Factor
Cus
tom
ers
Fact
or
![Page 15: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/15.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Products Factor
Cus
tom
ers
Fact
or
![Page 16: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/16.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Products Factor
Cus
tom
ers
Fact
or
Multiply these two factors to
produce a less-sparse matrix.
×
![Page 17: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/17.jpg)
Products
Cus
tom
ers
i
j Customer i
bought product j.
Products Factor
Cus
tom
ers
Fact
or
Multiply these two factors to
produce a less-sparse matrix.
×
New nonzero values become
product suggestions.
![Page 18: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/18.jpg)
![Page 19: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/19.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
![Page 20: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/20.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
![Page 21: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/21.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
![Page 22: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/22.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
![Page 23: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/23.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
![Page 24: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/24.jpg)
Every line has a clear purpose!
![Page 25: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/25.jpg)
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/
recommendation/ALS.scala
![Page 26: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/26.jpg)
25 lines’ worth of algorithm…
…mixed with 800 lines of performance code
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/
recommendation/ALS.scala
![Page 27: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/27.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
![Page 28: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/28.jpg)
U"="rand(nrow(X),"r,"min"="01.0,"max"="1.0);""V"="rand(r,"ncol(X),"min"="01.0,"max"="1.0);""while(i"<"mi)"{""""i"="i"+"1;"ii"="1;""""if"(is_U)"""""""G"="(W"*"(U"%*%"V"0"X))"%*%"t(V)"+"lambda"*"U;""""else"""""""G"="t(U)"%*%"(W"*"(U"%*%"V"0"X))"+"lambda"*"V;""""norm_G2"="sum(G"^"2);"norm_R2"="norm_G2;""""""""R"="0G;"S"="R;""""while(norm_R2">"10E09"*"norm_G2"&"ii"<="mii)"{""""""if"(is_U)"{""""""""HS"="(W"*"(S"%*%"V))"%*%"t(V)"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""U"="U"+"alpha"*"S;""""""""}"else"{""""""""HS"="t(U)"%*%"(W"*"(U"%*%"S))"+"lambda"*"S;""""""""alpha"="norm_R2"/"sum"(S"*"HS);""""""""V"="V"+"alpha"*"S;""""""""}""""""R"="R"0"alpha"*"HS;""""""old_norm_R2"="norm_R2;"norm_R2"="sum(R"^"2);""""""S"="R"+"(norm_R2"/"old_norm_R2)"*"S;""""""ii"="ii"+"1;""""}""""""is_U"="!"is_U;"}"
SystemML: compile and run at scale no performance code needed!
![Page 29: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/29.jpg)
0
5000
10000
15000
20000
1.2GB (sparse binary)
12GB 120GB
Run
ning
Tim
e (s
ec)
R MLLib SystemML
>24h >24h
OO
M
OO
M
![Page 30: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/30.jpg)
Architecture
SystemML Optimizer
High-Level Algorithm
Parallel Spark
Program
![Page 31: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/31.jpg)
Architecture
High-Level Operations (HOPs) General representation of statements in the data analysis language
Low-Level Operations (LOPs) General representation of operations in the runtime framework
High-level language front-ends
Multiple execution environments
Cost Based
Optimizer
![Page 32: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/32.jpg)
![Page 33: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/33.jpg)
W
S
U
U × S * ( ( t(U) t(U)×(W*(U×S)))(
×
Large dense intermediate
Can compute directly from U,
S, and W!
t(U)(%*%((W(*((U(%*%(S))(
wdivmm
W U S 1.2GB sparse800MB
dense
800MB dense
800MB dense
(weighted divide matrix multiplication)
![Page 34: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/34.jpg)
%*%
W U S
* t()
%*%
1.2GB sparse
80GB dense
80GB dense
800MB dense
800MB dense
800MB dense
800MB dense
All operands fit into heap ! use one
node
WDivMM
(MapWDivMM)
![Page 35: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/35.jpg)
![Page 36: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/36.jpg)
Browse the source!
![Page 37: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/37.jpg)
Browse the source!
Try out some
tutorials!
![Page 38: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/38.jpg)
Browse the source!
Try out some
tutorials! Contribute to the project!
![Page 39: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/39.jpg)
Browse the source!
Try out some
tutorials! Contribute to the project! Download the
binary release!
![Page 40: Apache SystemML - Declarative Large-Scale Machine Learning](https://reader033.fdocuments.us/reader033/viewer/2022042722/58a2ebbe1a28abc9648b698d/html5/thumbnails/40.jpg)
Demo