PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM...
-
Upload
cameron-harris -
Category
Documents
-
view
214 -
download
0
Transcript of PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM...
![Page 1: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/1.jpg)
PROJECT MANAGER: YOUNGHOON JEONSYSTEM ARCHITECT: YOUNGHOON JUNG
LANGUAGE GURU: J INHYUNG PARKSYSTEM INTEGRATOR: WONJOON SONG
VAL IDAT ION AND TEST ING: AKSHAI SARMA
MIPLMINING-INTEGRATED
PROGRAMMING LANGUAGE
Team 25
![Page 2: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/2.jpg)
DATA MINING
• HOT Trend
• + Big Data
• Mostly Implemented in Matrix Operations
C4.5PageRank
The k-Means AlgorithmSupport Vector Machines
Expectation-MaximizationAdaBoost
K-Nearest Neighbor ClassificationNaïve Bayes
CART
How to Parallelize?How to Port?
![Page 3: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/3.jpg)
WHAT DOES MIPL PROVIDE?
• Easy Data Mining Implementation• Matrix Operations
• Easiest Data Mining Usage• Fact, Rule, and Query
• Automatic Parallelization / Acceleration
• Convenient Interfaces in 3 modes
![Page 4: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/4.jpg)
PROJECT STATISTICS
• 14K LOC over 96 files• Total 356 commits
2/22
2/25
2/28 3/
23/
53/
83/
113/
143/
173/
203/
233/
263/
29 4/1
4/4
4/7
4/10
4/13
4/16
4/19
4/22
4/25
4/28 5/
10
2000
4000
6000
8000
10000
12000
14000
0
50
100
150
200
250
300
350
400
LOCCOMMITL
OC
![Page 5: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/5.jpg)
PROJECT LOG
• PROTOTYPE [3/28]basic FRQ, matrix op on local machines
• 1st RELEASE [4/4]matrix op over Hadoop, built-in matrix
support• 2nd RELEASE [4/11]
job support• 3rd RELEASE [4/18]
command line options, configuration• FINAL RELEASE [4/25]
interpreter support
![Page 6: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/6.jpg)
PROJECT TIMELINE
Dec-30-1899 Sep-08-1913 May-18-1927 Jan-24-1941 Oct-03-1954 Jun-11-1968 Feb-18-1982 Oct-28-1995 Jul-06-2009 Mar-15-2023
15
10
20
-5
-15
-10
55
10
-19
-15
-21
15
10
![Page 7: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/7.jpg)
MIPL COMPILER’S THREE MODES
CompilerMode
InteractiveMode
InterpreterMode
![Page 8: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/8.jpg)
MIPL COMPILER ARCHITECTURE
![Page 9: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/9.jpg)
LINGUISTIC CHARACTERISTICS
• Logical Programming Language
• Imperative Programming Language
• Automatic Conversion b/w Facts and a Matrix
• Multiple Returns
• Weak-typed
• Inclusion, Recursive Calls, Matrix Operations Support
![Page 10: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/10.jpg)
USED TECHNOLOGIES
• Java• Our compiler is written in Java
• Byacc/J• Parser Generator
• BCEL• To generate Java Byte Code
• Ant• Build Automation
• Junit• Unit Testing
![Page 11: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/11.jpg)
LANGUAGE GRAMMAR
• Fact, Rule, and Query (FRQ)• Compatible to Prolog Basic Syntax
• Fact• A fact is a predicate expression that makes a declarative
statement about the problem domain.
• Rule• A rule is a predicate expression that uses logical
implication to describe a relationship among facts.
• Query• A query is terminated with a ”?”. The MIPL language
responds to queries about the facts and rules.
![Page 12: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/12.jpg)
LANGUAGE GRAMMAR
• Fact, Rule, and Query Example
cat(tom). # factcat(foo). # factcat(tom)? # query -> truecat(X) ? # query -> tom, fooanimal(X) <- cat(X). # ruleanimal(tom) ? # trueanimal(jane) ? # false
![Page 13: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/13.jpg)
LANGUAGE GRAMMAR
• Job
• Like Function in C
• Supports parallel running
• Supports Multi-return
• Can be accelerated with the GPU
![Page 14: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/14.jpg)
CLASSIFICATION EXAMPLE
job classify(A, M, Ca, Cb, Cc) { B = A - urow(M). # Built-in Function urow B = B./abs(B). # Built-in Function abs
Ba = B * Ca. # Getting each column Bb = B * Cb. Bc = B * Cc.
R = (Ba - 1)/2 + (Ba + 1)/2 .* Bb. # Classification Formular R = R/2 + Bc.
@R. # Return the result}
![Page 15: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/15.jpg)
CLASSIFICATION EXAMPLE
# To create the identity matrixca(1). cb(0). cc(0).ca(0). cb(1). cc(0).ca(0). cb(0). cc(1).
# Temperature, Rain(1 = No Rain, 0 = Rain),# Girl Friend(1 = is coming, 0 = is not coming)a(60, 1, 0). # Temperature 60, No Rain, No Girla(60, 1, 1). # Temperature 60, No Rain, Girl! Yay!a(-40, 0, 0). # Temperature -40, Rain, No Girla(40, 1, 1). # Temperature 40, No Rain, Girl
# Coefficients for the classification formulam(50, 0.5, 0.5).
![Page 16: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/16.jpg)
MAPREDUCEPLAN
![Page 17: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/17.jpg)
MATRIX OPERATION IN MAPREDUCE
![Page 18: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/18.jpg)
MATRIX OPERATION IN MAPREDUCE
![Page 19: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/19.jpg)
TEST PLAN
The MIPL test plan : conceived at design
Sample input programs already written : test driven development. Tests as important as source
Iterative development withintegrations
Build process : automated testing
![Page 20: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/20.jpg)
TEST PLAN : UNIT TESTS
Core functionality of modules
60+ Unit Tests for modules
Written in JUnit (1-1 source).Ant used to run on build
Test failure = build failure => Repository clean
![Page 21: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/21.jpg)
TEST PLAN : REGRESSION TESTS
Interplay between modules& Test Driven DevelopmentSample programs : 17
Full top-down testing of compiler from source to execution
Critical during integrations
Used in build when code-base was young
![Page 22: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/22.jpg)
TEST PLAN : VALIDATION
Weekly top-down complete integrations of work
Partners in Code : Code Inspections. Design time decision
Coding Style : Long way toward writing less error prone code and extremely helpful in debugging
![Page 23: PROJECT MANAGER: YOUNGHOON JEON SYSTEM ARCHITECT: YOUNGHOON JUNG LANGUAGE GURU: JINHYUNG PARK SYSTEM INTEGRATOR: WONJOON SONG VALIDATION AND TESTING: AKSHAI.](https://reader035.fdocuments.us/reader035/viewer/2022062516/56649dca5503460f94ac11b4/html5/thumbnails/23.jpg)
CONCLUSIONS
What we learned: - Team work, Communication, Technical Skills, …
What worked well: - Modularization, Test Driven Development, ..
What we could have done differently- Bison
Why use MIPL?- Why not?