Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill...
Transcript of Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill...
![Page 1: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/1.jpg)
Shashank Srikant, Rohit Takhar, Vishal Venugopal, Varun Aggarwal
Aspiring Minds
CACM Special Section Workshop – India Region
Skill Evaluation as a Computer Science Question: Opportunities and Challenges
![Page 2: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/2.jpg)
http://www.aspiringminds.com
Opportunities and challenges
• Large number of students graduating every year, huge variation in quality, no signals – 3.5 million students
enrolled in 2017-18
• Capacity issues - Dearth of qualified teachers in universities
– Automatic evaluation/feedback systems
• Services economy, Huge Hiring Numbers, Need for
standardization – Need automation in recruitment
![Page 3: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/3.jpg)
http://www.aspiringminds.com
We conduct standardized computer based
assessment to judge ‘employability’
AMCAT GRE/SAT for Jobs
3 million assessments annually, 3000+ companies
![Page 4: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/4.jpg)
http://www.aspiringminds.com
A model to predict the logical correctness of a program, given the control and data dependencies it possesses
Grading programs
Automata – Automatic program evaluation engine
Machine Learning based
scoring engine Evaluation of programming best
practices Asymptotic complexity
evaluation
Lint-styled rule-based system to detect programs not following programming best practices.
measures the run-time of the code for various input sizes and empirically derives the complexity
KDD 2014: A system to grade computer programming skills using machine learning - Shashank Srikant, Varun Aggarwal
![Page 5: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/5.jpg)
http://www.aspiringminds.com
CONTROL FEATURES – COUNTS
Counts of control-related keywords/tokens
E.g. count(for) = 2
count(for-in-for) = 1 count(while) = 0
Control-context of these keywords
- The Print command as loop(loop(print)))
for(i =1 ; i<=N; i++){
print newline;
count = i;
CONTROL FLOW GRAPH
i = 1
i <= N
i++
j < i
count = i j = 0
print(count) count++
j++
END
Loop 1
Loop 2
Parent scope
for(j=0; j<i; j++)
print count; count++;
void print(int N){
}
}
TARGET PROGRAM
![Page 6: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/6.jpg)
http://www.aspiringminds.com
DATA OPERATION FEATURES IN CONTROL-CONTEXT
Counts of data-related tokens in context of the control structure
E.g. count(block1 :loop(loop(++))) = 2
count(block1 :loop(loop_cond(<))) = 1
Capture control-context of data-dependencies in groups of expressions
- i++ j < i : var (i) related to var (j) : appearing in a loop(loop_cond)
previously incremented : appearing in a loop
The relation and the increment happen in the same block
Loop 1
Loop 2
Loop 1
Loop 1
Loop 2
Loop 1
Loop 2
Loop 1
i = 1
i <= N
print(count)
count = i i++
j < i
count = 0
count++
j= 0
j++
Parent scope Parent scope
Loop 1
Loop 1
Loop 2
CONTROL FLOW
INFORMATION ANNOTATED
IN A D-D GRAPH
Loop 1
![Page 7: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/7.jpg)
http://www.aspiringminds.com
Results: Question Specific Models
Validation correlation >= 0.79
Matches inter-rater correlation between two human raters
(Published at KDD 2014)
PROBLEM # of
features
Cross-val
correl
Train
correl
Validation
correl
Test Case
Score
1 80 0.61 0.85 0.79 0.54
2 68 0.77 0.93 0.91 0.80
3 193 0.91 0.98 0.90 0.64
4 66 0.90 0.94 0.90 0.80
5 87 0.81 0.92 0.84 0.84
![Page 8: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/8.jpg)
http://www.aspiringminds.com
ML Models are question specific
Binary search Bubble sort
if-in-for for-in-for
![Page 9: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/9.jpg)
http://www.aspiringminds.com
Solving for the industry needs
scale
• The solution doesn’t scale!!!
• We have 500+ questions in our database and support 35+ languages
How to get question independent models?
![Page 10: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/10.jpg)
http://www.aspiringminds.com
Question independent ML models
![Page 11: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/11.jpg)
http://www.aspiringminds.com
Two main ideas
Feature transformation - Convert original features into ‘structurally invariant’ features How?
Exploit Automatic identification of good
responses –Test cases can tell us this.
![Page 12: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/12.jpg)
http://www.aspiringminds.com
How does it work?
We just need a set of good codes for a new question, no labeled sample….
![Page 13: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/13.jpg)
http://www.aspiringminds.com
Results Ques Set Metric QuesSpec QuesIndep-N1 QuesIndep-N0 Baseline-TC
Correl 0.84 0.8 0.76 0.65
Bias 0.14 0.24 0.28 0.35
MAE 0.41 0.58 0.66 0.85
Correl 0.85 0.8 0.76 0.65
Bias 0.14 0.27 0.34 0.31
MAE 0.43 0.62 0.7 0.84
All
questions
Unseen
questions
• Question-independent vs test-case baseline • Question-independent vs question-specific • Performance on unseen problems
(Published at KDD 2016)
• Correl – Pearson coefficient (r) • Bias – Average (ye – yp) • MAE – Average(|ye-yp|)
![Page 14: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/14.jpg)
http://www.aspiringminds.com
Country % Uncompilable
Codes
% Completely
Correct
US 18.5% 44.5%
India 58% 16.8%
Quality difference - Programmers
16% of candidates with uncompilable code had near correct logic!
![Page 15: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/15.jpg)
http://www.aspiringminds.com
For programs with syntax errors
Code
assignment
element = expr ;
100
statement
Syntax
Tree
• 26% more candidates were shortlisted for interviews
• 19% more were hired
(Published at IAAI 2019)
![Page 16: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/16.jpg)
http://www.aspiringminds.com
Various AI-led products
• Automatic Grading of essays
• Automatic Grading of emails
• A tab based motor skills test
• Simulated Chat customers
![Page 17: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/17.jpg)
http://www.aspiringminds.com
Lessons • Problems are multi-disciplinary – spawn fields
• Problems need novel theoretically plausible features
• Small data-sets; Expensive to get labels; Need to create good balanced data sets
• Need generalized solutions for problem space which scales
• Format assumptions of solution may break – need to handle by novel means
• Design of experiments and result interpretation is key
![Page 18: Skill Evaluation as a Computer Science Question: Opportunities … · 2019-04-25 · Skill Evaluation as a Computer Science Question: Opportunities and Challenges. ... AMCAT GRE/SAT](https://reader033.fdocuments.us/reader033/viewer/2022060209/5f0467b57e708231d40dceff/html5/thumbnails/18.jpg)
http://www.aspiringminds.com
Discussion?