Pascal Challenges - LRI
Transcript of Pascal Challenges - LRI
![Page 1: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/1.jpg)
Pascal Challenges
Michele Sebag
EuCOGIII, April 11th, 2013
![Page 2: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/2.jpg)
Challenges: several visions
Vision 1
I Structure ab ovo a research roadmap
I Define core issues and measures of progress
Vision 2 Requisite
I Designed by organizers
I Reviewed relevant
I Proposed to the community fun
I Defining actionable research questions doable
I Structuring a posteriori the communityThe website of the challenge remains open for post-challenge
submissions... the challenge and workshop websites [plus a survey
paper/book] offer a complete teaching toolkit and a valuable
resource for engineers and scientists.
![Page 3: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/3.jpg)
Pascal Challenges: Mission
a catalyst for both research and application development
Three categories of challenges
I Core enabling skills for cognitive systems:Vision; Speech and Language
I Advancing MLRepresentation (transfer, unsupervised); Active learning;Causality; Scalability, ...
I Reaching oute-Science, Health, CHI, CRM, Content selection, ...
![Page 4: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/4.jpg)
Pascal Challenges: Mission
Co-evolution of organizers/participants
I Needed: rigorous evaluation procedureIf someone can cheat, someone will...
I Comprehensive comparison of approachesefficiency / originality / insights
I Be prepared for unexpected results
Challenges set the trend
I Build resources −→ mldata.org
I Definition of tasks
I Definition of metrics
I Citations, BP Awards
I Super-methods (ensemble of submitted methods)
![Page 5: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/5.jpg)
Some lessons learned
Guyon, 2013
Challenges are a powerful tool to perform empirical research inMachine Learning and get (relatively) unbiased answers toquestions of general interest. Often, the answers go againstcommon opinions established with largely overfitted studies.
I Should one perform active learning? Most published studies:yes. Active learning challenge : NO unless you really need;once you start selecting examples your can easily distort yourinput distribution and learn the ”wrong” problem.
I Does unsupervised learning help as a preprocessing? Opinionsare split. Unsupervised challenge: YES!
![Page 6: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/6.jpg)
Some lessons learned, followed
I Should you use causal discovery methods to find morecausally relevant input features? Studies assuming an oracle isavailable to give you conditional independence relationships:yes. Causality challenges: NO unless you really know whatyou are doing. It is very hard to beat non-causal featureselection even to solve causal problems.
I Can you learn from just one training examples difficult taskslike recognizing gestures from video data? Opinions are split.Gesture recognition challege: YES!
![Page 7: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/7.jpg)
Some views
1. Setting the trend: Recognizing Textual Entailment
2. Pushing the state of the art: Visual Object Challenges
3. Reaching out:I Gesture recognitionI Causality studies
![Page 8: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/8.jpg)
Recognizing Textual Entailment
Ido Dagan, Bernardo Magnini et al.
Goal Given H and T, does H implies T ?
Example
H Reagan attended a ceremony in Washington to commemoratethe landings in Normandy.
→ ?
T Washington is located in Normandy.
![Page 9: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/9.jpg)
Recognizing Textual Entailment, 2
![Page 10: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/10.jpg)
Recognizing Textual Entailment, 3
A building block for:
I Reading comprehension
I Question answering
I Information extraction
I Paraphrase Acquisition
I Summarization
I Machine Translation
I Slot filling (Knowledge Population Base)
![Page 11: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/11.jpg)
RTE, 7 rounds and future
Last four RTEs
I Track of the Text Analysis Conference (TAC) at NIST, theU.S. National Institute of Standards and Technology since2008
I Plus summarization, novelty detection, slot filling (KnowledgeBase Population)
Next: toward educational technology
I Student Response Analysis task at SemEval 2013(http://www.cs.york.ac.uk/semeval-2013/task7/).
I Given a question, a known correct reference answer and astudent answer, classify student answer ascorrect; partially correct but incomplete; contradictory;irrelevant; not in the domain.
![Page 12: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/12.jpg)
Some views
1. Setting the trend: Recognizing Textual Entailment
2. Pushing the state of the art: Visual Object Challenges
3. Reaching out:I Gesture recognitionI Causality studies
![Page 13: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/13.jpg)
Visual Objects Challenge 2012
Williams, Zisserman, Everingham et al
![Page 14: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/14.jpg)
Visual Objects Challenge, tasks
![Page 15: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/15.jpg)
VOC, advancing the state of the art
State of art
I Features: Dense SIFT, HOG, colour
I Encodings: spatial pyramid, BOW, Fisher vector
I Detectors: DPM
I Classifier: SVM
This year
I Complex log-normal features
I Sub-clusters for classes
I Combinations of clusterings and projections
I New metrics: timeliness (AP vs time).
![Page 16: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/16.jpg)
Visual Objects Challenge 2012, 4/4
Super-methodsCascade: use methods score as attributes; train using linear SVMs.
ContinuedImageNet Large Scale Visual Recognition Challenge
![Page 17: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/17.jpg)
Some views
1. Setting the trend: Recognizing Textual Entailment
2. Pushing the state of the art: Visual Object Challenges
3. Reaching out:I Gesture recognitionI Causality studies
![Page 18: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/18.jpg)
Gesture recognition
Guyon et al
http://gesture.chalearn.org
![Page 19: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/19.jpg)
Gesture recognition
A building block for:
I Action and activity recognition
I Recognition of sign languages
I Human action recognition
I Teaching communication
I Video indexing/retrieval
I Emotion recognition
I Video surveillance
I Performing arts
I Games
![Page 20: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/20.jpg)
SettingsEasy
I Fixed camera
I Depth available
I Within a batch, single user, small vocabulary, homogeneousrecording conditions
I Mostly arm and hand gestures
I Camera framing upper body
Challenging
I A single example of each gesture within a batch
I Skeleton tracking data not provided
I Variation in background, clothing, skin color, lighting, resolutionamong batches
I Some parts of body occluded
I Some users more skilled than others
![Page 21: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/21.jpg)
Settings
![Page 22: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/22.jpg)
Results, advances
1stnovel technique inspired by the neural mechanisms underlyinginformation processing in the visual system.
2ndHOG/HOF features. Recognition and temporal segmentation withDTW. Quadratic-chi kernel similarity.
3rdDTW temporal segmentation. Bag of features from 3D motionSIFT from RGB-D data. KNN classifier.
![Page 23: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/23.jpg)
Some views
1. Setting the trend: Recognizing Textual Entailment
2. Pushing the state of the art: Visual Object Challenges
3. Reaching out:I Gesture recognitionI Causality studies
![Page 24: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/24.jpg)
Causality identification
Societal impact
![Page 25: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/25.jpg)
Causality identification
Societal impact
![Page 26: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/26.jpg)
Causality identification
Societal impact
![Page 27: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/27.jpg)
Causality identification
Features
I Observations available
I Correlation 6= causality
I Experiments would serve: but ! (costly, infeasible, unethical)
Setting
I A simple benchmark: financial series, possibly reverted;predict whether time has been reverted.
I No feedback-loop, no time
I With an independence oracleind. tests require data and simplifying assumptions (causal
sufficiency, Gaussian noise...)
![Page 28: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/28.jpg)
Causality identification
Modelling causalityA = F(B, noise); B = F(A, noise);
![Page 29: Pascal Challenges - LRI](https://reader031.fdocuments.us/reader031/viewer/2022012002/61d90c380eecd50049649db8/html5/thumbnails/29.jpg)
Some concluding remarks...
Vision 1provide a common plan and vision, a consensus on what should bedone first and what counts as success...
Vision 2
I Proposed by the organizers to the communityplatform heterogeneity ?
I Defining actionable research questionse.g. Knowledge: innate / acquired / generality ?
Challenge: an operational/conceptual way of structuring thecommunity.