Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of...
Transcript of Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of...
![Page 1: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/1.jpg)
Between Custom andOff-the-shelf NLP
Yves Peirsman
![Page 2: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/2.jpg)
NLP LANDSCAPE
Cloud APIs:train your own
models+
Libraries: pre-trained
models
Cloud APIs: pre-trained
models
Libraries: train your own
models
2
![Page 3: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/3.jpg)
SENTIMENT ANALYSIS
“The process of computationally identifying and categorizing opinions expressed in a piece of text, especially in order to determine whether the writer's attitude towards a particular topic, product, etc., is positive, negative, or neutral.”
3
![Page 4: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/4.jpg)
SENTIMENT ANALYSIS: APPLICATIONS
http://tarsier.monkeylearn.com
4
![Page 5: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/5.jpg)
SENTIMENT ANALYSIS: APPLICATIONS
http://varianceexplained.org/r/trump-tweets/
5
![Page 6: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/6.jpg)
SENTIMENT ANALYSIS: APPLICATIONS
http://www.stockfluence.com/
6
![Page 7: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/7.jpg)
OVERVIEW
Off-the-shelf NLP DIY NLP Conclusions
7
![Page 8: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/8.jpg)
NLP LANDSCAPE
Cloud APIs:train your own
models+
Libraries: pre-trained
models
Cloud APIs: pre-trained
models
Libraries: train your own
models
8
![Page 9: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/9.jpg)
DATA SETS
Domain Source Categories Baseline
Movie reviews rottentomatoes.com positive, negative 50.0%
Baby products Amazon positive (****, *****),negative (*, **) 81.5%
Android apps Amazon positive (****, *****),negative (*, **) 72.7%
Android apps Amazon, Wikipedia positive (****, *****),negative (*, **), neutral 51.7%
Hotels, restaurants Yelp positive (****, *****),negative (*, **) 70.8%
9
![Page 10: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/10.jpg)
DATA SETS: EXAMPLES
PositiveIf you're looking for something scary, this is the first great horror film of the spooky season.
NeutralAvernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh and Windows-based computers. Several are available for iPad and Android tablet.
NegativeIt's Starbucks only with bad customer service. Baristas with attitude that don't know their own product. If I'm paying $7.00 for a coffee at least drop the 'tude.
10
![Page 11: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/11.jpg)
OFF-THE-SHELF NLPIs it OK to be lazy?
11
1.
![Page 12: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/12.jpg)
Variables
OFF-THE-SHELF MODELS
Data
12
![Page 13: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/13.jpg)
OFF-THE-SHELF MODELS: MOVIE REVIEWS 13
Indico 76.8%
IBM AlchemyAPI 73.4%
Stanford CoreNLP 71.9%
![Page 14: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/14.jpg)
OFF-THE-SHELF MODELS: BABY PRODUCTS 14
Indico 92.6%
MonkeyLearn (Product) 87.5%
TextBlob Pattern 82.5%
![Page 15: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/15.jpg)
OFF-THE-SHELF MODELS: YELP REVIEWS 15
Indico 92.9%
Google 91.0%
IBM AlchemyAPI 90.4%
![Page 16: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/16.jpg)
OFF-THE-SHELF MODELS: ANDROID APPS (2-WAY) 16
Indico 90.6%
Google 90.5%
MonkeyLearn (Product) 87.1%
![Page 17: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/17.jpg)
OFF-THE-SHELF MODELS: ANDROID APPS (3-WAY) 17
Indico 80.0%
HavenOnDemand 79.1%
Google 77.8%
![Page 18: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/18.jpg)
OFF-THE-SHELF MODELS: CONCLUSIONS
There is enormous variation between and within off-the-shelf solutions.
High quality is possible, but not guaranteed.
Comparing available solutions on your data is crucial.
18
![Page 19: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/19.jpg)
DIY MODELSAre you better off building your own custom models?
19
2.
![Page 20: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/20.jpg)
DIY MODELS: PROCESS 20
Data Library Model
![Page 21: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/21.jpg)
DIY MODELS: PROCESS 21
![Page 22: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/22.jpg)
DIY MODELS: BABY PRODUCTS 22
DIY SVM 93.4%
Best off-the-shelf 92.6%
DIY Naive Bayes 86.3%
![Page 23: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/23.jpg)
DIY MODELS: ANDROID APPS 23
DIY SVM 93.1%
Best off-the-shelf 90.6%
DIY Naive Bayes 90.3%
![Page 24: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/24.jpg)
DIY MODELS: YELP REVIEWS 24
DIY SVM 95.6%
Best off-the-shelf 92.9%
DIY Naive Bayes 89.2%
![Page 25: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/25.jpg)
DIY MODELS: BABY PRODUCTS 25
![Page 26: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/26.jpg)
DIY MODELS: ANDROID APPS 26
![Page 27: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/27.jpg)
DIY MODELS: CONCLUSIONS
You need sufficient relevant data to build a good model.
DIY models built with sufficient data will typically outperform off-the-shelf solutions.
DIY models may or may not be worth the effort.
27
![Page 28: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/28.jpg)
Conclusions
28
3.
![Page 29: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/29.jpg)
CONCLUSIONS
Off-the-shelfno data
little effort
good quality possible, but
not guaranteed
no control
DIYlots of datamore effort
superior qualityfull control
29
![Page 30: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/30.jpg)
BUT SOME ARE
30
WRONGALL MODELS ARE
USEFUL- George Box
![Page 31: Yves Peirsman Between Custom and Off-the-shelf NLP · 2020. 11. 3. · Avernum is a series of demoware role-playing video games by Jeff Vogel of Spiderweb Software available for Macintosh](https://reader035.fdocuments.us/reader035/viewer/2022071502/6121dec3ab2d04737445e6d2/html5/thumbnails/31.jpg)
31