Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have...
Transcript of Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have...
![Page 1: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/1.jpg)
NeuralMT
![Page 2: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/2.jpg)
Announcements• HW2directorystructurepenaltytoberemovedduetogradinginconsistencies.• Thosewholost15pointswillgain15points
• DanJurafskywillaAendthebeginningofclassnextTuesday• BepreparedwithquesEons.Yourchance!!!
• RupalPatel:Monday,Dec.4th,11:30,Davis
![Page 3: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/3.jpg)
• DataScienceIns,tuteColloquiumSeriesEvent:DANJURAFSKY,STANFORDUNIVERSITY|Tuesday,December5that5PMinDavisAuditorium(412CEPSR)
• "DoesThisVehicleBelongtoYou?"ProcessingtheLanguageofPolicingforImprovingPolice-CommunityRela,ons
• • ABSTRACT• Policebody-worncamerashavethepotenEaltoplayanimportantroleinunderstandingandimprovingpolice-communityrelaEons.InthistalkIdescribeaseriesofstudiesconductedbyourlargeinterdisciplinaryteamatStanfordthatusespeechandnaturallanguageprocessingonbody-camerarecordingstomodeltheinteracEonsbetweenpoliceofficersandcommunitymembersintrafficstops.WeusetextandspeechfeaturestoautomaEcallymeasurelinguisEcaspectsoftheinteracEon,fromdiscoursefactorslikeconversaEonalstructuretosocialfactorslikerespect.Idescribethedifferenceswefindinthelanguagedirectedtowardblackversuswhitecommunitymembers,andoffersuggesEonsforhowthesefindingscanbeusedtohelpimprovethefraughtrelaEonsbetweenpoliceofficersandthecommuniEestheyserve.
![Page 4: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/4.jpg)
Today• MulElingualChallengesforMT• MTApproaches• StaEsEcal• Neuralnet(Thursday)
• MTEvaluaEon
![Page 5: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/5.jpg)
MTEvaluation• Moreartthanscience• WiderangeofMetrics/Techniques• interface,…,scalability,…,faithfulness,...space/Emecomplexity,…etc.
• AutomaEcvs.Human-based• DumbMachinesvs.SlowHumans
SlidefromNizarHabash
![Page 6: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/6.jpg)
5 contents of original sentence conveyed (might need minor corrections)
4 contents of original sentence conveyed BUT errors in word order
3 contents of original sentence generally conveyed BUT errors in relationship between phrases, tense, singular/plural, etc.
2 contents of original sentence not adequately conveyed, portions of original sentence incorrectly translated, missing modifiers
1 contents of original sentence not conveyed, missing verbs, subjects, objects, phrases or clauses
Human-basedEvalua,onExampleAccuracyCriteria
SlidefromNizarHabash
![Page 7: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/7.jpg)
5 clear meaning, good grammar, terminology and sentence structure
4 clear meaning BUT bad grammar, bad terminology or bad sentence structure
3 meaning graspable BUT ambiguities due to bad grammar, bad terminology or bad sentence structure
2 meaning unclear BUT inferable
1 meaning absolutely unclear
Human-basedEvalua,onExampleFluencyCriteria
SlidefromNizarHabash
![Page 8: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/8.jpg)
Today:Crowdsourcing• AmazonMechanicalTurkorCrowdFlower• CreateaHITforeachsentence• GetmulEpleworkerstorate• Pay.01to.10perhit• CompleteanevaluaEoninhours(vsdays/weeks)• Ethics?
![Page 9: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/9.jpg)
AutomaticEvaluationExampleBleuMetric(Papinenietal2001)
• Bleu• BiLingualEvalua;onUnderstudy• Modifiedn-gramprecisionwithlengthpenalty• Quick,inexpensiveandlanguageindependent• CorrelateshighlywithhumanevaluaEon• BiasagainstsynonymsandinflecEonalvariaEons
SlidefromNizarHabash
![Page 10: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/10.jpg)
AutomaticEvaluationExampleBleuMetric
TestSentence
colorlessgreenideassleepfuriously
GoldStandardReferences
alldulljadeideassleepiratelydrabemeraldconceptssleepfuriouslycolorlessimmaturethoughtsnapangrily
SlidefromNizarHabash
![Page 11: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/11.jpg)
AutomaticEvaluationExampleBleuMetric
TestSentence
colorlessgreenideassleepfuriously
GoldStandardReferences
alldulljadeideassleepiratelydrabemeraldconceptssleepfuriouslycolorlessimmaturethoughtsnapangrily
Unigramprecision=4/5
SlidefromNizarHabash
![Page 12: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/12.jpg)
AutomaticEvaluationExampleBleuMetric
TestSentence
colorlessgreenideassleepfuriouslycolorlessgreenideassleepfuriouslycolorlessgreenideassleepfuriouslycolorlessgreenideassleepfuriously
GoldStandardReferences
alldulljadeideassleepiratelydrabemeraldconceptssleepfuriouslycolorlessimmaturethoughtsnapangrily
Unigramprecision=4/5=0.8Bigramprecision=2/4=0.5
BleuScore=(a1a2…an)1/n=(0.8╳0.5)½=0.6325è63.25
SlidefromNizarHabash
![Page 13: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/13.jpg)
BLEUscoresfor110translationsystemstrainedonEuroparl
Koehn,MTSummit,2005hAp://homepages.inf.ed.ac.uk/pkoehn/publicaEons/europarl-mtsummit05.pdf
![Page 14: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/14.jpg)
![Page 15: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/15.jpg)
AutomaticEvaluationExampleMETEOR(LavieandAgrawal2007)
• MetricforEvaluaEonofTranslaEonwithExplicitwordOrdering• ExtendedMatchingbetweentranslaEonandreference• Porterstems,wordNetsynsets
• UnigramPrecision,Recall,parameterizedF-measure• ReorderingPenalty• ParameterscanbetunedtoopEmizecorrelaEonwithhumanjudgments• Notbiasedagainst“non-staEsEcal”MTsystems
SlidefromNizarHabash
![Page 16: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/16.jpg)
MetricsMATRWorkshop• WorkshopinAMTAconference2008• AssociaEonforMachineTranslaEonintheAmericas
• EvaluaEngevaluaEonmetrics• Compared39metrics• 7baselinesand32newmetrics• VariousmeasuresofcorrelaEonwithhumanjudgment• DifferentcondiEons:textgenre,sourcelanguage,numberofreferences,etc.
SlidefromNizarHabash
![Page 17: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/17.jpg)
AutomaticEvaluationExampleSEPIA(HabashandElKholy2008)• AsyntacEcally-awareevaluaEonmetric• (LiuandGildea,2005;Owczarzaketal.,2007;GiménezandMàrquez,2007)
• UsesdependencyrepresentaEon• MICAparser(Nasr&Rambow2006)• 77%ofallstructuralbigramsaresurfacen-gramsofsize2,3,4
• Includesdependencysurfacespanasafactorinscore• long-distancedependenciesshouldreceiveagreaterweightthanshortdistancedependencies• HigherdegreeofgrammaEcality? 0%
5%10%15%20%25%30%35%40%45%50%
1 2 3 4plus
![Page 18: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/18.jpg)
![Page 19: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/19.jpg)
NeuralMTtakesover• WMT(WorkshoponMachineTranslaEon)• 2015–firstneuralMT,lowerbleuresults
• 2016:neuralMTbeatsphrase-basedandsyntax-based
![Page 20: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/20.jpg)
0
5
10
15
20
25
30
2015 2016 2017
NeuralMT
Phrasebased
ResultsfromWMT(WorkshoponMachineTransla,on)GermantoEnglish2015:Montreal2016and2017:Edinburgh
![Page 21: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/21.jpg)
WMT2017• Tasks• NewstranslaEon• QualityesEmaEon• AutomaEcpost-ediEng• Metrics• MulEmodalMTandmulElingualimagedescripEon• BiomedicaltranslaEon
![Page 22: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/22.jpg)
![Page 23: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/23.jpg)
NewsTranslationTask• 7languages,14tasks(fromandintoEnglish)• Chinese• Czech• German• Finnish• Latvian• Russian• Turkish
• Testdata:3000sentencesperlanguagepairexceptLatvian:2000sentences
![Page 24: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/24.jpg)
TrainingData• Europarl• CommonCrawl• YandexRussian-Englishdata• WikipediaHeadlines• UnitedNaEons• NewsCommentaryV12• EUPressReleaseparallelcorpusforGerman,FinnishandLatvian
![Page 25: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/25.jpg)
SubmittedSystems• 103systemsfrom31insEtuEons(nocompanies)• CompanyreleasesofNeuralMT• Microsou:February2016• Systran:August2016• Google:September2016
![Page 26: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/26.jpg)
HumanEvaluation• Assessonadequacyalonga100pointscale(DirectAssessment)(vsRelaEveRanking)• HowadequatelydoesthetranslaEonexpressthemeaningofthereferencetranslaEon?
• OnetranslaEonperscreen/hit
• 151individualResearchers• 29differentgroups• Contributed12,693translaEonscores• 24days,22hours
• 754AMTworkers• Contributed237,200scores• 47days,23hours
![Page 27: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/27.jpg)
![Page 28: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/28.jpg)
SomeResults
![Page 29: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/29.jpg)
![Page 30: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/30.jpg)
Today• MulElingualChallengesforMT• MTApproaches• StaEsEcal• Neuralnet(Thursday)
• MTEvaluaEon
![Page 31: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/31.jpg)
Encoder-DecoderApproach
![Page 32: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/32.jpg)
BasicRNNApproach
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That is almost
ENCODER
DECODER
![Page 33: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/33.jpg)
BasicRNNApproach
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That is almost
ENCODER
DECODER
EnEreinputrepresentedhere
![Page 34: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/34.jpg)
Recurrentdecoderbut
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That is almost
ENCODER
DECODER
TransiEonzt=f(zt-1,yT-1,hn)BackpropagaEon=Σtδzt/δh
zt zt zt
![Page 35: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/35.jpg)
Choetal2014
![Page 36: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/36.jpg)
ResultsforLongFrequentPhrases
Choetal2014
![Page 37: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/37.jpg)
OtherVariants:Trainweightsseparately
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That is almost
ENCODER
DECODER
![Page 38: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/38.jpg)
AlsoUseful• TrainstackedRNNSusingmulEplelayers• UseabidirecEonalencoder• Thiscanhelpinrememberingtheearlypartofthesourceinputsentence
• Traintheinputsequenceinreverseorder:S1S2S3->T1T2T3wouldbetrainedasS3S2S1->T1T2T3• Why?
![Page 39: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/39.jpg)
ReplacingRNNwithLSTMimprovesperformancefurther
![Page 40: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/40.jpg)
AligningandTranslating
[Bahdanau,Cho,BengioICLR2015]
![Page 41: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/41.jpg)
AttentionMechanism-Scoring
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That ?
ENCODER
DECODER
H’1 H’2 H’3
Score(h’t-1,hs)
3
![Page 42: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/42.jpg)
AttentionMechanism-Scoring
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That ?
ENCODER
DECODER
H’1 H’2 H’3
Score(h’t-1,hs)
3 5
![Page 43: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/43.jpg)
AttentionMechanism-Scoring
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That ?
ENCODER
DECODER
H’1 H’2 H’3
Score(h’t-1,hs)
3 5 1
![Page 44: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/44.jpg)
AttentionMechanism-Scoring
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That ?
DECODER
H’1 H’2 H’3
Convertintoalignmentweights
.3 .5 .1αt
![Page 45: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/45.jpg)
AttentionMechanism-Scoring
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
That ?
DECODER
H’1 H’2 H’3
Buildcontextvector:weightedaverage
αtct Ct=Σsαt(s)hs
![Page 46: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/46.jpg)
Howdoyouscoreit?
das ist fur
h1 h2 h3
x1 X2√ X3√
Y1 Y2 Y3
?
DECODER
H’1 H’2 H’3
Score(hs,H’t)=H’tThsor=H’tTWαhs(Luongetal2015)
αtct
![Page 47: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/47.jpg)
Performance• WithoutaAenEon,LSTMworksquitewellunElasentencegetslongerthan30words• AAenEondoesbeAer,however,evenwithshortersentences• OthertricksinWMT2017:• Improvementsof1.5–3bluepoints(Edin)• LayernormalizaEon,deepernetworks(encoderdepthof5,decoderdepthof8)• BasePhraseEncodings(BPE)• Reducedvocabularyimprovesmemoryefficiency
• Data:parallel,back-translated,duplicatedmonolingual
![Page 48: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/48.jpg)
Questions?
![Page 49: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/49.jpg)
InformationExtraction• ExtracEonofconcretefactsfromtext
• NamedenEEes,relaEons,events• Ouenusedtocreateastructuredknowledgebaseoffacts
![Page 50: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/50.jpg)
• KathyMcKeown,aprofessorfromColumbiaUniversityinNewYorkCity,tookatrainyesterdaytoWashingtonDC.
![Page 51: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/51.jpg)
NamedEntities• KathyMcKeownper,aprofessorfromColumbiaUniversityorginNewYorkCityloc,tookatrainyesterdaytoWashingtonDCloc.
![Page 52: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/52.jpg)
NamedEntities,Relations• KathyMcKeownper,aprofessorfromColumbiaUniversityorginNewYorkCityloc,tookatrainyesterdaytoWashingtonDCloc.• KathyMcKeownfromColumbia• ColumbiainNewYorkCity
![Page 53: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/53.jpg)
NamedEntities,Relations,Events• KathyMcKeownper,aprofessorfromColumbiaUniversityorginNewYorkCityloc,tookatrainyesterdaytoWashingtonDCloc.• KathyMcKeowntookatrain(yesterday)
![Page 54: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/54.jpg)
EntityDiscoveryandLinking• KathyMcKeown,aprofessorfromColumbiaUniversityinNewYorkCity,tookatrainyesterdaytoWashingtonDC.
![Page 55: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/55.jpg)
StateoftheArt(English)
• NamedEnEEes(news)• RelaEons(slotfilling)• Events(nuggets)
F-measure
• 89%• 59%• 63%
Methods:Sequencelabeling(MEMM,CRF),neuralnets,distantlearningFeatures:linguisEcfeatures,similarity,popularity,gazeteers,ontologies,verbtriggers
![Page 56: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/56.jpg)
WhereHaveYouBeenEntityDiscoveryandLinking?GrowwithDEFT 2006-2011 2012-2017HENGJI,RPIMenEonExtracEon Human(most) AutomaEc
NILClustering None 64methods
ForeignLanguages Chinese(5%-10%lowerthanEnglish)
Systemfor282languages(Chinese/Spanishcomparableto/OutperformEnglish);researchtoward3,000languages
DocumentSize - 500à90,000documents
Genre News,webblog News,DiscussionForum,Webblog,Tweets
EnEtyTypes PER,GPE,ORG PER,GPE,ORG,LOC,FAC,hundredsoffine-grainedtypesfortyping
MenEonTypes Nameorallconcepts(most)
Name,Nominal,Pronoun(forBeST)
KB Wikipedia FreebaseàListonly
TrainingData 20,000queries(enEtymenEons)
500à0documents;unsupervisedlinkingcomparabletosupervisedlinking
#(Good)Papers 62 110(newKBPtrackatACL);6tutorialsattopconferences
SlidefromHengJi
![Page 57: Neural MT - Columbia Universitykathy/NLP/2017/ClassSlides/... · • Police body-worn cameras have the potenEal to play an important role in understanding and improving police-community](https://reader033.fdocuments.us/reader033/viewer/2022052804/5f8af68011f9324a6d7b1101/html5/thumbnails/57.jpg)
OntheHorizon:EntityDiscoveryandLinkingPanel:HoaTrangDang,JasonDuncan,HengJi,KevinKnight,ChristopherManning,DanRoth
DEFTPIMee,ng10;30am-11:30amMay25,2017
• Amgoingcrazy• 3,000languages• 10,000enEtytypes• AllmenEontypes• MulE-media• Streamingmode• List-onlyKB• Context-aware,living• Nomoretrainingdata• On-callevaluaEon• Morenon-tradiEonalknowledgeresources
• Lotsofdevandtestsetsinlotsoflanguages
• Amstayingcool• Successinend-to-endcold-startKBP
• What’ssEllwrongwithnametagging
• SmartercollecEveinference
• ResoluEonoftruealiases• ResoluEonofhandlesusedasenEtymenEons
SlidefromHengJi