PeTra: A Sparsely Supervised Memory Modelfor People Tracking
Shubham ToshniwalAllyson Ettinger Kevin Gimpel Karen Livescu
github.com/shtoshni92/petra
People Tracking
Amelia Shepherd, M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone. In her debut appearance, Amelia visited her former sister-in-law.
People Tracking
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia1 visited her1 former sister-
in-law.
People Tracking
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia1 visited her1 former sister-
in-law.
Coreference Resolution for people references
Coreference Resolution
Mention Ranking Models
Amelia Shepherd Caterina Scorsone her Amelia
her
Mention Ranking Models
Amelia Shepherd Caterina Scorsone her Amelia
her
Mention Ranking Models
Amelia Shepherd Caterina Scorsone her Amelia
her
Impractical for long documents!
Entity-Mention Models
Entities
...
Caterina Scorsone
Amelia Shepherd
her
Entity-Mention Models
Entities
...
Caterina Scorsone
Amelia Shepherd
her
Entity-Mention Models
Entities
...
Caterina Scorsone
Amelia Shepherd
her
Number of entities can be quite large!Fixed-size Memory
PeTra
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
Model Details
BERT Encoder
wt
ht
Mt−1 Mt. . . . . .
Controller
et, ot, ct
. . . . . .
. . . . . .
GRU HiddenStates
Input Tokens
Memory
ControllerOutputs
Key Outputs:et Entity Probabilityot Overwrite Prob. Vectorct Coreference Prob. Vector
Sparse Supervision
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia visited her former sister-in-
law.
Auxiliary Loss
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia visited her former sister-in-
law.
Most tokens don’t refer to people!Penalize high entity probability outside labeled spans.
Results
GAP Results
2 4 6 8 10 12 14 16 18 20
# of memory cells
80
81
82
83
84
85
86F
-sco
re
Referential Reader
GAP Results
2 4 6 8 10 12 14 16 18 20
# of memory cells
80
81
82
83
84
85
86F
-sco
re
Referential Reader
GAP Results
2 4 6 8 10 12 14 16 18 20
# of memory cells
80
81
82
83
84
85
86F
-sco
re
Referential Reader
PeTra
Memory Log Visualization
CROW
CROW
CROW
[CLS
]Am
elia
Shep
herd , M . D . is a
fictio
nalch
arac
ter .
.....
portra
yed by
Cat##er
ina Sc
##ors
##one . In her
debut
appea
rance in
seas
onth
ree ,
Ameli
avis
ited
herfo
rmer
......
[SEP
]
CROW
PeTra can resolve references beyond pronouns!
Interpretability
Tracking People
Are the models actually tracking people?
Tracking People
Are the models actually tracking people?
What does it mean for memory model to “track people”?
Detect all references to people.One-to-one correspondence between people and memory cells.
Human Evaluation
Neef took an individual silver medal at the 1994 European Cup behind Rus-sia’s Svetlana Goncharenko and returned the following year to win gold. Shewas a finalist individually at the 1994 European Championships and camesixth for Scotland at the 1994 Commonwealth Games.
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
Human Evaluation
Neef took an individual silver medal at the 1994 European Cup behind Rus-sia’s Svetlana Goncharenko and returned the following year to win gold. Shewas a finalist individually at the 1994 European Championships and camesixth for Scotland at the 1994 Commonwealth Games.
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
PeTra Ref. Reader Neutral74% 8% 18%
Conclusion
- Propose a memory model for tracking people with sparse pronoun resolutionannotations.
- Outperforms previous approach on both the end task and interpretability measures.
- Future work: Extend this work to longer documents.
References
Fei Liu, Luke Zettlemoyer, and Jacob Eisenstein.The Referential Reader: A recurrent entity network for anaphora resolution.ACL, 2019.Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge.Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns.TACL, 2018.Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun.Tracking the world state with recurrent entity networks.ICLR, 2017.Alex Graves et al.Hybrid computing using a neural network with dynamic external memory.Nature, 2016.
Top Related