PeTra: A Sparsely Supervised Memory Modelfor People Tracking

Shubham ToshniwalAllyson Ettinger Kevin Gimpel Karen Livescu

github.com/shtoshni92/petra

People Tracking

Amelia Shepherd, M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone. In her debut appearance, Amelia visited her former sister-in-law.

People Tracking

Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia1 visited her1 former sister-

in-law.

People Tracking

Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia1 visited her1 former sister-

in-law.

Coreference Resolution for people references

Coreference Resolution

Mention Ranking Models

Amelia Shepherd Caterina Scorsone her Amelia

her

Mention Ranking Models

Amelia Shepherd Caterina Scorsone her Amelia

her

Mention Ranking Models

Amelia Shepherd Caterina Scorsone her Amelia

her

Impractical for long documents!

Entity-Mention Models

Entities

...

Caterina Scorsone

Amelia Shepherd

her

Entity-Mention Models

Entities

...

Caterina Scorsone

Amelia Shepherd

her

Entity-Mention Models

Entities

...

Caterina Scorsone

Amelia Shepherd

her

Number of entities can be quite large!Fixed-size Memory

PeTra

Ideal Run

Amelia

Amelia

OW

Shepherd

CR

, portrayed by

. . .

IG

Caterina

OW

Scorsone

CR

, visits

. . .

IG

her

CR

friend

IG

IG = Ignore, OW = Overwrite, CR = Coreference

Ideal Run

Amelia

Amelia

OW

Shepherd

CR

, portrayed by

. . .

IG

Caterina

OW

Scorsone

CR

, visits

. . .

IG

her

CR

friend

IG

IG = Ignore, OW = Overwrite, CR = Coreference

Ideal Run

Amelia

Amelia

OW

Shepherd

CR

, portrayed by

. . .

IG

Caterina

OW

Scorsone

CR

, visits

. . .

IG

her

CR

friend

IG

IG = Ignore, OW = Overwrite, CR = Coreference

Ideal Run

Amelia

Amelia

OW

Shepherd

CR

, portrayed by

. . .

IG

Caterina

OW

Scorsone

CR

, visits

. . .

IG

her

CR

friend

IG

IG = Ignore, OW = Overwrite, CR = Coreference

Ideal Run

Amelia

Amelia

OW

Shepherd

CR

, portrayed by

. . .

IG

Caterina

OW

Scorsone

CR

, visits

. . .

IG

her

CR

friend

IG

IG = Ignore, OW = Overwrite, CR = Coreference

Model Details

BERT Encoder

wt

ht

Mt−1 Mt. . . . . .

Controller

et, ot, ct

. . . . . .

. . . . . .

GRU HiddenStates

Input Tokens

Memory

ControllerOutputs

Key Outputs:et Entity Probabilityot Overwrite Prob. Vectorct Coreference Prob. Vector

Sparse Supervision

Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia visited her former sister-in-

law.

Auxiliary Loss

Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia visited her former sister-in-

law.

Most tokens don’t refer to people!Penalize high entity probability outside labeled spans.

Results

GAP Results

2 4 6 8 10 12 14 16 18 20

# of memory cells

80

81

82

83

84

85

86F

-sco

re

Referential Reader

GAP Results

2 4 6 8 10 12 14 16 18 20

# of memory cells

80

81

82

83

84

85

86F

-sco

re

Referential Reader

GAP Results

2 4 6 8 10 12 14 16 18 20

# of memory cells

80

81

82

83

84

85

86F

-sco

re

Referential Reader

PeTra

Memory Log Visualization

CROW

CROW

CROW

[CLS

]Am

elia

Shep

herd , M . D . is a

fictio

nalch

arac

ter .

.....

portra

yed by

Cat##er

ina Sc

##ors

##one . In her

debut

appea

rance in

seas

onth

ree ,

Ameli

avis

ited

herfo

rmer

......

[SEP

]

CROW

PeTra can resolve references beyond pronouns!

Interpretability

Tracking People

Are the models actually tracking people?

Tracking People

Are the models actually tracking people?

What does it mean for memory model to “track people”?

Detect all references to people.One-to-one correspondence between people and memory cells.

Human Evaluation

Neef took an individual silver medal at the 1994 European Cup behind Rus-sia’s Svetlana Goncharenko and returned the following year to win gold. Shewas a finalist individually at the 1994 European Championships and camesixth for Scotland at the 1994 Commonwealth Games.

CROW

CROW

CROW

CROW

CROW

CROW

CROW

[CLS

] N##

ee ##f

took an

indi

vidua

lsil

ver

med

al at the

1994

Euro

pean Cup

behi

ndRu

ssia ' s S

##ve

t##

lana Go

##nc

h##

are

##nk

oan

dre

turn

ed the

follo

win

gye

ar to win

gold .

She

was a

final

istin

divid

ually at the

1994

Euro

pean

Cham

pion

ship

san

dca

me

sixth for

Scot

land at th

e19

94

Com

mon

wea

lthGa

mes

.[S

EP]

CROW

Mem

ory

Cells

CROW

CROW

CROW

CROW

CROW

CROW

CROW

[CLS

] N##

ee ##f

took an

indi

vidua

lsil

ver

med

al at the

1994

Euro

pean Cup

behi

ndRu

ssia ' s S

##ve

t##

lana Go

##nc

h##

are

##nk

oan

dre

turn

ed the

follo

win

gye

ar to win

gold .

She

was a

final

istin

divid

ually at the

1994

Euro

pean

Cham

pion

ship

san

dca

me

sixth for

Scot

land at th

e19

94

Com

mon

wea

lthGa

mes

.[S

EP]

CROW

Mem

ory

Cells

Human Evaluation

Neef took an individual silver medal at the 1994 European Cup behind Rus-sia’s Svetlana Goncharenko and returned the following year to win gold. Shewas a finalist individually at the 1994 European Championships and camesixth for Scotland at the 1994 Commonwealth Games.

CROW

CROW

CROW

CROW

CROW

CROW

CROW

[CLS

] N##

ee ##f

took an

indi

vidua

lsil

ver

med

al at the

1994

Euro

pean Cup

behi

ndRu

ssia ' s S

##ve

t##

lana Go

##nc

h##

are

##nk

oan

dre

turn

ed the

follo

win

gye

ar to win

gold .

She

was a

final

istin

divid

ually at the

1994

Euro

pean

Cham

pion

ship

san

dca

me

sixth for

Scot

land at th

e19

94

Com

mon

wea

lthGa

mes

.[S

EP]

CROW

Mem

ory

Cells

CROW

CROW

CROW

CROW

CROW

CROW

CROW

[CLS

] N##

ee ##f

took an

indi

vidua

lsil

ver

med

al at the

1994

Euro

pean Cup

behi

ndRu

ssia ' s S

##ve

t##

lana Go

##nc

h##

are

##nk

oan

dre

turn

ed the

follo

win

gye

ar to win

gold .

She

was a

final

istin

divid

ually at the

1994

Euro

pean

Cham

pion

ship

san

dca

me

sixth for

Scot

land at th

e19

94

Com

mon

wea

lthGa

mes

.[S

EP]

CROW

Mem

ory

Cells

PeTra Ref. Reader Neutral74% 8% 18%

Conclusion

- Propose a memory model for tracking people with sparse pronoun resolutionannotations.

- Outperforms previous approach on both the end task and interpretability measures.

- Future work: Extend this work to longer documents.

References

Fei Liu, Luke Zettlemoyer, and Jacob Eisenstein.The Referential Reader: A recurrent entity network for anaphora resolution.ACL, 2019.Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge.Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns.TACL, 2018.Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun.Tracking the world state with recurrent entity networks.ICLR, 2017.Alex Graves et al.Hybrid computing using a neural network with dynamic external memory.Nature, 2016.