Slide 1

CVPR2013 Poster Modeling Actions through State Changes Slide 2 Outline 1.Introduction 2.Method 3.Results 4.Dicussion Slide 3 1. Introduction What makes an action (e.g. open the jar) identifiable? How can we tell if such an action is performed?. Slide 4 1. Introduction We are interested in two kinds of changes 1. changes in the state of objects 2. changes the transformation of stuff we adopt egocentric paradigm for recognizing daily activities and actions. we propose a weakly supervised method for learning the object and material states that are necessary for recognizing daily actions. Slide 5 2. Method 2-1 Discovering State-Specific Regions 2-2 States as Action Requirements 2-3 Modeling Actions through State Changes 2-4 Activity Segmentation Slide 6 2-1 Discovering State-Specific Regions two assumptions (1) an object state or material does not change unless an action is performed (2) an object state or material change is associated with an action only if it consistently occurs at all instances of that action. Slide 7 Slide 8 2-1 Discovering State-Specific Regions 1. we identify regions that either appear or disappear as a result of the execution of each action instance. 2. we prune changes that are not consistently associated with an action. 3. we learn a detector for each group of discovered state- specific regions. Slide 9 Change Detection we find the regions that either appear or disappear we sample a few frames from its beginning and a few frames from its end. we match their pixels using large displacement optical flow [3]. Then for each pair of matched pixels, we compute change based on their color difference Slide 10 We compare each beginning (ending) image to multiple ending (beginning) images. We set the amount of change to the minimum amount computed among all the comparisons. Slide 11 Consistent Regions We only keep the subset of those regions that consistently occur across the instances of an action type. Slide 12 Slide 13 State-Specific Region Detectors Purpose: learn for cluster We describe each region Color:128 Texture:256 Shape:16 Slide 14 2-2 State as Action Requirements Environment state (before/after) Two Criteria Two way Regions exist/absent ? Moved by hands ? Response vector Chose highest score > Threshold ->1 Other -> 0 Foreground segmentation method Slide 15 2-3 Modeling Action through State Changes Slide 16 2-4 Activity Segmentation takes all the action detection scores as input and infers the frames that are assigned to each action in that video Train two state detectors State detectors are trained using linear SVM Result in two |A| x T matrices Slide 17 2-4 Activity Segmentation Slide 18 3.Result 3-1 Action Recognition 3-2 Activity Segmentation Slide 19 3-1 Action Recognition Use GTEA Datebase STIP SIFT Previous work Our method Slide 20 Slide 21 Slide 22 3-2 Activity Segmentation Slide 23 4.Dicussion building a taxonomy of possible states of objects and materials. A potential challenge would be modeling the changes that do not correspond to observable visual patterns.