Action Assessment by Joint Relation Graphs
Transcript of Action Assessment by Joint Relation Graphs
![Page 1: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/1.jpg)
Action Assessment by Joint Relation Graphs
Jia-Hui Pan 1, Jibin Gao 1, Wei-Shi Zheng 1,2,3
1 Sun Yat-sen University, China 2 Peng Cheng Laborator
3 Key Laboratory of Machine Intelligence and Advanced Computing
ICCV 2019
![Page 2: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/2.jpg)
Introduction
Action Assessment: Video -> Score
Whole Scene Separate JointsJoint Relations
Prior works Prior worksThis paper
Why joint relations? An example in diving:
bending knee + bending ankle and hip = Good (e.g. the rolling stage)bending knee + straight ankle and hip = Bad (e.g. the water-entering stage)
![Page 3: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/3.jpg)
Introduction
Good Performance
Body CoordinationBody Part Movement(The commonality module) (The difference module)
Spatial Relation Graph Temporal Relation Graph
![Page 4: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/4.jpg)
Overview
![Page 5: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/5.jpg)
Spatial Relation Graph
Nodes: Joints at the same time stepEdge: Learnable relations between joints
Temporal Relation Graph
Non-negtive and learnableEdges not in the skeleton are set as zero
Nodes: Joints at the adjacent time stepEdge: Learnable relations between joints
Non-negtive and learnable
![Page 6: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/6.jpg)
The Commonality Module
Learning the features within locally connected joints
Extracted video featuresUpdated features
x
Feature aggregation by average pooling:
![Page 7: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/7.jpg)
The Difference Module
Learning coordination in joint neighbourhoods
Extracted video featuresSpatial difference: J x M
Temporal difference: J x M Learnable weight
Feature aggregation by average pooling:
![Page 8: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/8.jpg)
Regression Module
Input:The whole scene featureThe commonality featuresThe difference features
Weighted feature pooling: Orthogonal regularization:
Final regression with two FCs:
![Page 9: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/9.jpg)
Experiments
I3D features and Mask-RCNN human poses
Skeleton on JIGSAWS
The Olympic Actions
JIGSAWS: Kinematic features for joints
Video
Why ST-GCN so low
Kinematic
VideoKinematic
BothBoth
![Page 10: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/10.jpg)
Ablation Study
![Page 11: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/11.jpg)
Visualization
For gymvault:In the spatial graph, hips, shoulders, and knees are closely related.
In the temporal graph, shoulders are more attended.
![Page 12: Action Assessment by Joint Relation Graphs](https://reader034.fdocuments.us/reader034/viewer/2022050512/6271eb847598bc49227d18ac/html5/thumbnails/12.jpg)
Conclusion:• Assess the action performance through graph-based joint relation modelling • Joint commonality module and the joint difference module
Comments:- - Similar to the methods in skeleton-based action recognition- - Depend on the existence of joints and the human pose estimation method