1

AR Display for Observing Sports Events based on Camera Trackingbased on Camera Tracking Using Pattern of Ground

Akihito Enomoto, Hideo Saito

saito@hvrl.ics.keio.ac.jpwww.hvrl.ics.keio.ac.jp

HVRL H Vi i R h L bHVRL: Hyper Vision Research Lab.

Keio University

July 23, 2009

Recent Trend for Visual Media S th i d A l i

2

Synthesis and Analysis

Multiple CameraMultiple CameraMultiple Camera

Virtualized Reality (CMU)

Multiple Camera

Virtualized Reality (CMU)Virtualized Reality (CMU) Matrix

NFL EyeVision

Virtualized Reality (CMU) Matrix

NFL EyeVision 3D Display Technology

Free-Viewpoint VideoAuto Stereoscopic DisplayAuto-Stereoscopic Display

Volumetric 3D Display

Virtual Reality

Mixed Reality Display

Virtual Reality

Mixed Reality DisplayMixed Reality DisplayMixed Reality Display

3Virtualized Reality (CMU 1997 99)Example:

(CMU,1997-99)

Inputsequence

3-Men Basketballsequence

(These are movies.)

Synthetic court

4D Model

Synthetic court

4

For soccer scenes

cam 1 cam 4cam 2 cam 3

cam1 cam2 cam3 cam4

[Inamoto and Saito ICPR2002][Inamoto and Saito, ICPR2002][Inamoto and Saito, IEEE Trans. MM, 07]

5Calculation of Calculation of

Viewpoint PositionViewpoint PositionArbitrary View Synthesis Arbitrary View Synthesis

of Soccer Sceneof Soccer SceneRendering on Rendering on The StadiumThe StadiumViewpoint PositionViewpoint Position of Soccer Sceneof Soccer Scene The StadiumThe Stadium

・

・・・

Virtual Views of

Multiple View Images Captured at A Stadium

The Stadium

GUI

Ref.Cam1 Ref.Cam2

GUI

Overlay on The yStadium ImageVirtual View of The

Dynamic Regions

6

Example of Free Viewpoint ImagesExample of Free Viewpoint Images

Cam 1Cam 2Cam 2

Cam 3 Cam 4

7

Observe Soccer Match on the DesktopObserve Soccer Match on the Desktop[Inamoto, Saito ISMAR03]

User sees a desktop stadium model in the real world with video see-through HMD and observes dynamic objects of

Video See Through HMD

soccer scene overlaid onto the display.

Desktop Stadium Model

Video See-Through HMD

C t d S

Virtual View Virtual View GenerationGeneration

Captured Soccer Match at Stadium

8Determination of Determination of

Viewpoint PositionViewpoint PositionArbitrary View Synthesis Arbitrary View Synthesis

of Soccer Sceneof Soccer SceneRendering on Rendering on The StadiumThe StadiumViewpoint PositionViewpoint Position of Soccer Sceneof Soccer Scene The StadiumThe Stadium

・・・

M lti l Vi I

Virtual Views of The StadiumMultiple View Images

Captured at A Stadium

GUI

Stadium

GUI

Overlay on The Stadium ImageStadium ImageRef.Cam1 Ref.Cam2

Virtual View of The Dynamic RegionsHMD Camera Image

Overlay on Desktop Stadium Model

9

Experimental ResultsExperimental ResultsWe have implemented immersive observation systemWe have implemented immersive observation system for actual soccer matches.

Captured soccer images : 720×480 pixel, 24-bit-RGB color

Camera 1

Camera 2 Camera 3 Camera 4

Camera Configuration at A Stadium Canon Video See-Through HMDCamera Configuration at A Stadium Canon Video See Through HMD

10

ExampleExample

Stadium Camera 1 Stadium Camera 2Frame 335

( Camera 1-2 w = 0.47 z = 1.07 )( Camera 1-2 w = 0.5 )On Real Stadium Image On Tabletop Stadium Model

11

•Unstable Camera Tracking•Limited Camera Movement•Limited Camera Movement

12

AR Baseball Presentation Systemy[Uematsu, Saito, ICAT06, IVC09]

• Virtual baseball game is overlaid onto a real field model• Virtual baseball game is overlaid onto a real field model.• A user watches the game through a handheld monitor.

The baseball game is replayed from a scorebook data file– The baseball game is replayed from a scorebook data file.– Multiple planar markers are automatically integrated.

web-camera baseball field model

2D markers

handheld monitor

2D markers

user

13

Demo Video

In this presentation14

In this presentation…..

• Real soccer player captured with multiple cameras in stadium

• Observering cameraWid t– Wide-area movement

– Real-time, stable tracking

15

System ConfigurationSystem ConfigurationSystem ConfigurationSystem Configuration

Fi ld P d AR M kField Pattern and AR MarkerObserver’sViewView

ObserveringCameraCamera

16

Camera Tracking for RegistrationCamera Tracking for RegistrationCamera Tracking for RegistrationCamera Tracking for Registration

•AR Toolkit + Natural Features

[M t k ISMAR06][Motokawa ISMAR06]

17

18

Data Processing FlowData Processing FlowOff-Line On- Line

Observing Camera

Pose/Positionof Observing CameraCamera

Player ExtractionOverlay

Extracted

Stadium Camera (Fixed)

y ExtractedTextures

•Ball•Penalty Area

19

On Line PhaseOn Line PhaseCalibration Stadium Camera Selection AR Display

Observing Camera

Position of

Selection

Position of Projection

SelectionOverlayStadium Camera

Projection Matrices

20

CalibrationCalibrationCalibrationCalibration

Observing Camera

Position of

Selection

Position of Projection

SelectionOverlayStadium Camera

Projection Matrices

21

HomographyHomography Based CalibrationBased CalibrationHomographyHomography Based CalibrationBased CalibrationGilles Simon, et al.:Markerless tracking using planar

y

Markerless tracking using planar structures in the scene, ISAR2000

ZY

Z

X

xImage World

22

OnOn--Line Corner DetectionLine Corner DetectionOnOn Line Corner DetectionLine Corner Detection

Observing Camera

Intial Estimate

Template Matching(a) Initial Estimate

Refine

(b) Template (c) Refine4 Corner Positions

23

Selection of Stadium CameraSelection of Stadium CameraSelection of Stadium CameraSelection of Stadium Camera

Observing Camera

Position of

Selection

Position of Projection

SelectionOverlayStadium Camera

Projection Matrices

24

Stadium Camera with Closest Pan Angle (around Z axis) to Observing Camera is selected

YZ

YZObservingCamera

XX

Stadium Camera

25

AR DisplayAR DisplayAR DisplayAR Display

Observing Camera

Position of

Selection

Position of Projection

SelectionOverlayStadium Camera

Projection Matrices

26

Player PositionPlayer PositionyyPosition in Image

⎥⎥⎤

⎢⎢⎡

⎥⎥⎤

⎢⎢⎡

⎥⎥⎤

⎢⎢⎡

~ Y

X

y

xHomography

Position onGround

⎥⎥⎦⎢

⎢⎣⎥⎥⎦⎢

⎢⎣⎥

⎥⎦⎢

⎢⎣ 1

~

1

YyH3×3

G ou d

YZ

y

(X,Y,0)x X

(X,Y,0)(x,y)

Stadium Camera Image

27

Scaling Magnitude of PlayerScaling Magnitude of PlayerScaling Magnitude of PlayerScaling Magnitude of PlayerRatio of Translational Component of Both Cameras

Ratio =T

Ta

′×

a：Coefficient Ta：CoefficientT：Stadium CameraT’：Observing Camera

T T’T ：Observing Camera

28

Ball PositionBall PositionBall PositionBall PositionStereo Matching

(X,Y,Z)(X,Y,Z)

(u,v)m (u’,v’)m’( )

v v’

P P’u u’

29

Overlaying Virtual Objects:Overlaying Virtual Objects:Overlaying Virtual Objects:Overlaying Virtual Objects:P

30

ResultsResults

Ajinomoto Stadium〈Stadium Capturing〉‐Ajinomoto Stadium‐Three CamerasVideo Size:960×540

10m 20m‐Video Size:960×540

〈AR S t 〉〈AR System〉‐CPU： Core2Duo 3.00GHzMemo 2GB‐Memory： 2GB

‐Video Size： 640×480

31

32

Effect of using Corners and MarkerEffect of using Corners and MarkerEffect of using Corners and MarkerEffect of using Corners and Marker

G d t th iGround truth ismanually measured

Average ( i l) Maximum ( i l)Average (pixel) Maximum (pixel)

Both 1.9 4.1Marker only 16.7 22.8

33

Additional FunctionalitiesAdditional FunctionalitiesAdditional FunctionalitiesAdditional Functionalities-Off-Side Detection-Ball Trajectory Display

Conclusion34

Conclusion

• AR Display for Observing Sports Events – Camera Tracking Using Pattern of GroundCamera Tracking Using Pattern of Ground

• Marker + Corner points tracking

35

• saito@hvrl.ics.keio.ac.jp• www hvrl ics keio ac jp• www.hvrl.ics.keio.ac.jp• Google : HVRL