Image interpolation for virtual sports scenarios

Machine Vision and Applications (2005) 16(4): 236–245DOI 10.1007/s00138-005-0177-6

ORIGINAL PAPER

Tomas Rodrıguez · Ian Reid · Radu Horaud ·Navneet Dalal · Marcelo Goetz

Image interpolation for virtual sports scenarios

Received: 25 March 2004 / Accepted: 18 February 2005 / Published online: 10 June 2005C© Springer-Verlag 2005

Abstract View interpolation has been explored in the scien-tific community as a means to avoid the complexity of full3D in the construction of photo-realistic interactive scenar-ios. EVENTS project attempts to apply state of the art viewinterpolation to the field of professional sports. The aim is topopulate a wide scenario such as a stadium with a number ofcameras and, via computer vision, to produce photo-realisticmoving or static images from virtual viewpoints, i.e wherethere is no physical camera.

EVENTS proposes an innovative view interpolationscheme based on the Joint View Triangulation algorithm de-veloped by the project participants. Joint View Triangulationis combined within the EVENTS framework with new initia-tives in the field of multiple view layered representation, au-tomatic seed matching, image-based rendering, tracking oc-cluding layers and constrained scene analysis. The computervision software has been implemented on top of a novel highperformance computing platform with the aim to achievereal-time interpolation.

Keywords Image interpolation · Morphing · Layeredsegmentation · Joint view triangulation · Real time virtualscenarios

1 Introduction

One simple but compelling idea to have emerged in recenttimes is the notion of a Panorama or Image Mosaic. Thegeneral underlying concept is based on a set of overlappedshots taken around a central point using a photo camera,

T. Rodrıguez (B)Eptron SA. R&D Department Madrid, SpainE-mail: [email protected]

I. ReidOxford University, Department of Engineering Science, UK

R. Horaud · N. DalalINRIA Rhone-Alpes, Montbonnot, France

M. GoetzC-Lab, Paderborn, Germany

which are spliced or blended together by means of projec-tive geometric transformations. This pervasive idea can nowbe found in devices such as IPIX or de-facto standards suchas QuickTime VR. Current software for panoramas typicallyruns off-line, is based on a single camera with unchangingintrinsic parameters, viewing a static scene, and uses verysimple camera motions (restricted to pure rotation at a sin-gle viewpoint) to ensure that image overlap can be describedby a simple transformation.

However, as this is the crux of the need for innova-tion, traditional panorama technology, being based on a sin-gle viewpoint, is fundamentally inadequate for the scenariosforeseen in the EVENTS project in which we aim to solvemuch more complex problems of multiple viewpoints anddynamic scenes. For multiple viewpoints there no longerexist the simple geometric transformations between imagesthat characterize the panoramic approach and new methodsare required.

EVENTS is a project sponsored by the European Com-mission’s IST programme whose objective is to populatea wide scenario such as a stadium with a number of cam-eras observing a specific part of the scene such as a goal-mouth, and via software implementations of computer vi-sion algorithms, to produce photo-realistic views from view-points where there is no physical camera. The system willenable the user to select a virtual viewpoint somewhere inbetween the physical cameras and to generate a novel viewfrom there, or more impressively, command the software togenerate a sequence as if a camera had swept through thefull range covered by the cameras, smoothly and realisticallyinterpolating intermediate virtual views. EVENTS system isable to process in real-time input images from multiple cam-eras and compute a dynamic synthetic view of the scenario(i.e. sort of virtual video); empowering the viewer to selecthis view position in time and space within the range coveredby the cameras.

The difficulty associated with creating a novel view ofa scene is principally related to the fact that the relativelocation of scene features is directly dependent on theirdepth. Thus, for realistic novel view creation one needs to

Image interpolation for virtual sports scenarios 237

compute, either implicitly or explicitly, the depth of scenepoints. In addition, the following problems must be ad-dressed: how best to match features between views, howto represent the implicit or explicit scene structure, how torender parts of the scene not seen in all cameras, how todeal with occluding boundaries to avoid joining foregroundand background, how to deal with natural scene constraints,how to manage independent motion in the scene andfinally, how to compute all of the above efficiently. Thereis no generally accepted solution to these questions in thescientific community, hence the need for innovation arises.Techniques for achieving novel views [1–3] can be broadlycategorized as follows:

1. Computing full 3D structure: the most direct approachis to determine camera locations via calibration (possi-ble via self-calibration), to match points between images,and hence to triangulate the depth of every point in thescene; i.e. exhaustively recover the scene structure. If thisscene structure were then texture mapped, realistic viewsfrom any viewpoint could potentially be created. As wellas being computationally intensive and potentially error-prone, this technique does not address the issue of how torender parts of the scene where no structure is available,either because of errors, or because no match exists, suchas in occluded regions.

2. Morphing: at the other extreme, involving no structurecomputation, whatsoever is the technique of ‘morphing’developed within the graphics community (in one sense,mosaicing can be thought of as morphing in which thetransformation is known a priori to be a homography).Applying morphing techniques to the 3D scenarios en-visioned in EVENTS leads to nonsensical intermediateimages in which spatial features are at best blurred anddo not exhibit realistic parallax. Within the frameworkof morphing technologies, EVENTS introduced a newparadigm of structure based view morphing, called JointView Triangulation (JVT) as described in [4–8]. This tri-angulation technique creates a dense map of point cor-respondences between views, and piece-wise smooth as-sumption is used to create triangle matches between twoimages.

3. Related to 3D reconstruction techniques, but not involv-ing explicit 3D structure, is the notion of point transfervia the so-called multifocal tensors. Here dense disparitymatches, together with geometric relationships betweenimages, are used to create physically valid novel views.The idea of using point transfer for visual tracking andnovel view generation within EVENTS has been evalu-ated in [9, 10].

Image-based rendering: Various ideas from the so-called image-based rendering can achieve photo-realisticresults, but suffer from a number of significant drawbackswith respect to EVENTS’ goals. The so-called lumigraph[11], or light-field was developed as a representation of ascene, which enables relatively easy generation of novelviews with photorealistic results. However, its computa-tional and storage requirements limit its application in dy-

namic scenes. An alternative, which has become popularin recent years is based on the observation that a pointin a novel view should be rendered in a colour that isconsistent with all the physical views. This ‘naive’ state-ment hides the technical detail of what it means to be con-sistent, but the eventual algorithm implicitly computes adepth for every point in the scene by looking for a match,which is consistent in a large number of views. Gener-ally such methods are slow, requiring orders of minutesor even hours for a single view.

In EVENTS, we initially evaluated this technique [9]with the conclusion that its effectiveness is crucially de-pendent on having a large number of views, which arevery accurately calibrated (close to the idea of ‘space-carving’). Our experiments also revealed that for foot-ball scenarios, where there are large areas of uniformcolour/texture, a technique based on extracting the con-sistent colour at every point is highly ambiguous anderror-prone. However, the project has since developed anovel version of the algorithm that performs better onsmall numbers of real views (as few as 2 or 3), and whichis considerably faster. Though being too slow for directuse in EVENTS, it is likely to be useful for improvingthe quality of rendering the ‘static’ background.

By extending and combining some of the above men-tioned research paths, EVENTS was expected to im-prove over other existing initiatives like Kewazinga Corp.’sSwingView and Eyevision, that currently exploit hardwarebased view interpolation to create ‘Matrix’ like effects forprofessional applications. In contrast, EVENTS is a soft-ware only solution that offers better image quality, widerviewing angles and real-time interpolation as its most at-tractive features. This paper describes the different elementscomposing the EVENTS system. We start out in Sect. 2with an introduction to the computer vision algorithms usedin the project. Section 3 follows with a description of theEVENTS’ platform. The evaluation of the results obtainedand comparison with previous experiences are presented inSect. 4. We end in Sect. 5 with the conclusions and proposalsfor further improvement.

2 Computer vision software

In EVENTS, we have developed a new image interpolationmethod that automatically synthesizes new views from ex-isting ones, rapidly, robustly, and realistically. The methodfirst started with a so-called ‘joint view triangulation’ (JVT)[12, 13] for the complete image; later, it was improved by ap-plying JVT independently to motion-segmented backgroundand foreground; finally it evolved to a fully layered JVT in-terpolation.

The first step of the computer vision chain describedin Fig. 1 consists of automatic matching of correspond-ing points between views for sufficiently textured images,leading to a quasi-dense matching map. Next, a generalised

238 T. Rodrıguez et al.

Fig. 1 The Computer vision chain. Background and foreground layersare computed separately and then rendered together

motion-segmentation method segments the scene into lay-ers; one of them being the static background. Subsequently,JVT is computed independently for background and fore-ground layers and a robust algorithm is invoked for sepa-rating matched from unmatched areas and handling of thepartially occluded areas. Finally, we developed an originalalgorithm for image warping, which renders background fol-lowed by foreground JVTs.

2.1 Joint view triangulation

Since image interpolation relies exclusively on image con-tent with no depth information, it is sensitive to changesin visibility. Therefore, to handle the visibility issue, weproposed a multiple view representation called Joint ViewTriangulation (JVT), which triangulates simultaneously andconsistently two images without any 3D input data. JVTcombines the best of the structure-based approach by com-puting dense disparity matches but bypassing the explicitconstruction of a 3D model. Instead an intermediate JVTrepresentation captures the scene structure and a novel-rendering algorithm creates a novel intermediate view. TheJVT is composed of two main steps: quasi-dense disparitymap construction and merging and triangulation.

2.1.1 Quasi-Dense disparity map construction

An initial pre-processing step is applied with the objectiveto: remove black boundaries, reduce resolution, detect pointsof interest and correlate between images using reduced max-imum disparity (1/10) due to repetitive textures. The methodstarts from extracting points of interest, which have the high-est textureness, from two original images using a Harris cor-ner detector. Then we use a zero-mean normalised cross-correlation (ZNCC) to match the points of interest acrosstwo images. This gives the initial list of point correspon-dences, or matches, sorted by the correlation score.

Next, the points of interest are used as seeds to propa-gate the matches in its neighbourhood from most textured

(therefore most reliable) pixels to less textured ones, lead-ing to a quasi-dense matching map. At every step, the seedmatch with the best ZNCC is removed from the list. Then,for each point in a 5 × 5 neighbourhood window centred atthe seed match in the first image, we use again ZNCC toconstruct a list of tentative match candidates in a 3 × 3 win-dow in the neighbourhood of its corresponding location inthe second image. Successful matches (i.e. when the ZNCCscore is greater than some threshold) are added simultane-ously to the match list and the seed list while preserving theunicity constraint. The process is repeated until the seed listbecomes empty.

2.1.2 Merging and triangulation

The brute quasi-dense matching disparity map may still becorrupted and irregular. Although the unconstrained view-interpolation approach makes no rigidity assumptions aboutthe scenes, we also assume that the scene surface is at leastpiece-wise smooth. Therefore, we will use local geomet-ric constraints encoded by planar homography to regularisethe quasi-dense matching by locally fitting planar patchesin both the images. The construction of the matched pla-nar patches starts by partitioning the first image into regularsquare patches of different scales. For each square patch, weobtain all matched points within the square from the quasi-dense matching map and try to fit the corresponding matchedpatch in the second image. A plane homography H betweenthe two images is tentatively fitted to the matched points tolook for potential planar patches. Because a textured patch israrely a perfect planar facet, the putative homography for apatch cannot be estimated by standard least squares estima-tors. Hence, a robust RANSAC method have been adopted,which provide a reliable estimate of the homography even ifsome of the matched points of the square patch are not ac-tually lying on the common plane. For each RANSAC trial,four matches are selected in the neighbourhood of the fourcorners of the square patch in the first image (see Fig. 2).Then we count the number of matches in the square compat-ible with the tentative homography defined by the selectedfour matches. RANSAC trials are continued until we find thebest homography, i.e. the one which maximises the numberof inliners.

Fig. 2 Matches 1, 2, 3 and 4 have been selected by the RANSAC trialand define the optimum homography, which maps the square patch inimage 1 to the distorted patch in image 2


Fig. 3 Every square patch is divided in two triangles. Then, triangulation is incrementally constructed starting from two triangles in each image.Consistent matched triangles (gray) are added incrementally in the triangulation row by row from top to bottom. The set of matched trianglesgrow simultaneously in both images in a coherent way

This process of fitting the square patch to a homographyis first repeated for all the square patches of the first imagefrom the larger to the smaller patches. Since the resultingmatched patches are not exactly adjacent in the second im-age, we next perturb the vertex locations to eliminate smalldiscontinuities and overlaps between the patches.

The previous steps produce a globally incoherent set ofmatched patches because of intersections. The merging stepselects one of these and converts it to an incomplete but co-herent JVT (i.e. a one to one correspondence between allvertices and contours). A JVT is then grown simultaneouslyby successive merging of the previous matched patches inthe two images.

The algorithm (Fig. 3), described in detail in [6, 8, 13,14], can be summarised as follows:

1. The JVT starts from two triangles in each image.2. Starting from left to right, top to down in the first image,

each matched planar patch is divided in two triangles andincrementally inserted into each triangulation if the coor-dinates of its three point match(es) in the two images areconsistent with the current structure. A new point matchis consistent if its two points are corresponding verticesof a match of the contour, or are both outside the set ofmatched triangles and must also verify that the resultingconstrained edges would not intersect a current contour ineither image. Otherwise, the triangle is left unmatched.

3. The structure is improved in a last step by further check-ing if the remaining unmatched triangles could be fit-ted to an affine transformation. If an unmatched trianglesucceeds in fitting an affine transformation, its label ischanged from unmatched to matched.

After computing the JVT (Fig. 4), a robust algorithm isinvoked for separating matched from unmatched areas andhandling partial occlusions.

2.2 Motion segmentation

Representing a moving scene as a static background anda moving foreground (Fig. 5) is a powerful, though by nomeans a novel idea. Within the context of EVENTS, such asegmentation has two principal benefits: it helps avoid thedanger of foreground objects being ‘joined’ incorrectly tothe background in the novel view (common in triangulation

Fig. 4 Example of a JVT for the left and right images

Fig. 5 Left to Right: Original Left, Interpolated in between, OriginalRight. Up to down: Segmented, Background, Rendered results

based methods such as JVT and Taylor’s), or foregroundregions being obliterated by unmatched regions, by explic-itly rendering a background JVT followed by a foreground.Also, since JVT for the background is unchanged over time,considerable savings can be made by only computing fore-ground triangulation for each frame. More generally, JVTrendering is prone to errors in regions where there are oc-cluding boundaries in the scene, and this applies both tomoving and static regions. Furthermore, the same problem(albeit on a smaller scale) is still exhibited when two movingforeground regions occlude one another (as often happens ina crowded penalty area).


Our approach was to generalise the foreground/ back-ground method to a layered segmentation and tracking ofnon-rigid objects in one or more views using a probabilis-tic model of object occupancy and visibility in order to rea-son about object locations and occlusions. The method ismost significantly distinguished from previous layered mod-els [15, 16] by its use of multiple views across time. Workin layered video has mostly concentrated on a single cam-era and independently moving objects; i.e. temporal seg-mentation. In other less proliferous work there have beenattempts to achieve spatial segmentation into layers (typi-cally for stereo or moving cameras). In EVENTS we tackledboth. Although other methods have variously used appear-ance models, motion models, new proposals for layers, etc,our method uniquely combines all of these and could furthersupport shape models.

We developed a method to segment automatically a pairof densely matched views into layers based on compliancewith a homographic model of transfer. Our basic premisewas that the scene can be represented as a collection of n +1 independent depth ordered layers (homographies for thebackground, homographies or affinities for each foregroundobject), which in general, may overlap and therefore occludeeach other. According to this model, the value of an observedimage pixel is generated by the foremost or visible layer atthat point.

The layer model (Fig. 6) at time t is denoted as Lt =(L0

t , L1t . . . Ln

t ), where Lit = (Oi

t , Ait , �

it ) are the param-

eters of the i th layer. Occupancy (Oit ) and Appearance

(Ait ) are computed from the input images, while Alignment

(�it = {φi j

t }, one per view) encode the transformation relat-ing the coordinate frame of the layer to each view.

An observed image I jt in the j th view is generated at

time t from a mixture distribution, where the value of eachpixel x is sampled according to the realisation of a randomvariable described by the appearance model of the foremost

Fig. 6 Layered model parameters: Oit is a probabilistic map denoting

the shape of the object; Ait is an intensity map representing the appear-

ance of the pixels composing the object; �it is a transformation relating

the coordinate frame of the layer to each camera view; V it is computed

from the Occupancy of all layers using relation (2).

layer at x :

P(I jt (x)

) =n∑

i=0

P(I jt (x)

∣∣V jt (x) = i

)P

(V j

t (x) = i)

(1)

The foremost layer is obtained with the aid of the viewdependent Visibility indicator V j

t and the probability that aparticular layer i is visible in the j th view can be computedin terms of the occupancy of all layers:

P(V j

t (x) = i) = Oi

t

(φ

i jt

−1x) n∏

k=i+1

[1 − Ok

t

(φ

k jt

−1x)]

(2)

On the other hand, the observed intensity is assumed to bedistributed normally, conditioned on the Visibility and hasmean given by the aligned Appearance map:

P(I jt (x) | V j

t (x) = i) ∼ N

(Ai

t (φi jt

−1x), σ 2

I

)(3)

New views can be generated simply by renderingeach layer appropriately, and the fact that visibilities areprobabilistically represented yields beneficial smoothing atboundaries. If we know which layer is visible at each pixel,then our problem is partitioned into n+1 sub-problems. Themethod models visibilities as hidden variables and uses anEM-algorithm to solve the problem. We wish to compute theparameters of the layered model that maximise the posteriorlikelihood of current image pair, given the previous layeredparameters using a Bayesian network that takes the form of ahidden Markov model. The joint probability of our Bayesiannet can be factored as:

P(L0)

t∏

τ=i

P(Iτ | Lτ )P(Lτ | Lτ−1), (4)

where the function to maximise is: F(Lt ) = ln P(It | Lt ) +P(Lt | Lt−1). In our case, the E-step corresponds to:

Q(k)(V ) = P(V | It , L(k−1)

t), (5)

while the M-step is denoted by:

L(k) = argmaxLt

∑

V

Q(k)(V )ln P(V, It | Lt )

+ ln P(Lt | Lt−1) (6)

We partition the M-step into three parts, in which thealignment, occupancy and appearance for each layer arecomputed in sequence. (See Ref. [10] for implementationdetails).

At each time t a set of new views is obtained fromthe cameras, and pixel intensities/colours in each image arecombined with a prediction from the previous frame (basedon both a motion model and a measurement of the visualmotion from a simple affine tracker); i.e. the use of a mo-tion model enables consistent tracking even under extreme


Fig. 7 The steps shown are performed as one cycle per frame. How-ever, the EM steps may be iterated. We found that two or three itera-tions are sufficient

occlusion. The layer parameters are propagated from thosecomputed at the previous time according to the mode of theprior distributions. This procedure acts much like a predic-tion and serves as the starting point of the EM-algorithm(see Fig. 7). The next stage is to reconsider the order of themodel, i.e does the model explain the data well and if notshould there be additional layers? New objects are initialisedautomatically when dense clusters of unexplained pixels ofa given minimum size are found. The new layer is initialisedby setting its occupancy to 0.8 inside the region and takingcurrent image pixel values as appearance. Once new layerproposals have been dealt with, we can solve for the new pa-rameters. A more detailed description of the implementationof the EM-algorithm can be found in [10, 17].

The background layer is composed using constrainedanalysis to account that various frames are coherent in time.Assuming the fixed field of view, we estimate the static back-ground model and then compute the View Interpolation forthis static background. We performed various experimentsto estimate the reliability of our background estimations us-ing a mosaicing technique. These experiments showed thatgiven enough interest points are present on the static back-ground, background can be estimated reliably. Robust statis-tical methods were employed to further improve our estima-tion by taking into account the information from all viewssimultaneously.

Tackling panning, tilting and zooming cameras is a nat-ural extension to the described procedure. Here, the back-ground does not appear as static anymore and it becomesnecessary to estimate the camera motion parameters (pan,tilt, and zoom) in order to be able to characterize image re-gions corresponding either to the background or the fore-ground. Within this task, a mosaicing technique was im-plemented, which proceeds as follows: the image-to-imagetransformation parameterized by pan, tilt, and zoom isroughly estimated by localizing a few static points in thescene and tracking them through the image sequence. This

transformation is then optimized such that textures comingfrom different images perfectly overlap. Based on three con-secutive images, dynamic regions are detected. Finally a mo-saic composed of static regions from several images is built.

2.3 View interpolation

In-between images are obtained by shape interpolation andtexture bleeding of the two original images, which are theendpoints of the interpolation path. We have developed asimple OpenGL interface, which renders background JVTsin real-time and is compliant with visibility constraints pre-viously defined (i.e. layers are rendered from down to top).This is done via geometric transfer using the trifocal tensorand a JVT of two views.

The background JVT is rendered using the pseudopainter’s algorithm described in [12]. Here, the triangula-tion of both the original images is warped to the intermedi-ate view and rendering starts with the unmatched, followedby the matched triangles. Finally a weighted blending of thetextures is performed.

Our chosen approach [9] to foreground image-basedrendering is summarised in Fig. 8. To choose a foregroundcolour to render at a particular point x in the novel view, weobtain in the nearby cameras the epipolar lines of the line ofsight through x . The pixel colours along these lines are sam-pled and rectified into the canonical frame of one referenceimage from the input set. The rectification is equivalent to atrifocal transfer as described in [18]. In the rectified frame,colours are compared between images at each depth, to givea colour consistency measure, representing the likelihood ofscene structure at this depth being the source of the renderedpoint. For this purpose, we use a simple 5 × 5 normalisedintensity gradient texture classifier. To avoid loosing lowfrequency variations in images, we employ a scale spaceconcept, where searches for texture consistency in theepipolar lines is done at multiple scale resolutions. Can-didate depths are isolated first at low resolution and thenpropagated to higher resolution in a region around thechosen depth to refine the solution. Consequently, the costof the search has been reduced, while the method incor-porates global changes in the images that would otherwisebe overlooked by the use of local operators only (Fig. 9).The algorithm is implemented using a pyramid in the inputimages and increases the efficiency of the scale spacealgorithm by generating multiple resolution novel viewsand retaining the likelihood maxima found at each pixelfor propagation to the next layer. In practice, the algorithmrenders coarse level pixels only when required to by theneed to render nearby pixels at finer resolutions. See Refs.[9, 19] for implementation details.

3 EVENTS platform

EVENTS is composed of two main applications: the ImageInterpolation Engine and the Interpolated Video Player. The


Fig. 8 Image-based rendering process. Epipolar lines in the real cameras are sampled and rectified. The novel viewpoint will be typically betweenthe real cameras, but has been placed to one side for clarity

Fig. 9 Rendering an intermediate novel view from 2 cameras 13 mapart. Note that the reconstructed novel view preserves both high (pitchlines) and low (grass stripes) frequencies

first tool processes video data captured from multiple cam-eras and produces the stream of interpolated images. Thesecond tool offers the viewer the possibility to change theview position and display the output of the Interpolation En-gine, thus allowing independent production and playback ofinterpolated video material.

The user interface of the Image Interpolation Engine(Fig. 10) implements the following functions: aid the user to

Fig. 10 Screen shots of EVENTS’: a Image Interpolation Engine andb Interpolated Video Player

install and configure the interpolation engine, operate the un-derlying computer vision software and monitor in real timethe interpolated images stream. Among others, the tool en-ables the user to: select and link cameras from the providedgraphical interface, start/stop the interpolation engine, visu-alise input images from the different cameras configured,modify computer vision parameters and select manual seedmatches. This last option is seldom required since automaticmatching is sufficient in most cases.

The Image Interpolation Engine platform has beenimplemented by means of a high performance computer-cluster with real-time communication support, based on theScalable Coherent Interface (SCI) environment. In orderto assure the required time constrains, a channel-oriented,component-based, real-time communication middleware,with bandwidth reservation support, has been developed ontop of an SCI-cluster.

The Player is a downloadable Java application (Fig. 10)aimed to display interpolated images streams in real time. Inaddition to the usual video controls, the Player presents twosliders that enable the viewer to modify the view positionin time and space respectively (i.e. change the virtual view-point of the interpolation process). The GUI also providescontrols to move automatically to predefined viewpoints orto swap automatically through the full range covered by thecameras with selectable steps. The player runs over a stan-dardised media format that basically contains the JVT, plusoriginal images data. In this way, since the player takes careof the final computation of the rendering, changing the viewpoint is possible while still minimising the amount of infor-mation that needs to be pre-computed and optionally trans-mitted from the Interpolation Engine.


4 Results and evaluation

A typical EVENTS scenario (Fig. 11) consists of a numberof cameras distributed in a football stadium with maximuminter-viewing angle of 20◦. For practical reasons the systemis arranged in two independent units, each covering a goalarea, and controlling between 5 and 10 cameras. EVENTSuses low cost cameras that must be carefully aimed at thegoalmouth and 18 yard area to ensure sufficient overlappingbetween cameras. Using images acquired at Real Madridand Oxford United stadiums, several tests were arranged toassess the suitability of EVENTS system for the target appli-cation purposed, which according to specifications consistedof a fast replay system able to produce 30 s of delayed in-terpolated video in less than 60 s at 12 img/s and 800 × 600resolution. The quality of the resulting images was evaluatedin terms of image resolution, frame rate, colour consistency,severity of interpolation artifacts, image definition, etc.

The effective image resolution that can be obtained bythe system is only limited by the resolution attainable fromthe input images (the cameras) and the computer power re-quired to fulfill performance requirements. The optimumresolution will depend on the application scenario selected.For TV broadcasting applications the minimum resolutionconsidered was 640×480, while maximum resolution wouldbe 800×600 (EVENTS standard). Higher resolutions wouldimply increased processing requirements without noticeablequality improvements; while lower resolutions would resultin poor images. For Internet applications, however, resolu-tions of 320 × 240 and lower could be accepted.

Once fixed the standard image resolution, the frame ratecan be scaled with available computing power. EVENTS de-fines 12 img/sec as the adequate balance between cost andquality (higher rates can be achieved as more processorsare added or individual processors are faster). According tothese requirements, the system must be able to process 360images in no more than 90 s with a maximum latency of60 s. Figure 12 shows the target can be achieved using a4 nodes cluster. Other combinations of resolutions versus

Fig. 11 Possible camera configurations for: 5, 6 and 10 cameras

Fig. 13 a Original left. b Interpolated. c Original right

Fig. 12 Performance for different image resolutions (target 4 img/s)

number of processors can be seen in the figure: i.e. a fiveprocessor machine is able to compute in real-time full framerate (25 img/s) interpolated video at 320 × 240.

In terms of image quality, EVENTS interpolated imageswere, in general, crisp, correctly focused and showedadequate accuracy in the details (see examples in Fig. 13).No noticeable blurring has been detected in the tested imagesequences and irregular borders caused by the interpolationprocess were automatically removed by the system. Whengenerating novel views, it is common to have differentillumination levels in each of the cameras. EVENTS usestechniques for automatic illumination compensation andblending of textures to ensure colour consistency. However,when analysing individual images, minor inconsistencies(Fig. 13) could still be detected close to the borders, wherediscrepancies between image pairs is greater. The effect ismore apparent in homogeneous areas, such as the green,and depends on the view position: the closer you are to areal camera view, the less noticeable. Scenes with strongsun lighting are also more prone to these type of problems.On the other hand, the effect is hardly perceptible in movingimage sequences since the eye is highly tolerant to errorsthat do not affect the dynamics of the scene.

Due to the intrinsic nature of the problem, no knowncomplex image processing process (i.e. interpolation, com-pression) is free from image artifacts. The important is-sue is to determine how the mentioned artifacts could af-fect the perceived image quality. It is obvious that as moredefects are produced, affecting greater areas of the image,during more time, the resulting quality gets more degraded.Defects continued in time are more noticeable than sporadic


defects, especially in dynamic scenes when some small ar-tifacts ‘are’ routinely tolerated by the eye (i.e. MPEG2 ar-tifacts). However, concentration of wrong pixels, especiallyif they affect systematically certain classes of objects, maydisturb seriously the development of the scene. This is thecase, for example, when defects concerns the dynamic partsof the scene, or objects of special relevance (i.e. active play-ers). In that sense, EVENTS’ mission is to produce highlydynamic and realistic image sequences that are pleasant tothe eye. For this purpose, the following criteria for successwas used to evaluate EVENTS’ results:

1. Large artifacts must be eliminated anywhere in the scene.2. All artifacts must be removed around players participat-

ing in the action.3. All artifacts must be eliminated at the ball.4. Small artifacts are tolerated around inactive players and

the background.

Results in Fig. 13 show active players are reasonablyrendered most of the time, the ball is correctly tracked andonly small defects appears on players off the main scene;hence the system complies with requirements: 2, 3 and 4.However, some large defects still remain in the static part ofthe scene (i.e. the goal post in Fig. 13). Those artifacts inthe background can be eliminated to a high degree using atool to segment manually the offending parts before launch-ing the interpolation engine. Attempts to achieve this objec-tive automatically are still under development. On the otherhand, EVENTS successfully handle situations that plagueother interpolation approaches: i.e. players close to the endof the field of view suddenly appearing in scene, errors aris-ing from occluding views, abrupt changes from image to im-age, rendering incorrectly the background, etc.

5 Conclusion

An innovative view interpolation system has been pre-sented. The paper outlines how it is possible to exploit theadvantages of JVT in sports scenarios while compensatingits shortcomings using novel multi-layered video segmenta-tion and constrained scene analysis. EVENTS did importantcontributions in an number of computer vision processes ofgeneral application: we developed a Wide Base Novel ViewSynthesis algorithm using JVT to triangulate simultaneouslyand consistently two images without any 3D input; we im-plemented a method based on a probabilistic model of objectoccupancy and visibility to segment and track non-rigid ob-jects in one or more views; we implemented an EM methodfor automatic segmentation of a pair of densely matchedviews into layers based on compliance with a homographicmodel of transfer; we developed a novel Background Esti-mation Method using constrained analysis to exploit tempo-ral coherency.

The results obtained demonstrate EVENTS correctlyhandles most of the situations caused by occluding bound-

aries and improper seed matching due to dominant tex-tureless features. As compared with the state of the art,summarised in Sect. 1, EVENTS displays: crisper interpo-lated images, better colour consistency, fewer artifacts andsmoother transition between cameras. The system offers asoftware only solution that allows flexible camera distribu-tion and wider viewing angles (20◦ compared with 7◦ orless), resulting in less cameras required for the same area.The computing platform is scalable and may perform in arange of resolutions and frame rates according to the needs.

Sports is a demanding environment for view interpola-tion. EVENTS was initially aimed at football, but other ap-plication scenarios have been also tested successfully: ath-letics, icehockey, etc. EVENTS technology might have animpact on any application demanding arbitrary novel viewsin real time: live events (theater, concerts, TV contests), spe-cial effects, games, etc. Moreover, some of EVENTS’ un-derlying technologies, such as improved JVT and real-timesegmentation methods would be suitable for applicationsrequiring high quality real-time segmentation in dynamicscenarios dominated by a background. The system is espe-cially well-suited for outdoors where there exist a numberof potential applications: video matting (either for highlight-ing, removal or replacement of objects), intrusion detection,tracking of simultaneous objects (tracking persons, trafficmonitoring), etc. Compression technologies may also ben-efit from the technology; i.e. MPEG-4 and its successors arelayer based, and rely on similar techniques.

At this moment, several research paths are open with theaim to reduce more the presence of interpolation artifactsand come closer to broadcasting image quality: improvingreliability of automatic seed matching using recent ideasfrom scale invariant features; merging image-based render-ing and JVT with the aim to improve the rendering of thestatic background; designing a layered segmentation methodthat combines multi-view appearance and motion models,and could further support shape models if required; extend-ing EVENTS’ features to rotating and zooming cameras; etc.

References

1. Zitnick, C., Kang, S., Uyttendaele, M., Winder, S., Szeliski, R.:High-quality video view interpolation using a layered representa-tion. SIGGRAPH (2004)

2. Xiao, J., Shah, M.: Tri-view morphing. Comput. Vision Image Un-derstanding (2004)

3. Vedula, S., Baker, S., Kanade, T.: Spatio-temporal view interpola-tion. Eurographics Workshop Rendering (2002)

4. Horaud, R., Dalal, N.: Indexing key positions between multiplevideos. In: Proceedings of the IEEE Workshop on Visual Motion.Florida, USA (2002)

5. Horaud, R., Dalal, N.: From video sequence to motion panorama.In: Proceedings of the IEEE Workshop on Visual Motion. Florida,USA (2002)

6. Lhuillier, M., Quan, L.: Robust dense matching using local angglobal geometric constraints. IAPR 00 (2000)

7. Lhuillier, M., Quan, L., Tsiu, T., Shum, H.: Relief mosaic by jointview triangulation. In: Proceedings of the IEEE Computer Visionand Pattern Recognition Conference. Hawaii (2001)


8. Quan, L., Lu, L., Shum, H., Lhuillier, M.: Concentric mosaic(s),planar motion and 1d cameras. ICCV

9. Connor, K., Reid, I.: Novel view specification and synthesis.In: Proceedings of the 10th British Machine Vision Conference.Cardiff (2002)

10. Connor, K., Reid, I.: A multiple view layered representation fordynamic novel view synthesis. BMVC03 (2003)

11. Gortler, S., Grzeszczuk, R., Szeliski, R., Cohen, M.: The lu-migraph. In: Proceedings of the 23rd Conference on ComputerGraphics and Interactive Techniques, pp. 43–54 (1996)

12. Lhuillier, M.: Joint view triangulation for two views. Vision Inter-face 99 (1999)

13. Lhuillier, M.: Towards automatic interpolation for real and distantimage pairs. INRIA, Technical Report 3619 (1999)

14. Lhuillier, M., Quan, L.: Edge-constrained joint view triangulationfor image interpolation. CVPR’00 (2000)

15. Xiao, J., Shah, M.: Motion layer extraction in the presence of oc-clusion using graph cut. In: Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition (2004)

16. Ayer, S., Sawhney, H.: Layered representation of motion videousing ro-bust maximum-likelihood estimation of mixture modelsand mdl encoding. In: Proceedings of the International Conferenceon Computer Vision (1995)

17. Connor, K., Reid, I.: Tracking occluding layers in multiple views.University of Oxford, Department of Engineering Science, Tech-nical Report 21125-0401 (2004)

18. Laveau, S., Faugeras, O.: 3d scene representation as a collectionof images and fundamental matrices. Inria, Technical Report 2205(1994)

19. Knight, J., Connor, K., Reid, I.: Image based rendering usingscale-space. University of Oxford, Department of EngineeringScience, Technical Report 21125-0412 (2004)

Image interpolation for virtual sports scenarios

Documents

Transcript of Image interpolation for virtual sports scenarios