Range Image Acquisition with a Single Binary-Encoded Light Pattern

8/13/2019 Range Image Acquisition with a Single Binary-Encoded Light Pattern

1/17

148 I E E E T R A N S A C T I O N S O N PAT T E R N A N A LY SI S A N D M A C H I N E I N T E L L I G E N C E . V O L .12. NO. 2. F E B R UARY 1990

Range Image Acquisition with a SingleBinary-Encoded Light Pattern

P. V U Y L ST E K E A N D A. OOSTERLINCK

Abstrac t -Th is pape r addresses the p rob lem of s t r ike iden t if ica t ionin range image acqu is i t ion systems based on t r iangu la t ion wi th per iod-ica l ly s t ruc tu red i l luminat ion . T he ex is t ing so lu t ions to s t r ike iden t i f i -ca t ion re ly on subsequen t p ro jec t ion o f mul t ip le encoded i l luminat ionp a t t e rn s . T h e se c o d i n g te c h n i q u e s a r e r e s t r i c t e dto s ta t ic scenes. Wepresen t a novel cod ing scheme based on a s ing le f ixed b inary encodedi l luminat ion pa t te rn , which con ta ins a l l the in fo rmat ion requ ired toiden ti fy the ind iv idual s t r ikes v is ible in the cam era image . Every sa m-p le po in t ind ica ted by the l igh t pa t te rn , i s mad e iden t i f iab le by meansof a b inary s igna tu re , which is loca l ly shared am ong i t s c losest ne igh-bors . The app l ied code is der ived f rom pseudono ise sequences, and i ti s op t imize d as to make the iden t i fica t ion fau l t - to le ran t to the la rgestex ten t . Th is instan ta neou s i l luminat ion encod ing scheme is su i ted fo rmoving scenes.

We r e a l iz e d a w o r k i n g p r o t o t y p e m e a su r e m e n t sy s te m b a se d o n t h i s

cod ing p r inc ip le . The measurement se tup is very s imple , and i t in -vo lves no moving par ts . I t consis ts o f a comm erc ia l s l ide p ro jec to r an da CCD - c a m e r a . A minicomputer system is cu r ren t ly used fo r easy de-ve lopment and eva lua t ion . Exp er imen ta l resu l ts ob ta in ed wi th them e a su r e m e n t sy s t e m a r e a l so p r e se n t e d . T h e a c c u r a c y o f t h e r a n g emeasu reme nt was found to be exce l len t . The me thod is qu i te insensi tiveto re f lec tance var ia t ion , geometr ic d is to r t ion due to ob l ique su r faceinc l ina t ion , and b lu r o f the p ro jec ted pa t te rn . Th e majo r l im i ta t ion o fth is i l luminat ion cod ing scheme was fe l t to be i t s suscep t ib i l i ty to theeffec ts o f h igh con trast tex tu red su r faces .

Zndex Terms-Depth maps , e r ro r -cor r ec t ing codes, pseudono ise se-quences, r ange- f ind ing , space-encod ing , s t ruc tu red l igh t , 3 -D v is ion ,t r iangu la t ion .

I . INTRODUCTION

RIANGULATION with structured light is a well es-T ablished technique for acquiring range pictures ofthree-dimensional scenes. The spatial position of a pointis uniquely determine d as the intersection of a straigh t linewith a plane, of three planes, oras the intersection of twostraight lines within a plane. The se geometric principleshave been applied in wide range of simple triangulationtechniques, using a directional illumination device (a pro-jector), and a directional sensor (a TV-camera). Th e basicschemes use a directed light beam to indicate a single ob-ject point [11-[3], o r a light sheet to indicate a cross-sec-tion of the visible scene surface [4]-[6]. An additional

Manuscript received May 4, 1988; revised August 21, 1989 . Recom-mended for acceptance by J . L. M undy. This work was supported by theBelgian Ministry of Science Policy.

P. Vuylsteke was with the E.S.A.T. Laboratory, Faculty of Engineer-ing, Catholic Universityof Leuven , Leuven , B elg ium. He is now with theR&D Div is ion , Agfa-Gevaer t N.V. , B2510 Mor tse l , Belg ium.

A. Oosterlinck is with the E.S.A .T. Laboratory, Facultyof Engineer-ing, C atholic University of Leuven , Leuven , B elg ium.

IEEE Log N umber 8931853 .

deflection mechanism with one or two degrees of freedommakes it possible to cover the full field of view. Varioustechniques have been developed to improve the accuracy[7], to expand the range of depth [8] or the field of view[9], or to realize a compact design [lo],[1 13.

These triangulation methods w ith a simple illuminationpattern yield unambiguous measurement results, but theacquisition speed is rather low, optomechanical deflectiondevices tend to be delicate, and the available sensor sur-face is sparsely used. O ne could consider to project a fanof light sheets instead of a single one in order raise theacquisition spe ed [121, but this extension is not obvioussince the light stripes are not individually recognizable inthe camera ima ge, and c onsequently it is not known fromwhich light sheet they originate. T his identification prob-lem is illustrated in Fig. 1; it is similar to the well-knowncorrespondence problem in stereo vision [13]. If a pointgrid (or an orthogonal line grid) is projected instead of aparallel line grid, then the am biguity may be alleviated bytaking into account the discrete nature of the projectedgrid [141, [151, or it may even be eliminated by applyingthe epipolar constraint several times in a dyna mic config-uration where the projector is successively rotated[161.The latter technique requires that the observed grid pointsbe tracked from frame to frame.

The identification problem has been solved in a directway by encoding the illumination pattern. T he intensityratio method [17] is based on monotonic m odulation oflight intensity across the scene. An additional referenceillumination is required to cancel the effect of varying re-flectance. Tajima used a single monochroma tic illumina-tion pattern with gradually increasing wavelength (i.e. , arainbow pattern), in combination with a two-channel colorcamera [181. Both techniques provide for high lateral res-olution (limited by the numb erof camera pixels), but ac-ceptable range accuracy can only be obtained with a cam-era combining e xcellent response stability w ith very largedynamic range. Furthermore it is not obvious how to dealwith the effects of mutua l illumination from n earby re-flecting surfaces.

These inconveniences are greatly reduced by imple-

menting the illumination pattern with only two intensitylevels, bright and da rk. It is still possible to pro-vide for sufficient discrimination between say2 - 1stripes with only two intensity levels by projectingn bi-nary patterns in succession. This means that every pointto be identified is marked with a sequence ofn bits adding

0162-8828/90/0200-0148 01 OO 990 IEEE


2/17

V U Y LS TEK E A N D O O S TERLI N CK :R A N G E IM A G E A C Q U ISITIO N I49

\camera \

\\

Fig . I . The identification problem with multiple stripe triangulation.Anobserved image pointP , on a light stripe may correspond to anyof theprojected light sheetsII . The index value j is required to compute thecoordinates of the actual object pointP .

up to a unique codeword. A few alternative schemes ofthis time-sequential encoding technique have been d e-scribed in literature, using an encoded point grid insteadof a line grid [19]; a Hamming [20] or Gray code [21]instead of a simple binary co de. Th e effects of reflectancevariation may be canceled by making use of additionalreference illumination patterns (full bright, full dark), oreven a set of complemen tary patterns [22]. Very accuratealignment of the subsequent pattern masks is generallyobtained with a programmable liquid crystal shutter win-dow. These range acquisition systems seem to performexcellently, but they are not suited for mo ving scenes dueto their time-sequential nature. For an y movem ent be-tween two subsequent coding steps would result in erro-neous identification.

It is therefore essentialto

restrict the encoding processto a single instance (e .g. , by mea ns of a flash light source),such that all the information required for identifying theindicated scene points is available in a single camera pic-ture. In the scheme proposed by Boyer and Kak [23] asingle colored stripe pattern is projected on the scene.Only very few different colors are used, typically three orfour. The illumination pattern can be thought of as a con-catenation of identifiable subp atterns (typically six stripeswide). Th e recognition of any valid subpattern codewordin the observed stripe sequence leads to the identificationof the stripes in that subpatter n, and possibly also of someadjacent stripes (by some kind of crystal growing pro-cess).

Our approach which w e shall present next show s some

similarity to the latter, in that the indexing information islocally distributed. However while the encoding schemeof Boyer and Kak is based onadjacent subpatterns, as-signed to fixed positions in the stripe pattern, we devel-oped a code grid in which identifiable subpatternsover-lap. This means that any window of some fixed minimalsize in an arbitrary position of the illumination grid will

return a codeword which uniquely identifies the columnindex. Our option is motivated by the inherent lack ofsynchronization, i .e., the boundary between adjacent sub -patterns is not explicitly available in the fixed subpatternapproach. T he conceptual difference between adjacent andoverlapping subpatterns is also very important in the per-spective of fault-tolerance: in [23] the color sequence isassigned to the line grid in such a way that ther e is a spec-ified minimal code distance between every pair of (fixedboundary) subpatterns. How ever, since the decoder is notinformed about the subpattern boundaries, real fault-tol-erance may only be guaranteed if one specifies a minimalcode distance between every pair of window s at arbitrarygrid column positions. These considerations have led toa novel binary illumination-encoding schem e, which weshall present next. First we focus on the basic principlesof our en coding technique, and the resulting illuminationpattern layout. The third section deals with the variousaspects of an ex perimental measurement system based onthis encoding technique. Experimental results are pre-sented in the last section.

11. ILLUMINATION-ENCODINGASED N PSEUDONOISESEQUENCES

We have investigated a space-encoding scheme basedon the following target specifications. 1) A squa re grid ofpoints is projected on the scene.2) The projected gridpoints must b e easily detectable in the camera image, andit should be possible to accurately estimate their imagecoordinates U , ) (cf. Fig. 1). 3) The index va lue j of anobserved grid point must be identifiable ( i. e. , the columnnumber of the grid points that are projected on the sceneby the plane n j ) . n a calibrated measurement setup theknowledge of the image coordinates U , ) and the indexj is sufficient to compute the spatial position of the cor-responding object point.4) All this information must be

supplied by means of a single binary illumination pattern.With binary modulation it is very hard to associate morethan a few bits with every grid point to make it identifi-able. One could consider some kind of shape-m odulation,which alters the height, width, orientation, compactness,or the like to discriminate between individual grid points.It should be noted how ever that most of these features arenot invariant with respect to perspective distortion. More-over they are very susceptibleto noise if the grid spacingis very small (only a few pixels in the camera image). Wedecided therefore to assign only asingle codebitto everygrid point. With this low information density it is stillpossible to identify the corresponding index value, bysystematically distributing the binary information amongneighboring grid points in such a way that every window

of adjacent grid points of some specified size yields acodeword which uniquely identifies the index of the gridpoint. This means that the grid points are identifiable onlywithin their local context.Thus for valid identification itis required that the original contextof a point in the pro-jected grid be preserved locally in the camera image. Thisassumption holds true almost everywhere in the image,


3/17

150 I E E E T R A N S A C T I O N S O N PAT T E R N A N A LYS I S A N D M A C H I N E I N T E L L I G E N C E .VOL 12. N O 2 , F E B R U A RY 1990

except at ju mp boundaries, where the projected bit mapwill be locally disturbed. Any identification windowwhich contains such a context disruption must not be usedfor index identification.

Ideally any context disruption should be detectable bythe fact that the observed window value is inconsistent,i .e., that it is not compatible with the projected bit map.An optimally designed binary code grid should thereforecomply with the following requirement: a specified de-gree of fault-tolerance must be guaranteed with the small-est possible identification window size. The degree offault-tolerance may be expressed as the minimal numberof binary positions in which any two w indows (taken fromarbitrary columns in the grid) differ, i .e., the minimalHamming distance between any window pair. We devel-oped a highly redundant code grid based on pseudonoisesequences, the details of which w ill be elaborated in Sec-tion 11-B. First, we shall present a modulation schemesuited for the implementation of such a binary illumina-t ion code.

A . Basic Layout of the Illumination Pattern

An appropriate illumination pattern with only two in-tensity levels which 1) indicates a grid of points and2)marks each grid point individually with a single codebit,is depicted in Fig. 2 . Note the basic chess-board appear-ance. By definition, the grid points to b e indicated o n thescene are located at the intersection of the horizontal andvertical chess-board edges, i . e. , where four squares mee t.Upon projection every column generates a planeITj thein de xj of which must be know n for locating the indicatedobject points by triangulation. In our experimental mea-surement system the entire pattern consists of 63 columnsof 64 points each (only10 rows are shown).

For the purpose of identification every grid point is in-dividually m arked with a bright o r a dark spot, represent-ing a binary 1 o r 0, respectively. Additionally thechess-board naturally exhibits two basic appearances ofthe grid points, which alternate both in horizontal and ver-tical direction. We denote them as either+ o r -.Hence one may th ink of th e complete pat tern to be com-posed of the fou r primitives depicted inFig . 3 .

The assignment of individual codebitsto the grid pointsis based on two binary sequences{ Ck } and {bk }, of 6 3bits length. The grid points on thek th column of the +type are assigned the binary valueck while those of the

type are assigned bk as illustrated in Fig. 4.As aconsequence the bit map assignment is repeated everysecond row. With a proper choice of the sequences{ck }and {b k } every window of2 X 3 grid points at any ar-bitrary grid position carries a 6-bit codeword (composed

of an ordered ck-triplet and a bk-triplet) which is in one-to-one correspondence with the underlying column num-b e r o r i n d e x j = k . This means that an identification win-dow size of 2 X 3 is sufficient for unique identification.Th e actual choice of the sequence pair is very critical withrespect to the uniqu eness of the identification-if a largerwindow is used it will also influence the degree of fault-

-, ,

0 I 62I

Fig. 2 . Detail of the projected binary pattern.

O+ 0- 1+ 1-

Fig. 3 . The entire illumination pattern is com posedof four primitives.

1 3 b4 5 b6 9 8 %I b1 c11bf2c13b~4c15b16c17

Fig. 4 . Two binary sequences are bit-wise assigned to the grid such thatevery window of 2 X 3 bits has a unique signature in every positionwhen it is shifted horizontally. Pointsof the + type are marked witha bit of the {c k } sequence, and those of the type with a bit of{b k } .The assignment is repeated every second row.

tolerance-but this topic is postponed till the next subsec-tion. At present we simply assume that such sequencesexist.

We now concentrate on the main characteristics of theillumination pattern. The choice of a chess-board as thebasic structure has been motivated by so me typical ad-vantag es. F or a specified grid density its bandwidth isonly half that of a point grid (or a parallel line grid, ifonly one direction is considered). The grid points in thechess-board are defined at the intersection of the horizon-tal and vertical ed ges. In g eneral it is very difficult t o es-timate the exact location of an edge in an image withoutscene-dependent bias, since the observed intensity profileacross an edge may vary according to the local surfaceinclination and reflectance. I n the case of a chess-board,however, the horizontal position of a grid point is definedby two vertical edg e segments (meeting at the grid point)which have opposite sign. In the sam e way two horizontaledge segments with opposite sign define the vertical po-sition of the interjacent grid point. If on e assumes the sur-face characteristics to be locally smooth, then the ob-

served edge locations will be biased in op posite sense byequal amounts, both for the horizontal and vertical edgepairs, so these offsets will tend to cancel out in estimatingthe location of the grid points. Notice that for triangula-tion the precision of the range m easurement critically de-pends on the accuracyof the grid point coordinates in thecamera image.


4/17

V U Y LS TEK E A N D O O STER LIN C K : R A N G E I M A G E ACQUISITION 5

The specific choice of the additional modulation whichsuperimposes a bright o r a dark spot at every grid positionhas been motivated by the following arguments. First, de-ciding wh ether a spot is bright o r dark in the camera im-age is largely facilitated by the fact that both a bright anddark reference is locally supplied at any grid position. Ifthe evaluation of the spot brightness (and hence the code-

bit value) is made relative to some function (e.g., themean) of these reference intensities, then the effects ofreflectance variation and ambient light may be canceledout, as long as these parameters vary smoothly within thereference neighborhood. Second, the actual image posi-tions where the codebit primitives (i.e., the spots) haveto be ev aluated are clearly indicated since the latter co-incide with the grid vertices. A third argum ent concernsthe interference between the basic modulation com ponentfor locating the grid points and the additional componentfor marking them with a codebit. It is clear that the ob-served gray-level distribution in the immediate neighbor-hood of a grid point will depend on the actual value of thecorresponding codebit. However, the spot being bright ordark will not bias the estima tion of the grid point location ,since the gray-level function is locally point-symmetricwith respect to the grid point in either case. This is themain reason why we rejected some other binary modula-tion schem es in which the identification primitives didnorcoincide with the grid vertices.

The proposed identification scheme based on local bi-nary signatures requires that the network of grid points belocally reconstructed in the camera image. The chess-board structure facilitates the search of adjacent gridpoints in horizontal and vertical direction, since every pairof 4-neighbors is connected by a high co ntrast edge seg-ment. Without this pictorial information 4-neighbors anddiagonal neighbors might be confused in regions wherethe observed pattern is highly distorted. In fact it is suf-ficient to consider that immediate horizontal or verticalneighbors should have opposite sign (in the sensementioned earlier).

The cod e grid has been designed such that any grid pointmay be identified by the signature of a2 x 3 windowcovering its immediate neighbors. Consequ ently, any pairof adjacent rows of the grid carry different code infor-mation. In general, if the identification should be basedon a t X b window form at, then the bit map must be or-ganized such that any t consecutive rows provide for in-dependent information. This requirement may be achievedby repeating the bit assignment everyt rows. In the sim-plest case when the bit map is composed of equal rowsr= 1 ) the appropriate identification window format wouldbe very elongated (covering at least6 adjacent grid points

in horizontal direction, if63 column positions have to beidentified). We mentioned however that a window willfail to be identified if it is disrupted by a jump boundary.Obviously a compa ct window is less likely to be disruptedthan an elongated one. This is the main reason why wepreferred a ( 2 X 3)-identification scheme instead of( 1x 6 ) .

Although the latter approachof distributing the identi-fication data among t subsequent rows leads to a morecompact window format, at the same time it introducesadditional ambiguity. At any given column position anidentification window may exhibitr different signatures,depending on its row position. In fact shifting a windowover the bit map in vertical direction causes its rows to

rotate. Without special precautions the same signaturemay be found at different column positions, if both win-dows do not belong to the sa me row. This problem is il-lustrated in Fig. 5 , with r = 2. Although the Hammingdistance between any p air of windows on the same row isat least 2 , they cannot be uniquely identified unless theiractual row index (modulo2) is known. One could subjectthe bit map d esign to additional constraints, such that aminimal Hamming distance between every pair of win-dows is guaranteed, irrespective of their relative row po-sitions, but this would increase the window size for aspecified minimal distance. The ambiguity is elegantlysolved for r = 2 by taking into account the alternate ap-pearances of vertices in the chess-board (indicated as + or -) . With the specific assignment as in Fig. 4the ck-bits n a window may alw ays be distinguished fromthe 6,-bits irrespective of the vertical window position,since they are assigned to grid points of opposite sign. Inother words the additional sign information makes it pos-sible to reorder the contents of any arbitrary window to astandard six-tuple, in which ck- and bk-bits are at fixedpositions. This way both windows indicated in Fig. 4would correspond to the same 6-tuple( ( c3 ,c4, c 5 ) , b3,b,, b5)).

B. Bit Map Assignment

For any arbitrary identification window format one mayalways define an equivalent n-tuple with a specified orderof c k - and b,-bits, n being the total number of bits in thewindow. The window even need not be rectangular. Forsome specified window format the corresponding n-tuplemay be regarded as a codeword, and the set of all thecodewords which are obtained by considering the windowat every horizontal position o n the bit map defines a code.Every window format defines a distinct code. If the iden-tification is to be performed with a window of some spec-ified format, say 2 X b bits, then a pair of binary se-quences { ck} and {b } should be found such that thecorresponding code is optimal, in the sense that it has thelargest possible minimal Hamming distance. Finding anoptimal code of rather small dimension in general is nota problem, since the best ones have been exhaustively de-scribed in the literature [24]. However, the codes to be

found in our specific case are subject to a strong addi-tional restriction, due to the fac t that the identificationwindows overlap. This implies that the codewords cannotbe chosen independently.

We investigated the application of a special kind of re-cursive sequences, called pseudonoise sequences, togenerate an optimal bit map . A binary sequence generated


5/17

152 I E E E T R A N S A C T I O N S O N PAT T E RN A N A LY S I S A N D M A C H I N E IN T E L L I G E N C E . VOL 12. N O 2. F E B R U A R Y 1990

011 111 10101100110111011010100111000101111001010001100001000010 1~0 10 010 01 11 000 1 11 1 01010001 100001000001 11 11 1010101 100101 11 11 1010101 1001 101 11 ~01001110001011110010100011000010000

Fig. 5 . Ambiguity may arise from the fact that the grid code is composedof more than one sequence: the indicated windows belongto differentcolumns, although they carry the same signature.

by the recursion (modulo2 ) :

m

ck = c - h ic k - i , k m ( 1 )i =with initial valuesc o, c l r , c,- l r is an mth order PN-sequence if the corresponding characteristic polynomialdefined as:

m

h ( x ) = C hkxk, ho = 1, h, = 1 ( 2 )

is an irreducible factor ofx k - 1 for k = 2 - 1, but notfor any smaller k , which means that it is a primitivepolynomial. If the initial values are not all zero then theresulting sequence will have periodp = 2 - 1, w hichis the maximal period length achievable with a linear re-

currence. For more details on PN-sequences we refer to[ 2 5 ] and [26]. Two properties deserve particular atten-tion. By the window property every group ofm con-secutive bits is unique over an entire period of the se-quence. This is clearly the required property for uniqueidentification. Secondly the set consisting of th e p-bitwords obtained by cyclically shifting a given period of aPN-sequence, extended with the all-zero wordis a cycliccode having dimensionm , ord lengthp = 2' - 1, anda constant Hamming distance d = 2 -' [ 2 7 ] ,whichmeans that any pair of words exactly differ by one morebit than the number of positions in which they do agree.'

Now let {ck} be an mth order PN-sequence, and{b k }= {ckpV1 the same sequence with a phase differencecpand let these sequences be assigned to the bit map as il-lustrated in Fig. 4. Then the code defined by applying awindow to every position of the bit map is equivalent tothe code which results from puncturing the simplex codedefined by the PN-sequence. This means that the resultingcode is obtained by systematically deleting specified bitpositions in the original simplex code (for more infor-mation on puncturation see[ 2 8 ] ) .Every window formatcorresponds to a specific puncturation.* If the window sizeis 2 X b then in the punctured code all bit positions willbe deleted except for two groups ofb bits wide, separatedby (o - b positions. Th is is illustrated in Fi g.6 .

Starting from a simplex code with Hamming distanced- 2 -1 , it is clear that the minimal distance will eitherdiminish by one, or will be unaffected, every time a bit

position is deleted. In the special case whencp = 6 , the

k = O

'This kind of equidistant cyclic code is called a simplex or minimalcode.

*In fact, the m oving window m ay not genera te every code word of thepunctured simplex code, unless the bit map is cyclically extended at its leftand right borders, such that the windowis defined over the entire horizontalrange of p positions. The all-zero word is also m issing.

cp=5

CJ + 0 1 1 1 1 0 1 0 1 1 0 0 1 0 01 1 1 1 0 1 0 1 1 0 0 1 0 0 01 1 1 0 1 0 1 1 0 0 1 0 0 0 11 1 0 1 0 1 1 0 0 1 0 0 0 1 11 0 1 0 1 1 0 0 1 0 0 0 1 1 1

bk] + 0 1 0 1 1 0 0 1 0 0 0 1 1 1 11 0 1 1 0 0 1 0 0 0 1 1 1 1 0

simplex code

0 1 1 0 1 01 1 1 1 0 11 1 1 0 1 11 1 0 1 1 01 0 1 1 0 00 1 0 0 0 11 0 1 0 1 0

punctured code...

Fig. 6 . A simplex code of dimension 4 n = 15), and the resulting punc-tured code of length n = 6 , when a 2 x 3 window is applied to a gridwhich is com pose d of the first and the sixth word of the simple x code(i.e . , two periods of the corresponding PN-sequence with a relative phaseshift p = 5 ) .

punctured code will consist of2b adjacent positions of theoriginal simplex code. This situation is equivalent to ap-plying a row-window of2b bits wide to a bit map con-sisting of equal rows defined by a single PN-sequence.For a g iven PN-sequence and window size,cp is the onlyparameter which influences puncturation. Consequentlythe minimal cod e distance resulting from an optimal twin-row organization will be at least as large as the distancecorresponding to a single-row schem e, since in the form er

approach puncturation may be camed out with an addi-tional degree of freedom.We examined the distance behavior of sixth-order PN-

sequences (m = 6 ; p = 63 ) and the influence of the rel-ative phase shift cp. In a twin-row organization uniqueidentification will require a window size of at least2 x3 . Hence our first criterion states thatd6 = 1, w here d26denotes the minimal Hamming distance obtained with a2X b window. Where possible the decoding scheme willuse a larger window size, so that local signature incon-sistencies may be detected . Therefore we also maximizea second criterion in orde r to optimize the bit map simul-taneously for a set of larger window sizes, more specifi-cally for the range between2 X 4 nd 2 x 8 :

8 d Z b + m - 1is maximal.

b = 4 2 X b 3 )

This criterion is derived from the Singleton distancebound, which states that any code of word lengthn , di-mension m, and distanced ( i . e . , a n [n , m, d ]-code) mustsatisfy d + m - 1 ) /n . In appl icat ions l ike th iswhere the number of essentially different PN-sequences(only three of sixth order) and the number of meaningfulphase comb inations are rather small, it is feasible to findthe optimal sequence combination by exhaustive search.According to the abo ve criteria one sixth-order sequence-phase combination was found to score better than all theother ones, i .e., the PN-sequence with characteristic

polynomial h (x) = 1 + x + x 6 , combined with the samesequence shifted by cp = 17 positions:{C k ) = 1111 11000001000011o0o10100111101o0o

1110010010110111011001101010

{b k } = {c k - 1 7 ) . (4)


6/17

V U Y L S T E K E A N D OOSTERLINCK: R A N G E I M A G E ACQUISITION 153

TABLE 1C O MPA R ISO N OF T H E C O D E S O B TA I N E D B Y A P P LY I NG W I N D O W S OF S I Z E S( 2 X 3)- 2 X 8 ) TO T H E B I TMAPG E N E R AT E DY TH E SEQ U EN C ES 4 ) ,

W I T H T H E BESTCODES E SCRI BE DN LITERATURE

window code best code of equalw e distance dimension m=6, and2xb d2b d>b+5)12b equal length n=2b d+ 5)/n

2x3 1 1 [6,6.11 1

2x4 2 0875 I8 6 21 0 8752x5 3 0 8 10.6,3] short Hamm 0 82x6 4 075 l12.6,41 0 752x7 4 0 643 [14,6,5] short BCH 0 7142x8 5 0625 116,6,61 0 688

Moreover, a better performance was obtained in compar-ison with the single-row schem e over the equivalent rangeof window sizes 8-16 in 2-increments), for any of thethree sequences. This demonstrates that an identificationscheme based on rectangular wind ows, besides yieldingmore compact windows, also may result in a higher de-gree of fault-tolerance. The codes generated by applyingthe windows of sizes 2 X 3 through 2 X 8 to the bit mapspecified by the optimal sequence pair(4) are compared

in Table I with the best codes described in literature [24].Although these codes simultaneously result from a singlebit map, they prove to be optimal up to the window size2 x 6 , in the sense that no known codes of equal wordlength and dimension have a larger Hamming distance.We also investigated the use of nonlinear recursive se-quences and combinations of different PN-sequences, buttheir distance behavior was found to be worse. M ore de-tails are supplied in [29].

111. E X P E R I M E N TA LA N G EMEASUREMENTYSTEMWe realized a prototype based on this binary encoding

principle. The measurement setup is very simple and itinvolves no moving parts. T he basic geometry is sketchedin Fig. 1. The illumination pattern depicted in Fig. 2 hasbeen recorded on a 24 X 36 mm Ortho-negative and isprojected on the scene with a standard slide projectorequipped with a 25 0 W xenon bulb and a90 m m / f 1 : 2 .5objective. The entire pattern consists of64 rows by 63columns. The camera (High Technology Holland, MXtype) has a MOS-CCD frame transfer image sensor of6 0 4 H ) x 5 7 6 V ) pixels. It delivers a 625-line CCIRvideo signal (linear response, constant gain), which issampled and digitized to a512 X 512 by 8 bit digitalpicture with 1: 1 aspect ratio. T he focal length of the cam-era objective ( 16 mm/ f 1 : 1 . 8 ) has been matched to theprojection optics, such that approximately the entire il-lumination pattern is visible when it is projected on afrontal surfa ce at nominal de pth. In that case the observedgrid period roughly corresponds to8 pixels in both direc-tions.

The digitized picture is processed by aVAX-750 mini-computer. Demodulating and decoding an illumination-encoded image comprise the following steps:1) detectingthe grid points, estimating their precise location in theimage, determining their appearance (sign) in the

chess-board, and their individual codebit indicated by abright or dark spot; 2) searching for the closest N-,S- ,W-, and eastern neighbors of every detected grid point,i .e. , recovering the local grid context in the image; and3) identifying grid points by their local signature, usingdynamic windows of some specified minimal size, andchecking fo r signatu re consiste ncy. Identified points are

next spatially located by triangulation. We developed aprogram package which also includes routines for geo-metric calibration, and diagnostic and display routines forthe presentation of interm ediate and final results. Th e pro-grams are written in Pascal. T he flexibility of a high levelenvironment with many I/O-facilities makes the devel-oped software package a powerful tool for evaluating theperformance and limitations of the new range acquisitionmethod.

The processing time is quite long(50 s for a range im-age of 64 x 63 points) , but 80 percent of this time isspent to low-level picture proc essing, which is well suitedfor hardware implementation. The size of the completeprogram package is of the order of10 000 lines of (com-mented) source code; the specific routines for processing

the encoded im age and fo r triangulation comprise onlysome 1000 lines. 1 . 3 MB of mem ory are needed for datastorage, including the input im age. In the following sub-sections we outline the relevant characteristicsof the sub-sequent processing steps.A more extensive algorithmicdescription is presented in [29]. For the purpose of illus-tration consider the picture of Fig.7 which represents ascene consisting of a vase, a teapot and a cup illuminatedby the coded pattern. We denote this single input pictureby the discrete functiong u , U ) , 0 g u , U ) 255, 0

U 5 11 , O U 511.

A . Demodulation

correlation with the reference pattern:

\

The detection of grid points in the image is based on

1 9

- -- 1 - 1 - 1 0 1 1 1

- 1 - 1 - 1 0 1 1 1

- 1 - 1 - 1 0 1 1 1

0 0 0 0 0 0 0

1 1 1 0 - 1 -1 - 1

1 1 1 0 -1 - 1 - 1

1 1 1 0 - 1 - 1 -1- -- 1 0 0 0

0 0 0 0

= 1 / 9 [ i ;I [0 0 0 00 0 01 0 0 0 :]1 ( 51


7/17

154 I E E E T R A N S A C T I O N S O N PATT E RN A N A LYS I S A N D M A C H I N F I N T E L L I G E N C E .VOL 12. NO 2. F E B R U A RY 1990

digit ized picture is theonly input required for computing the correspond- (a)ing range image

which is actually implemented as a3 x 3 local averagingoperation followed by correlation with a5 x 5 sparse ref-erence pattern. If we denote the local average image byg u , U ) , then the correlation image is obtained by per-forming the local o peration:

CO = g-2,2 + g2.-2 - g-2,-2 - F2,2 6)where the subscripts indicate the relative pixel offsets withrespect to the reference point of the considered neighbor-hood. This intrinsic im age will exhibit extrema at the gridpoints, the negative peaks corresponding to grid points ofthe - ype and the positive peaks to the + typegrid points.

Every pixel which meets the following requirements isretained as a grid point:

IC, is a local maximumAND 7 )

8 )

9 )

/ C O ( > TI ( 1 % - 2 . 2 - g2.-21 + J g - 2 . - 2 - g 2 . 2 AND

where the local maxima are considered within centered7X 7 neighborhoods, and To and T I are threshold param-eters. Th e heuristic term 8) is unaffected by offset or gainchanges in the input image, since these effects cancel outat both sides of the inequality. The adaptive thresholdexpression at the right-hand side of(8) also improves se-lectivity, for it represents the degree of asymmetry of thepicture function along both diagonals through the candi-date grid point; it will appro ach zero if the opposite brightand dark squares are equally bright, whichis a typicalcharacteristic of a grid point obs erved without se vere dis-tortion. T he net effect of this criterio n is illustrated in Fig.8(a). For comparison, Fig. 8(b) shows the response ofpure Correlation. A rather low fixed threshold To in thelast term (9) is applied additionally for eliminating falsematches in image regions of very low contrast.

The precise location of a grid point may be estimatedby fitting a (e.g., quadratic) surface through the correla-tion data points in a small neighborhood around the cor-responding local maximum, and computing the analyticalmaximum of the fitted surface. We followed an alterna-tive approach based on the center of mass of the correla-tion peaks, which requires fewer computations. In fact the

(b)Fig. 8. Correlation imagesof the vase-teapot-cup scene (200 x 200 pixel

details taken from the center). (a) Responseof the operation co -z - 2 , zgz + R - z , - z- g2 ,21 ,ith T , = 1 and all negative

values cl ipped to black. (b ) Absolute value of pure correlation I c ( u ,1 J I .

center of mass is a good approximation to the correlationmaximum, and even coincides with it if the correlationfunction is symmetric with respect to its maximum. Sincethe projected pattern s hows the required sym metry the lat-ter condition will generally be met unless the object sur-face exhibits nonlinear reflectance variation in the im-mediate neighborhood of the considered grid point, e.g.,when the surf ace is strongly textu red. If( uo, v O ) epresentthe pixel coordinates of the local correlation maximum,then the subpixel offsets of the grid point coordinates areestimated as:

I

c c t c u 0 + x U 0 + y)).c c t ( c ( u 0 + x , U0 + Y ) )

c c + ( U 0 + ., 0 + Y ) ) Y. r = - l \ . = - I

6uo = I -

x = l v = - l

1 1

x = - l \ . = - I

= I - (10)

with t ( c ) = c - B c(uo ,v 0 ) if c - B c ( u o ,u0 > 0O otherwise.

The weight coefficients t ( c ) are computed by referringthe correlation values to some basic level, which is spec-ified as the Bth fraction of the local maximum, and clip-


8/17


9/17

156 I E E E T R A N S A C T I O N S ON PAT TE R N A N A LY S I S A N D M A C H I N E I N T E L L I G E N C E , VOL. 12. NO. 2. F E B R U A R Y 1990

eferencegrid polnt

Fig. 10. Finding the N- and W-neighbors of any grid point is confined toparameterized regions at fixed offsets.

illustrated in Fig. 10 . The actual width, height, and ori-entation which specify these search areas, and their po-sition relative to the reference grid point, a re derived fromstatistical data gathered during system calibration.

Pairs of direct neighbors (both in horizontal and ver-tical direction) ar e then selected accordin g to the least Eu-clidean distance criterion, taking into account that twocandidate neighbors must have opposite sig n, and oneof them lies in the others search area. For every detectedneighbor pair a (bidirectional) link is created which con-nects the corresponding nodes in the graph. The results oflocal grid reconstruction are displayed in Fig. 11. Afterthe linking process, all essential information is available

in a graph data structure, in which every node includesthe following data fields: the im age coordinates U , U fthe corresponding grid point ( up to1/4 pixel), i ts signand codebit, pointers to the N-,S-, W-, and E-neighbor,and corresponding flags for indicating which neighborsa re m i ~ s i n g . ~

The identification is implemented as a two-stage pro-cess. First a window of codebits is iteratively evaluatedat every grid point. Starting at the grid point to be iden-tified a signature window is grown essentially in the W-and E-directions, by repeated reference to the respectivepointers in the graph. If the horizontal expansion getslocked at some stage because the W- or E-neighbor ismissing, then an attempt is made to proceed by reroutingthe search via a single step in theN - or S-di rec t ion; e .g . ,

if the W- neighbo r is not available, then the W-search pro-ceeds along the NW-neighbor instead, or if the latter isalso missing, along the SW-neighbor. The expansion isequally continued in both horizontal directions, unlesseither of the search paths is exhausted, in which case thesearch proceeds only in the other direction. At every s tageof expansion one or two codebits are appended two thecurrent signature: on e associated w ith the considered gridpoint, and the other one drawn from its imm ediate N- orS-neighbor (if any). These bits, which correspond to atuple (ck ,b k ) f the PN-sequences which build up the grid

For the purpose of execution speed this graph structure is actually beingcreated while searching for the local correlation maxima. Both the pixelimage data structure and the graph are used until after the grid links havebeen established. Moreover the graph is organizedas an array of 128 x

128 nodes, such that every fourth pixel (in both directions) of the imagearray correspo nds to a node . It is thereby explicitly assu med that everyneighborhood of 4 X 4 ixels contains at most one grid point. Each nodeis provided with an additional Rag, indicating whether it effectively cor-responds to a grid point , or is empty. This organization of the graph, likea low resolution image array, makesit very straightforward to access theappropriate node starting from an image pixel. The reverse way is effec-tuated by recording the image coord inatesof a grid point (if any) in theappropriate field of the corresponding node.

Fig. 11. Result of establishing links between direct neighbors. Althoughthis information is displayed pictorially as physical segm ents connectingthe grid points, i t is actually stored by means of a graph data structure,the nodes of which represent the located grid points.

code, are the only information available along the currentgrid column, since they a lternate in vertical direction (F ig.4). They are mutually distinguishable by the sign of thecorresponding grid point. The actual identification of thegrid point under consideration is accomplished by itera-tive elimination. Initially every element of the integer set{0 . . . 62 } is considered to be a valid index num ber. T heknowledge of the binary code tuple at the start positionalready eliminates 3 /4 (o r 1/ 2 i f only one codebit i savailable) of the initial candida te index num bers. At everysubsequent stage of the iterative window expansion theacquaintance of an extra code tuple further reduces the setof valid index numbers. We implemented this eliminationscheme as follows: the set of valid index numbers forsome grid point at any iteration stage is represented by a

63-bit vecto r, which is initially set to all ones , indicatingthat any index is valid. The elimination is accomplishedby repeated AND-ing the latter with a vectorof the samedimen sion, representing t he set of index values (indicatedby ones at the corresponding positions) that are compati-ble with the observed code tuple. A lookup table is veryconvenient for gen erating these eliminationvectors, sincethe latter are entirely specified by the acquired code tuple,and the current horizontal offset within the search win-dow. The elimination procedure is terminated if:1) aspecified minimal number of codebitsN b have been ac-quired, or 2) the search gets locked in both directions,or3) a specified maximal offset in either theW - or E-direc-tion has been exceeded. After termination valid identifi-cation is obtained ifN b codebits could be acquired and if

the resulting index vector is all-zero except for one posi-tion, which then indicates the actual index number of thegrid point. An all-zero index vecto r points to the fact thatthe local signature is inconsistent. The error-detectionperformance is directly determined by the specified min-imal window sizeN b .

Mo st of the grid points are identified with this iterative


10/17

VUYLSTEKE A N D OOSTERLINCK: RA N G E I MA G E ACQUISITION 157

procedure. Near jump boundaries howev er many gridpoints may not be identified due to the limited expansionpossibilities,so that the iteration procedure is terminatedbefore Nb codebits could be acquired. In a second stagean attempt is made to identify these grid points by somekind of index propagation process. If the index numbersof the existing 4-neighbors of a nonidentified grid point

are mutually compatible, then the latter is identified ac-cordingly, at least if its (possibly incomplete) code-tupleis compatible with the proposed index value. T his prop-agation process is repeated a few times. Fig. 12(a) showsthe result of identification before propagation, and (b)after three propagation iterations.

C. Triangulation

equations given by:We modeled the camera according to the pinhole

[ t u tu t] = S[x y z 11 ( 1 2 )

in homogeneous coordinates , where the ( 3X 4)-matrixS is called the cam era matrix. T hese equations specify themapping of an object point in Cartesian coordinates(x,y , z ) with respect to an arbitrary world frame, to the cor-responding im age point coordinates( U , U ) , expressed inpixel units (cf. Fig. 1). These equation s include the trans-formation from world to camera coordinates, perspectiveprojection, rescaling, and misalignment corrections. Al-though the projector can be modeled exactly the sameway, we preferred to use explicitly the equations of eachof the 6 3 planes I I j through the grid columns and the lenscenter. These are of the form:

( 1 3 )

The first coefficient l \ J ) is set to one, so for every planeII, only three independent parameters need to be deter-mined. These equations ar e suited to represent any planethat is not parallel to the x-axis. O ne can rewrite the equa-tions (12) and (13) a s a set of three linear equations in theunknowns x, , z , and solve them nume rically, given theimage coordinates of a point U , ) , and the correspond-ing index j . It is however possible to solve this set ofequations symbolically,so the spatial solution can be ex-pressed a s a linear transform of the image coordinates:

x + l y y + l:z + l y ) = 0 , j = 0 . . . 62.

[ h x h y hz h] = M [ u u 11 ( 1 4 )

where the (4 x 3 )-matrix M the so-called sensor ma-trix) is computed in terms of the camera matrixS and thecoefficients I:, i = 1 . . . 4, for each of the 63 planesITj. This direct im plementation of triangulation, adoptedfrom Bolles, Kremers, and Cain [30], is much faster than

the conventional technique of numerically solving the setof equations ( 12) and (13 ) .

D. CalibrationComputing the components of the sensor matrices

M or a given measurem ent setup requires the knowl-edge of the camera matrix S and the coefficients I of

(b)

Fie. 12. Result of identification, a) before index propagation, with aspecified minimal window sizeN b = 9, and (b) after three propagationiterations. The identified index values are represented by graphical sym-bols. Only 16 different symbols were available, hence they indicate theindex modulo 16. Detected grid points which could not be identified arerepresented by a dot.

the planes IT,. A standard technique for computing thecomponen ts of the camera matrix, using a set of calibra-tion points with known world coordinates, has been re-ported by Rogers and Adams [31] . Given a set of N cali-bration points with observed image coordinatesU ; , i ),and known world coordinates x i , i , z; ), the cameraequations (12) can be rewritten as a set of2N nonhomo-geneous linear equations, involving the 11 components ofthe camera matrix as unknowns (the lower right compo-nent of S being arbitrarily set to one). For obtaining awell-conditioned set of equation s, the focal length mustnot be very large, and the calibration points (at least six)have to be nicely spread ove r the space of view. T he effectof measurement noise is reduced simificantlv if a verv


11/17

158 I E E E T R ANS AC T I ONS ON PAT T E R N ANALYS IS AND M A C HI NE I NT E L L I GE NC E . VOLI? N O 2. F E B R UARY 1990

large number of calibration points are being inv olved,sothe camera parameters are co mputed as the least squaressolution of a largely overconstrained set of equations. Inpractice how ever the use of many calibration points (a fewhundred) is rendered difficult due to the requirement thateach of them must be identifiable in the camera image.We solved this problem by making the calibration points

identifiable in exactly the same way as the grid points inthe encoded illumination pattern. The calibration setupconsists of a flat panel, parallel to the x-y-plane, whichcan be fixed at calibrated z-positions (cf. Fig.1). Thepanel surface is uniformly covered with a set of identifi-able points at calibrated( x , y)-positions. Hence for somespecified position z = 0) , the calibration panel actuallydejines the world coordinate system.A set of evenly dis-tributed calibration points is obtaine d by placing th e panelat a few calibrated positions across the focus range of thecamera, and taking a picture in each case.Fig . 13 showsthe layout of the calibration p anel, w hich is essentially thesame as the projected pattern used for triangulation, butin which only 63 isolated windows of 2X 7 grid pointsare made visible on a white background. By definition the

upper middle grid pointof every window corresponds toa calibration point, the( x , y)-coordinates of which haveto be accurately measured in advance. Each calibrationpoint is identified by the 14-bit signature of the surround-ing window. The identification is quite robust since theHamm ing distance between any arbitrary pair of windowsis at least 4. Lo cating and identifying the calibration pointsin the subseque nt images is accom plished with essentiallythe same routines as those used for demodulating and de-codin g illumination-encoded imag es. Moreo ver this cali-bration scheme makes it possible to consider a large num-ber of calibration points (u p to 2.52, using 4 i ma ges ), withminimal human intervention.

The second part of calibration consists of determiningthe coefficients I of the planes I I j . At this point we as-sume that the camera matrix has already been calibrated.A large set of calibration points is generated by projectingthe encoded illumination pattern on a white panel, sub-sequently placed a t a few calibrated z-positions within thefocus range of both cam era and projector, using the samemechanical equipment as in the case of camera calibra-tion. The world coordinates of these calibration points areuniquely solved from the inverse camera equations( 12).since the known constant z-value of the panel surface canbe used as additional constraint. These calibration pointsare next partitioned into 63 subsets of points with equalindex number, so every subset will consistof all calibra-tion points belonging to the same planeII,. Finally theplane coefficients Z:, are computed by fitting planes

through each subset of calibration points. The pointswithin a subs et are not collinear, sin ce they are distributedamong the intersection lines of the planeII with the cal-ibration panel, placed at subsequent depth positions. With4 anel pos itions, u p to 2.56 calibra tion points are used tocompute the corresponding least squares plane, which en-sures excellent noise immunity.

Fig. 13. Layout of the calibration panel. The63 windows are at knowncalibrated positions on the panel, andare mutually distinguishable.

IV. EXPERIMENTALESULTS

A series of elementary experiments have been camedout to estimate the measurement accuracy and the perfor-mance of the prototype under varying circumstances, in-cluding d ynamic depth ran ge, surface reflectance and ori-entation range, and the extent to which textured surfacesdo interfere with it . The sa me functional p arameter valueshave been used in the experiments described next:To =36; T I = 1; B = 0.75 (cf. (9), 8) , and (lo), respec-tively); the identification is accomplished with a minimalwindow sizeNb = 9, followed by three index propagationiterations. These parameter values proved appropriatethroughout our experiments. Moreover we found thatsmall deviations from these default values do not signifi-cantly affect the measurement performance (except forNb:when this parameter is enlarged fe wer points will be iden-tified, but the identification error rate will also decrease,and conversely;Nb = 9 or 10 yielded the maximum num-ber of correctly identified points in most experiments).

A , CalibrationWe verified the appropriateness of the linear camera

model by backprojecting the imaged calibration points(used for camera calibration) on the planes of constantzfrom which they originate (i.e., the calibrated panel po-sitions) . For panel positions ranging from 95-140 cmw.r .t . the c amera-projector base line, the RMS-deviationin the panel plane between the actual calibration pointsand the backprojected points varied from 0.1- 0.2 mm .These figures are indicative of the small nonlinear lens

distortion, and the high precision of the grid point loca-tion (as far as random errors are concerned).The range precision has been evaluated by measuring

points on the (white) calibration panel with the fully cal-ibrated system, and considering the distance offsetof theresulting range points with respect to the known panel po-sitions . In the case of a triangulation setup with a camera-


12/17

VUY L S T E K E A N D O O S T E R L I N C K : R A N G E I M A G E A C Q U I S IT I O N 159

projector base width of 39 cm , the RMS depth error wasfound to be 0. 3 mm; this measurement comprised some14 200 points lying in fou r planes, at distances be tween105-140 cm from the base line. Although the absoluterange precision might be worse, this measure still in-cludes the error components caused by nonlinear behavior(lens aberrations, grid distortion, panel curvature), and

noise in locating the grid points, which has been our ma-jor concern.

B. Range LimitationsMaking use of a flat test scene we were ab le to evaluate

the measurement system quantitatively under extrem e cir-cumstances. In each of these experiments a least squaresplane is fitted through the obtained range points. Twobasic parameters are considered to be indicative of themeasurement performance: the RMS value of the distanced of the range points with respect to the fitted plane, andthe total number of range pointsNcop ying within a dis-tance to the plane of less than 2 mm. The first measureconcerns the measurement precision, while the secondfigure, i.e., the numberof coplan ar points, is almost equalto the number of grid points that have been correctly iden-tified, since any misidentification is likely to result in alarge range With Ndet representing the number ofdetected grid points in the image, andN d the number ofgrid points that have been identified (not necessarily cor-rectly), we also conside r the following ratios to be signif-icant: Ncop/Ndet, hich is the relative number of correctlyidentified points, and( Nld - Ncop) N l d , representing theerror rate of identified points.

In our experimental measurement system the dynamicdepth range is mainly limited by the focus depth of theprojector, due to the large (fixed) f-number 1 :2 .5. Al-though the projected pattern already s how s noticeable blurwhen the target is displaced only by a few centimetersfrom the plane of focus, the range measurement showedno remarkable degradation within the depth range from90-115 cm . Numeric al results (using a white frontal tar-get plane) are presented in Table 11. AlthoughNdet issmaller than 4032 outside the range100-105 cm , we ver-ified that all visible points have been effectively detectedin these experiments. The fact that many points are miss-ing outside this range is exclusively due to the limitedfield of view of the cam era. The same experiments wererepeated with the projecto r lens aperture reduced by a fac-tor of 4 (and the camera aperture enlarged accordingly).At 80 cm 73 percent of the points were correctly identi-fied, 99 percent at 85 cm , 87 percent at 125 cm, and 84percent at 1 30 cm . In the range 85-125 cm the identifi-cation error rate was less than 0 .9 percent.

The object reflectance range within which the system

4For that reason only points within a distance of2 mm to the plane areconsidered in computing theRMS deviation. Also the actual reference planeis determined iteratively, in such a way that anew estimate of the plane isbased on only the 50 percent most coplanar points obtained from the pre-vious iteration. This eliminate s the influenceof fatal errors on the fittedreference plane.

TABLE I1DEPTH ANGE F TH E MEASUREMENTY S T E MT 1 m N O M I N A LI S TA N C E .

C O P L A N A R )R I D OINTSRE TA B U LATEDS A FUNCTIONF TARGETD I S TA N C Ew . R . T. T H E B A S E L I N E ) ,S I N G F L AT R O N TA L ARGET. H ELASTC O L U M NH O W S T H E RMS D I STA N CEF T H E RANGE O IN TS TO TH E

FITTED EFERENCE LA N E

T H EN U M B E RF DETECTED,D EN TIFIED , A N D C O R R EC TLY ID EN TIFIED I . E . ,

Depth cm) N d e i N , d N m p N c o ,v d e t ( N id - N c o p )/ % RMS d (mm)

1438 776 -590 3310 3214 3191 0.96 0.007 0.295 3712 3708 3708 100 0.000 0.2

100 4032 4031 4031 1.00 0000 0 2105 4032 4031 4031 1.00 0 000 0 3110 3776 3757 3757 0.99 0000 0.3115 3520 3291 3291 0.93 0.000 0.4120 3300 2507 2434 0.74 0.029 0.6125 2841 1922 1502 05 3 0.219 0.7

keeps on working (with fixed lens apertu re) has been ev al-uated by means of a seriesof gray target planes, with rel-ative reflectance p varying from 15 to 100 percent, thelatter value corresponding to standard white paper. Allpoints were correctly identified within the reflectancerange 32-100 percent. Atp equal to 22 percent, 97 per-cent of the points were identified, with an erro r rate of 0.3percent. In this case the mean gray-level of the input im-age was only 16 units (o f 25 6), and theR M S value 17.The latter figures refer to smooth reflectance variation.We also evaluated the system behavior with respect totextured surfaces, making use of different textures on awhite (frontal) background. These included text patternswith characters ranging from 3.5to 14 mm (a t 0.8 m dis-tan ce), and vertical line patterns (2 .5 mm pitch and50percent duty cycle) with contrast ratio from 2: 1 to 8 : 1.The results are shown in Table 111. All grid points werevisible in the camera image. The considerable suscepti-bility to the effect of high contrast textured surfaces isclearly one of the major limitations of this coding tech-nique.

As long as the o bject surface is nearly perpendicular tothe bisector line of both optical axes (which we den ote by

frontal position ), and the angle subtended by the axesis not too large, geometric distortion of the observed gridwill be very small. Rotating the target about its verticalaxis by some angle0 away from the frontal plane causesthe observed horizontal pitch to decrease or increase, de-pending on the sign of$. Tilting the ob ject by$ about anaxis parallel to the base line causes the observed grid col-umns to dev iate from the vertical direction. W e examinedthe maximal range of 13 and $ within which the systemkeeps functioning correctly, using a flat target. The re-sults for the extreme target positions are shown in Fig.14. In the first case , Fig. 14(a), when the target plane wasnearly oblique w.r.t . the camera, 0 = -65 , the meanhorizontal pitch was4.1 pixels. The other extreme,0 =6 5 , Fig. 14(b), corresponds to a mean h orizontal pitchof 1 2.9 pixels. In Fig. 14(c) the plane was tilted by70 ,causing the observed grid columns to decline by 33O TheRMS distance d of the range points to the fitted referenceplane was: (a) 0.2 mm, (b) 0 .3 mm, and (c) 0.7 mm,


13/17

160 I E E E T R A N S A C T I O N S ON PAT TE R N A N A LY S I S A N D M A C H I N E I N T E L L I G E N C E . VOL. I ? . NO. 2. F E B R U A R Y 1990

TABLE I11I N FL U E N CEF T E X T U REN M E A S U R E M E N TE RFO RM A N CE .H EF R O N TA L

TA RG E TL A N Es AT 0.8 m FROM THE BA SE L I N E

Scene Ndei N NCO, Ncod4032 h cop)Nd RMS d (mm)

( 1 ) t ex t3 5 r n m 4032 3309 3190 0 7 9 0 0 3 6 0 3(2) text 7mm 4037 3530 3429 0 8 5 0029 0 4(3) text 14mm 4003 3437 3333 0 8 3 0 0 3 0 0 4(4) I n e s 2 5 m m contrast 8 1 3397 1637 1 4 4 5 0 3 6 0 117 0 5( 5 ) 4 1 3 9 6 3 1919 1697 042 0 1 1 6 0 4

6) 2 1 4032 3210 3073 0 7 6 0 0 4 3 0 3

respectively (at 156 cm nominal distance). We do notmention any quantitative results on the identification per-formance, because it was not clear how m any grid pointsactually were visible in the im age. N evertheless, the pic-tures plainly reveal the high detection rate, and negligibleidentification error rate [only two points in Fig. 14(b)].

C . Real ScenesAfter triangulation every identified grid point is spa-

tially located by its world coordinates( x , y , z ) . Thesespatial data are difficult to interpret however. Hence foreasy visualization the3-D grid points are subjected to aperspective transformation (the para mete rs of which areadjusted interactively, in order to get a nice view).Awire frame representation of the range image is obtainedby connecting neighboring grid points in the perspectiveview.5 The results displayed in Figs. 15-19 have beenacquired w ith a setup in wh ich the camera and th e projec-tor were tilted down by approximately 40 , and their op-tical axes subtended an angle of about 14 .Fig. 15 showstwo wire frame views of the range points acquired fromthe vase-teapot-cup scene. The axes of the superimposedworld coordinate frame [clearly visible in Fig. 15(b)], are10 cm long in reality. Fa ir results have been obtained witha scene consisting of a multicolor printed box and spraycans, as shown in Fig. 16. A camera picture of the scenewithout encoded illumination (b) has been added for thesake of clarity. Only the large text on the smallest side ofthe box significantly interferes with the range acquisition.Fig . 17 illustrates the effect of spatial texture. The surfaceenvelope of the long-haired brush and the su rface of thebricks are properly reproduced. Although the lateral res-olution of the range acquisition is inadequate to pick updetails of the keyboard inFig . 18, only a few identifica-tion errors have been encoun tered. Th e adverse influenceof specu lar reflection is apparent in F ig.19, where pointsof the overexposed corner regionof an untreated coach-work part are missing in the wire frame. F ig.20 illustratesthe suitability of this range acquisition method for mea-suring the human body. An application in detecting spinaldeformity is currently being investigated. The impracti-

cability of fixing the body during the measurement re-quires that the acquisition be carried outinstantaneously.This motivates the use of a nonsequential encoding tech-nique like the one described here.

'These connections actually correspond to the valid branches in the graphof grid points.

A = -650

. . .. . ... . . .. . . ,. . . .. . . . .. . . .. .1 mrn

+65

(I = +70

1 mrn

Fig. 14. Result images for nearly oblique target planes. Correctly identi-fied points are denoted by a spot, the areaof which is proportional to thedistanc e to the fitted reference plane . The standard measure at the lowerleft corresponds to 1 mm deviation. Noncoplanar points ( i .e. , more than2 m m eviation), are indicated by a box. (a), (b) Target rotated aboutitsvertical axis by k65 . c) Target tilted about an axis parallel to the baseline by 70 . The lower three rowsof boxes correspond to points on theedge of the 18 m m hick panel.


14/17

V U Y LSTEK E A N D O O STER LIN C K : R A N G E IMAGE ACQUISlTlOl

I

(b)

Fig. 15. Perspective views from two different view positions of the rangeimage acquired from the vase-teapot-cup scene. The range points (indi-cated by square dots) have been connected to enhance visualization.

V . CONCLUSIONS

This novel approach to illumination encoding makes itpossible to acquire a range image from a single camerapicture . In this respect the method differs from other trian-gulation techniques, which take their input from multiplepicture frame s. The basic snap shot approach makes themethod suited for applications in robotic assembly, whereseveral parts of the scene might be moving, or in medicaldiagnostics, where it is difficult tofix the human body.The smearing effect due to the frame integration time can

be entirely avoided by using high intensity pulsed illu-mination.Most of the difficulties caused by the initial restriction

of using only a single (monochrome) picture have beensolved successfully, and fair results have been obtainedwith an experimental measurement system.A few topicsneed further investigation however. It is clear that real-

161

1

Fig. 16. Scene consisting of two spray cans anda box . (a) Illumination-encoded image. (b) Picture illustrating the scene with normal lighting.(c) , (d) Perspective views of the range image.


15/17

162 I E E E T R ANS AC T I ONS ON PAT T E R N ANALYS I S AND M AC HI NE I NT E L L I GE N C E .VOL. 12. NO. 2 . FEBRUARY 1990

Fig. 17. A long-haired brush and two bricks. ( a) Illumination-encoded im-age. (b) Picture illustrating the scene with normal lighting. (c), (d) Per-spective views of the range image.

Fig . 18. A keyboard. (a) Picture illustrating the scene with normal light-ing. (b) Perspective viewof the acquired range image.


16/17

V U Y L S T E K E A N D O O S T E R L I N C K : R A N G E I M A G E A C Q U I S I T I O N Ih3

(b )

Fig. 19. A coachwork part . (a) Picture illustrating the scene with normallighting. (b ) Perspective view of the acquired range image.

time operation can only be achieved if computation isspeeded up by a factor of at least 300. In that respect thelow-level image processing routines have been conceived

to be easily implementable in hardware as a pipelineofdedicated proces sors, involving only basic arithmetic andlogical operations.

In the current prototype the range image size is limitedto 64 X 63 points, which may be insufficient for someapplications. One grid point corresponds to an averageof8 X 8 pixels, the input picture being 512X 512 pixels.By reducing this ratio one could improve the lateral res-olution of the range image; a few experiments that havebeen carried out in that perspective are reported in [32].Performance may further be enhanced by better condi-tioning of the video signal, and more efficient use of thesensor surface (only 512 X 390 sensor elements are ef-fectively used in the current prototype). The rather highsusceptibility to texture interferenceis perhaps the mosttypical limitation of this nonsequential encoding tech-nique. This problem could likely be alleviated by addingextra information channels (using a colored illuminationpattern and a color camera).

( b )

Fig. 20. Human ba ck. ( a) Illumination-encoded image.( b ) Perapectiveview of the acquired range image.

REFERENCES

[ I ] A. Kiesaling. A fast scanning methodfor three-dimensional scenes.in Proc. 3rd / t i t . Cotif. Pottcr)i Recogui t ron . 1976. pp. 586-589.

[2 ] P. R . Haugen , R. E . Keil. and C . Bocchi. A 3-D active vision sys-tem. Proc. Soc. Phoro-op~. nsrrurn. Engineers. v o l . 521, pp. 258-263, 1984.

[3] G . L . Oomen and W . J . P. A. Verb eek, A real-time optical profilesensor for robot arc welding, Pro( , . Soc.. of Photo-Opt. mrrurn.Etigiriecrs. vol. 449. pp. 62-71. 1983.

[4] G . J . Agin and T. 0 Binford, Computer description of curvedob-jects, l E E E Tr o n .Corriprrl vol. C - 2 5 . pp, 339-449. Apr. 1976.

[SI C . G . Morgan. J . S . E . Bromley, P. G . Davey, and A. R . Vidler.Visual guidance technique5 for robot arc-welding. Proc . Soc.

Photo-Opr. Insrrurri. Encqineers,vol. 449. pp. 390-399. 1983.[6] Y. Sato, H. Kitagawa. and H. Fujita. Shape measurementof curved

objects using multiple slit ray projections, l E E E Trnrrs. futrerir A n d .Muchirie / r i te / / . ,vol. PAMI-4, p p . 641-646. Nov. 1982.

[7] K . G . Harding and K . Goods on, Hybrid , high accuracy structuredlight profiler. Proc. Soc. Phoro-Opr. Imstritrti. Engineers. vol. 728 ,

[8] G. Bickel. G . Hausler. and M . Maul. Triangulation with expandedrange of depth. O p t . E i i g . . vol. 24 , pp. 975-977. N o v. 1985.

[9] M. Rioux. Laser range finder basedon synchronized scanners.A p p / . O p t . , vol. 23, pp. 3837-3844. Nov. 1984.

[IO] M . Rioux and F. Blais. Compact three-dimensional cam erafor ro-botic applications, J . opt. Soc.. A n ie r. A . vol. 3 , pp. 1518-1521.Sept. 1986.

[ I I ] M. Idesawa, A new tqpc of miniaturized optical range sensing

pp. 132-145. 1986.


17/17

164 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MAC HINE INTELLIGE NCE, VO L. I?. NO.2. F E B R UARY 1 9 9 0

scheme, in Proc. 7th Int. Conf. Pattern Recognition,Aug. 1984,pp. 902-904.

[I21 F. Rocker and A. Kiessling, Methods for analyzing three-dimen-sional scenes, in Proc. 4th Int. Joint Conf. Artificial Intelligence,

[13] S . T. Barnard and M. A . Fischler, Computational stereo,Comput.Surveys, vol. 14, pp. 553-572, Dec. 1982.

[14] J. B. K. Tio, C. A. McP herson, and E. L. Hall, Curved surfacemeasurement for robot vision, inProc . IEEE Con$ Pattern Rec-ognition and Image Processing ,June 1982, pp. 370-378.

1151 G. Stockma n and G. Hu, Sen sing 3-D su rface patches using a pro-jected grid, inProc. IEEE Conf. Computer Vision and Pattern Rec-ognition, June 1986, pp. 602-607.

[16] J. L abuz, Camera and projector motion for range mapping,Proc.

1975, pp. 669-671.

Soc. Pho to-Opt. Instrum. Engineers,vol. 728, pp. 2211234, 1986.B. Carrihill and R. Hummel, Experiments with the intensity ratiodepth sensor, Comput. Vision, Graphics, Image Processing,vol.32, pp. 337-358, 1985.J . Tajima, Rainbow range finder principle for range data acquisi-tion, in Proc. IEEE Workshop Industrial Applicationsof MachineVision and Machine Intelligence,Feb. 1987, pp. 381-386.M. D. Altschuler, B. R. Altschuler, andI . Taboada, Measuringsurfaces space-coded by a laser-projected dot matrix,Proc. Soc.Photo-Opt. Instrum. Engineers,vol. 182, pp. 187-191, 1979.M. M inou , T. Kanade , andT . Sakai, A method of time-coded par-allel planes of light for depth measurement,Trans. IECE Japan,vol. E64. DO. 521-528, Aug. 1981.

1211 S . Inokucii, K. Sato, and F. Matsuda, Range-imaging system for3-D object recognition, inProc. 7th Int. Conf. Pattern Recognition,

[22] K. Sato, H. Yamamoto, andS . Inokuchi, Tuned range finder forhigh precision 3D data, in Proc. 8th Int. Con Pattern Recognition,

[23] K. L. Boyer and A. C. Kak, Color-encoded structured light for rapidactive ranging, IEEE Trans. Pattern Anal. Machine Intell. , vol.PAMI-9, pp. 14-28, Jan. 1987.

[24] W. W. Peterson and E. J. Weldon, Jr.. Error-Correcting Codes.Cambridge, MA: M.I.T. Press, 1972, p. 124.

12.51 S . W olomb, Shift Register Sequences. San Francisco: CA: Hol-den-Day, 1967.

[26] F. J. MacWilliams and N. J.A . Sloane, Psuedo-random sequencesand arrays, Proc. IEEE, vol. 64 pp. 1715-1729, Dec. 1976.

[27] -, The Theory of Error-Correcting Codes. New York: North-Holland, 1978, ch. 14.

[28] E. R. Berlekamp,Algebraic Coding Theory. New York: McGraw-Hill, 1968.

1984, pp. 806-808.

Oct. 18 6, pp. 1168-1171.

[29] P. Vuy lsteke, Een meetsysteem voo r de acquisitie van ruimtelijkebeelden, gebaseerd op niet-sequentiele belichtingscodering, Ph.D .dissertation, Katholieke Universiteit Leuven, Dec. 1987.

[30] R . C. Bolles, J. H . Kremers, and R. A. C ain, A simple sensor togather three-dimensional data, S.R.I . Tech. Note 249, 1981.

[31] D. F. Rogers andJ. A. Adams, Mathematical Elements for ComputerGraphics. New York: McGraw -Hill, 1976, pp. 81-83.

[32] P. Van Ranst, Drie -dime nsionale beeldacquisitie met behulp van ge-structureerde belichting, M.Sc. thesis, Katholieke Universiteit Leu-ven. 1987.

P . Vuylsteke received the Candidate, M.S.E.E.,and Ph.D. degrees from the Catholic Universityof Leuven, Belgium, in 1977, 1980, and 1987,respectively.

From 1980 until 1988 he was with the E.S.A .T.Laboratory, Faculty of Engineering, at the Cath-olic University of Leuven. While there, he partic-ipated in realizing an image computer system, anddeveloped a new technique for 3-D image acqui-sition. In May 1988 he joined the R& D divisionof Agfa-Gevaert N.V., Mortsel, Belgium. Hisre-

search interests include computer vision and image analysis.

hundred internationalimage processing, picsua1 inspection, and t

A. Oosterlinck (S73-M80) received the Ph.D.degree from the Catholic University of Leuven,Belgium, in 1977, after having worked with sev-eral laboratories, including the Jet PropulsionLab . , PA. He is now a Professor of Electrical En-gineering at the Catholic University of Leuven,where he is director of the E.S.A.T. Laboratoryof Electronics, Systems, Automation, and Tech-nology. He is also the founder of an image pro-cessing company for automatic visual inspection.His work has been published in more than one

publications. His special research interests include:ture coding, medical imaging, robot vision, and vi-he development of vision hardware.

Range Image Acquisition with a Single Binary-Encoded Light Pattern

Documents

Transcript of Range Image Acquisition with a Single Binary-Encoded Light Pattern