Systematic information fusion methodology for static

and dynamic obstacle detection in ITS

Yajun Fang, Berthold K.P. Horn, Ichiro MasakiIntelligent Transportation Research Center

Microsystems Technology Laboratories (MTL)Massachusetts Institute of Technology, Cambridge, MA 02139, USA

yajufang@csail.mit.edu, bkph@csail.mit.edu, masaki@mit.edu

Abstract

Environment understanding technology is vital for intelligent vehicles that are ex-pected to automatically respond to fast changing environment and dangerous situations.To obtain perception abilities, it is expected to automatically detect static and dynamicobstacles, and estimate their related information. Conventional methods independentlydetect individual piece of overall information. Each process is computationally heavyand often produces noisy results without high reliability. Here we propose fusion-basedand layered-based methodology to systematically detect dynamic and static obstaclesand obtain their location/timing information for visible and infrared sequences. Theproposed obstacle detection methodologies take advantage of connection between dif-ferent information and increase the computational accuracy of obstacle informationestimation, yet reduce computing time, thus improving environment understandingabilities, and driving safety.

Keywords:intelligent transportation system, intelligent vehicle, sensor fusion, ob-stacle detection, time-to-contact, distance detection, image segmentation, motion stereo

1 Introduction

The research on intelligent transportation system has attracted more and more attentionbecause of the following concerns: safety, security, efficiency, mobile access, and environment.Around 75 percent of vehicular crashes are caused by inattentive drivers [1]. According to theNational Highway Traffic Safety Administration (NHTSA) in US (Traffic Safety Facts 2006)[3], more than 6.1 million police-reported motor vehicle crashes occurred in the United Statesin 2006, leading to 42,642 death and 2.575 million people injured. The lifetime economic costof these crashes is estimated over $150 billion annually [2]. Furthermore, age-related decreasesin vision, cognitive functions, and physical impairments affect the driving ability of seniordrivers(Owsley 1999). A 50-year-old driver needs twice as much light to see as does a 30-year-old driver[5]. Seniors are over represented in traffic fatalities. Nine percent of the populationare involved in 13 percent of the fatal crashes. Their fatality rate is 17 times as high as the25-65 age group (NHTSA) [2]. There is an increase of the number of senior drivers who havedifficulties in driving. According to the U.S. Census Bureau, in 1994 one out of every eight

1

Americans was age 65 or older. In 2050, those aged 65 and over will be one out of every fiveAmericans[6]. Between 1995 and 2005, the increase of licensed senior drivers is 17-percent incontrast to 14-percent for total drivers. Thus, while human driving errors lead to dangerousoutcome, driving safety becomes more and more challenging for aging drivers and the wholesociety.

1.1 The status and challenges of current safety enhancement re-

search

Since 1980, researchers have been working on technologies to enhance vehicle safety and drivercomfort as summarized in [1]. The previous research focus was to improve the capabilities ofvehicles, including electronic fuel injection, antilock braking systems, and cruise control, etc.The current interest is to develop driver assistance systems and automation technologies[17],including passive safety systems, such as anti-brake system, stability control, etc., and activedriver assistance systems such as adaptive headlamps, blind spot monitoring, autonomouscruise control, lane departure warning, driver drowsiness alert, etc.

About 60% crashes at intersections and about 30% head-on collisions could be avoided ifdrivers had an extra half-second to respond [10]. It is estimated that implementing collision-avoidance systems in vehicles could prevent 1.1 million accidents in US each year – 17 percentof all traffic accidents, which could save 17,500 lives (compared to the 10,500 lives saved byseatbelts and airbags) and $26 billion in accident-related costs [1]. The demand for in-carelectronic products, especially for, “safety applications”, is increasing. Actually around 35percent of the cost of car assembly comes from electronics[7]. Revenues in the U.S. will be $29billion by 2013[7]. Additional vehicle networking might result in a market worth $1 billionper year in 2014. However these above systems are far behind the drivers’ need.

1.2 Challenges of Conventional Methods to Obtain Obstacle In-

formation

Current algorithms involve heavy computational load while their reliability and accuracy arelimited. To detect driving environment, we need both the statistical and dynamic informa-tion. The static information are: location, obstacle/foreground at different distance ranges,etc. The dynamic information are: speed, time-to-contact, possible cross over or occlusion,obstacle/foreground at different time-to-contact range, collision/occlusion possibility, andother current/historical information, etc.

To obtain distance information for automatic cruise control, researchers have used radar,ladar or binocular stereo vision. A radar detects potential danger during bad weather andnight, but a plain radar does not provide horizontal information. Binocular stereo is subjectto correspondence error, which is hard to avoid and leads to the inaccuracy of depth estima-tion. The performance of binocular stereo is also subject to other disturbances. The bumpingof vehicles make it not easy to calibrate stereo cameras. To obtain obstacle location infor-mation, people apply image segmentation/recognition algorithms to visible images for daytime information and to infrared images for night time. Traditional segmentation algorithmsare challenged by noisy background [16], specifically for background with non-rigid objects,

2

such as trees, or constantly-changing unpredictable backgrounds. Static segmentation usuallydepends on special assumption and the performance is usually limited.

For intelligent vehicle application, people also take advantage of the following features,such as the symmetry of obstacle regions [13] and asymmetry of background, obstacle mo-tion [14], etc.Segmentation/tracking algorithms are also challenged by the significant varia-tion of image position and size of objects between successive video frames. Instead, we needa general algorithm with less assumption.

1.2.1 Expectation for Cooperated Sensors and Automatic Environment Under-

standing

So far, conventional methods independently detect individual information for dynamic en-vironment interpretation and lead to many in-vehicle sensor devices. While new techniquesenhance the driver’s capabilities and comfort, the infusion of the uncoordinated in-vehicledevices may just overwhelm drivers, and disturb drivers’ attention & alertness, thus maydegrade driving safety and performance[1]. In order to help drivers to respond better tofast changing environment and dangerous situations, all safety-enhanced devices should becooperated within a framework to manage & display information. All dynamic and static ob-stacle information should be integrated to provide an accurate description of current drivingenvironment, say, are there any obstacles (pedestrian, vehicles, etc.)? Where are they in thecamera? How far away? Whether and when will the current vehicle run into these obstacles?

In summary, in order to aid drivers, we need a highly coordinated and integrated sys-tem that will work cooperatively with drivers [1] to provide obstacle information. In thenext section we introduce fusion-based and layered-based schemes to systematically detectand combine obstacle information. The additional information from other sensors helps tosimplify original complicated task. Thus the scheme is simpler, more reliable and providesbetter performance.

2 Fusion-based and Layered-based Scheme to Obtain

Obstacle Information

Sensor fusion is a common technology to improve the detection performance by combininginformation from different resources. The technology can be divided into three categories:feature-level fusion, data-level fusion, and decision-level fusion[8][9]. Feature-level fusiontakes advantage of specific information from one sensor and uses the information in anothersensor’s process, which needs special algorithms to incorporate the additional information.Data fusion combines raw data from different resources statistically, for example, throughvoting technique, Bayesian, and Dempster Shafer, etc., which requires all data in similar for-mat. Decision-level fusion takes all sensors’ decision (after individual’s process) into accountwhen making final decisions at a higher level.

We are proposing a general frame work shown in Figure (1)(a) which belongs to feature-level fusion category. Our framework incorporates information obtained from other detec-tors/algorithms into the segmentation process to match corresponding obstacles information,which improves detection performance. Instead of estimating separately static/dynamic in-formation, including distance ranges, segmentation, motion, and classification features, our

3

scheme makes use of the physical connections among these features. Additional informationsuch as distance, motion, and dynamic history can all be used to enhance the segmentationaccuracy. The additional information can be of low quality, yet they can improve overall per-formance [18] [19] [23] [24]. Furthermore, the accurate segmentation information can be usedto improve the detection accuracy of these fused information or other timing information, forexample, time to contact.

(a) (b)

(c) (d)

Figure 1: (a) General framework of fusion-and-layer based methodology. (b) Framework for3D segmentation. (c) Framework for fusing historical information and dynamic estimation. (d)Framework for night vision.

Our scheme is not only fusion-based but also layer-based. With extra information, onecomplex task can be splitted into several simple tasks in different layers, which are easier tosolve than the complex one. Different signal combinations in Figure (1)(a) lead to differentapplications as highlighted by different color shading blocks. The first application is 3Dsegmentation shown in Figure (1)(b) which detects obstacles at different distance rangesand provides complete 3D information for the current road situation [18] [19] [21]. Thesegmentation scheme incorporates distance range information and motion information andsignificantly improves its obstacle segmentation performance. The second application shownin Figure (1)(c) is to incorporate historical information or predicted information from adynamic model into segmentation process in order to understand complicated scenario [12]and to obtain time-to-contact information for obstacles [24]. The third application shownin Figure (1)(d) is a pedestrian segmentation scheme for infrared images. The scheme is afusion-based and layer-based method and takes advantage of classification feature to enhancethe segmentation accuracy [23]. In the following sections, we discuss how the fusion-basedand layer-based principles are used in our different applications.

4

3 The principle of layered techniques: Divide and con-

quer

The major principle of our framework is to convert a complex segmentation task into severalsimple segmentation tasks so that the time-consuming full-image search can be avoided. Insection 3.1, we discuss how to separate an image into several distance-based image-layers, andeach layer only contains potential obstacles within a particular distance range. In section 3.2,we discuss how to separate an image into several vertical stripes, and each stripe containspotential pedestrian obstacles.

3.1 Segmentation based on different distance-based edge layers

Correspondence error of traditional binocular stereo leads to depth estimation noise andsegmentation error in detecting boundary pixels. Instead of directly utilizing noisy depth,we split an edge map into several edge layers at different distance ranges (that also includesbackground range), based on binocular disparity histogram as in [18][19].

3.1.1 Distance-based object layer separation

For each pixel in left edge images, we search right edge images for its candidate correspon-dence pairs within a given disparity range. For all edge pixels in the left edge image, wekeep the pixels that we can find correspondence within the given disparity range in the rightedge image. For each given distance range, those corresponding edge pixels make up the“distance-based edge layers”, which includes obstacle information within the range. Thuswe separate the original stereo edge map into different distance layers for further segmenta-tion. The distance range information can be from a radar, other types of sensors, or distancedetection algorithms, say, the disparity histogram acquired through edge-based trinocularmethods [11] or our motion-based binocular methods [21]. More details can be found in[18] [21].

−10 0 10 20 30 40 50 600

10

20

30

40

50

60

70

−10 0 10 20 30 40 50 600

50

100

150

(a) Binocular stereo images (b) Motion vector for (a) (c) Disparity histogram for (a).

Figure 2: Similarity of motion vectors for binocular stereo images & motion-based disparity his-togram. For (c) Left: motion-based. Right: Trinocular-based. x coordinate: disparity value. y

coordinate: the number of edge pixels within the disparity range.

For stereo image pairs shown in Figure (2)(a), the disparity histogram has three peaksrepresenting three nearest vehicles whose disparity ranges are 22-24, 17-19, 7-9 as shown inFigure (2)(c). Figure (3) shows the original edge map, the separated distance edge layer fordisparity range 22-24, and the final segmentation results. The results capture target objectswithin the disparity range.

5

(a) (b)

(c) (d)

Figure 3: Procedure to locate objects within a distance range (corresponding to disparity range 22-24) (a) Original edge map (b) Corresponding pixels within the disparity range. (c)(d) Segmentationresult.

3.1.2 Segmentation based on background removal

Distance layer separation can be used to detect background layers which consist of objects be-yond some distance ranges. When distance layers are separated, edge pixels may be assignedto several distance ranges in order not to lose obstacles pixels when there is correspondenceambiguity. Typically, extra pixels in different layers can be removed through segmentation,but extra pixels from background, for example, non-rigid objects, will lead to segmentationnoises shown in Figure (5)(b1)(c1). With background-layer separation[19], we can removepixels in background layers from each object layers as shown in Figure (5)(b2)(c2). Theoperation can reduce the impact of background on segmentation error, but it might removetarget pixels and lead to smaller segmentation than it should be, which can be compensatedby motion-based region expansion discussed in section 4.2.1.

(a) (b) (c) (d) (e) (f)

Figure 4: Binocular stereo frames and edge images. (a)(b) Left/Right images at time n. (d)(e)Left/Right images at time n − 1. (c)(f) Left/Right edge images at n. Images (a)(b)-(d)(e) arecourtesy of Daimler-Chrysler Research and Technology.

For stereo frames shown in Figure (4), Figure (5) shows the segmentation results for oneobstacle distance layer without and with applying background pixel removal operation. Theresult indicates that the distance-based background distance layer helps to remove most treepixels in the background while preserving all pixels of target objects including the ball.

(a1) (b1) (c1) (a2) (b2) (c2)

Figure 5: Distance-range-based segmentation results without and with background removal. Fig-ure(a1)(b1)(c1)/Figure(a2)(b2)(c2): Without/with background removal. (a) Edge layer at disparityrange 17-19 for Figure (4)(a)(b). (b)(c) Segmented regions.

6

3.2 Vertical stripe-based object separation

In previous section, we have discussed how to separate an image into several image layers.This section we will split an image with a size of nrow × ncol into several narrower verticalstripes with a size of nrow × ni, ni < ncol. The original full-image segmentation is solvedusing the strategy of “Horizontal-segmentation first, Vertical segmentation second.”

3.2.1 Vertical stripes separation for pedestrian candidates - brightness-based

for infrared frames

One of our method to split image horizontally is based on bright-pixel-vertical-projection

curves, i.e., bright-pixel number in image columns versus their corresponding horizontalpositions[23]. As shown in Figure (6)(a)(b), the horizontal locations and width of projectionwaves correspond to pedestrians and are robust to parameter choices. Then we can apply twovertical segmentation algorithms based on either brightness or bodylines and automaticallyestimate human sizes in scheme shown in Figure (1)(d).

(a) (b) (c) (d)

Figure 6: The Feature of Bright-Pixel-Vertical-Projection Curves for Infrared Images and Segmen-tation Results. For (a)(c): Winter. For (b)(d): Summer. (a): Top row: original infrared image inwinter. Center/Bottom row: bright-pixel-vertical-projection curves when using two different thresh-olds. (b): Top row: original infrared image in summer. Center row: bright-pixel-vertical-projectioncurve. Bottom row: horizontally segmented image stripes based on projection curve. Note thatSeveral separated stripes shown in the center row seem to be connected. For (c): Brightness-basedvertical segmentation results. (d): Bodyline-based vertical segmentation results. For all projectioncurves: X axis: Image column position. Y axis: Number of bright pixels in each column.

3.2.2 Vertical stripes separation for pedestrian candidates - difference-image-

based for visible frames

Our another method of “vertical image-stripe separation” is based on the vertical projectionof difference images between consecutive images. The vertical image stripes corresponding tosharp triangle spikes in vertical projections may contain pedestrian candidates as discussedin framework Figure (1)(c). Within these vertical stripes, human beings can be furthersegmented. This method can be used for difference-image-based moving-object detection.

7

4 The principle of fusion techniques

Our framework shown in Figure (1)(a) is based on information fusion. In Section 3, additionalinformation is used to separate the original image segmentation into segmentation in differentlayers/stripes. In this section, we discusses the fusion techniques used in our general scheme.

4.1 Information fusion for obstacle detection: Segmentation based

on fusion of segmentation and classification features

The scheme in Figure (1)(d) takes advantage of classification feature to enhance the seg-mentation accuracy for infrared images [23]. Within each separated image stripe, in orderto choose among multiple candidate regions during the vertical segmentation, we use onehistogram-based classification feature to search for the best candidate within the stripe andthus we avoid brute-force searching.

Our method balances the complexity and performance of two subsystems: segmentationand classification. The method focuses on improving the performance of combined segmen-tation/classification systems instead of maximizing one process while sacrificing the other.High quality segmentation can ease the classification task, while robust classification cantolerate segmentation errors. The performance comparison is shown in Figure (7) and moredetails are discussed in [23].

Figure 7: Detection Performance Comparison.

4.2 Fusion with historical information

4.2.1 Segmentation based on fusion with motion information

The segmentation performance can be improved by introducing motion information into theabove distance-based segmentation discussed in Section 3.1.1. Figure (8)(a) and (b) show thatmotion vectors of edge pixels in the obstacle distance layer stand out much better than in theoriginal edge pixels because of the impact from background noise. Thus the segmentationresult from Figure (5)(c2) can be enlarged to its real size as shown in Figure (8)(c) and(d). The process compares the motion/depth similarity of segmentation seed boxes and

8

surrounding edge pixels and includes target pixels lost in the process of removing backgroundpixels.

(a) (b) (c) (d)

Figure 8: (a) and (b): Motion vectors for all edge pixels and for obstacle distance layer for Fig-ure (4)(a). (c) and (d): Motion-based segment expansion results based on Figure (5)(c2).

Furthermore, we can eliminate false segmentation blocks in background region by remov-ing static blocks based on motion information when we are only interested in moving-object.As in Figure (5)(c1), two false boxes will be also erased when eliminating static blocks,leading to the same motion-based expansion results shown in Figure (8)(c)(d).

4.2.2 Complicated scenario understanding: segmentation based on fusion of

initial segmentation and dynamic tracking

In order to enhance segmentation accuracy, the fusion-based scheme in Figure (1)(c) takesadvantages of combined information, including, “initial horizontal segmentation” mentionedin section 3.2.2, estimated human locations from dynamic tracking model, and informationfrom previous detection.

The combination of segmentation and dynamic tracking makes the whole human detec-tion algorithm simple but effective in reducing the segmentation/tracking ambiguity whenhuman intersects and occludes. We detect potential intersection by observing whether thereare independent sharp triangle spikes at estimated regions. When two people start to merge,the original two independent triangle spikes also merge. Then we search for the potentialhead locations at all the peaks of merged waves by comparing their corresponding imageregions with human body trunk templates from last frames. After heads are detected, wecan reuse the size information from previous frames to finish human segmentation to avoidwrong segmentation during serious occlusion. The current detection results show that ouralgorithm has the capability to track humans with changing walking poses, to deal with seg-mentation with partial occlusions and target scale variations, and has significantly improvedthe detection accuracy from 50% to 85% and reliability for our 26 video sequences. Detailsare discussed in [12].

4.2.3 Motion-based correspondence matching criteria

In order to remove matching ambiguity for stereo vision, we introduce motion informationinto correspondence matching criteria as in [21]. Because of object rigidity, there existssimilarity of motion information for the same target feature-points calculated from the leftand right camera video frames. As in Figure 2(b), motion vectors for both stereo videoframes show similar patterns even if motion vector detection involves noise. Though motion

9

vectors themselves are very noisy, fusing both traditional epipolar line constraint and motionconstraint help to improve correspondence reliability.

4.3 Information fusion between boundary and interior pixels

4.3.1 Time to contact estimation based on the fusion of boundary feature points

and interior image pixels

Traditional time-to-contact methods are based on location and motion information of bound-ary feature points of interested obstacle. We proposed a method[24] using both boundarypixels and interior image pixels by accumulating sums of suitable products of image bright-ness derivatives. The method works directly with the derivatives of image brightness anddoes not require object detecting, feature tracking, estimation of the optical flow, or anyother “higher level” processing, which greatly enhances the calculation speed and accuracy.Specifically, the direct method can deal with situations when the obstacle size might changein sub-pixel.

4.3.2 Other cases

There are several other cases when we take advantages of both boundary and interior pixels.Traditional segmentation algorithms are mainly based on the features of boundary pixels. Theinterior pixels within an object are not effectively used except for the template matching.In the motion-based expansion as in section 4.2.1, the motion/depth information of interiorpixels is used to expand initial segmentation region based on the similarity between interiorand boundary objects. In infrared-based pedestrian detection, the interior pixels within theboundary are used to detect the vertical stripe containing targets, which helps to simplifytraditionally difficult tasks and thus improves segmentation accuracy. Binocular disparityhistogram is also acquired using both boundary and interior pixels.

5 Summary

We have proposed a sensor fusion scheme on how to set up sensors, how to associate dataand how to fuse data among various sensors according to different task requirements andspecific task environments. The central segmentation is fusion-based and layer-based. Bycommunicating information among sensors ahead of the final processing period and makinguse of their close relationship, better environment interpretation is achieved. The additionalinformation helps to divide original complicated segmentation task into several simpler tasksfor better performance. For example, we can split original image map either into severaldistance-based layers, including background layers, or projection-curve based vertical can-didate stripes, and differentiate static and moving candidates. The idea behind it is thatthe combination of several simple segmentation operators are better than one complicatedsegmentation operator. The scheme is task oriented and object oriented. Instead of payingattention to all frame pixels and their complete 3D reconstruction, we focus on target objectsand their related information. The central segmentation block is fusion-based which can in-corporate distance and motion information, classification feature, historical information, andinformation from dynamic tracking model at the feature level.

10

Our proposed scheme has the following advantages. Firstly, the scheme can be appliedin very general situations and fast changing environment with satisfying performance. Itdoes not make assumptions about the driving environment. It does not assume either thesymmetry of detected objects or the appearance of same objects with similar shape/size inconsecutive frames. For pedestrian detection, it only assumes that local contrast between theimage of a pedestrian and its surroundings and avoids using features like faces, skins, etc. Thescheme has the ability to automatically estimate size of pedestrian regions, to track them, andto detect multiple objects of different sizes within the same distance range. The algorithm canalso detect non-rigid objects, moving-objects, and differentiate objects from their occlusionin very complicated situations. Secondly, the computational load of our proposed schemeis low. The horizontal-first, vertical-second segmentation scheme involves only 1D searchingin vertical direction with computational load o(n) while conventional template-shape-basedsegmentation involves searching with computational load o(n2) [23]. For dynamic tracking,we only compare human template from previous frame with several candidates based onprojection curve of difference image [12]. We acquire simple extra vertical segmentation foreach frame, and avoid complicated tracking model nor long initialization time. Thus wecan significantly decrease potential ambiguity and computational load. Thus, our proposedscheme decreases the quality requirement of individual sensors and the computational load,while increasing estimation accuracy and robustness of obstacle information. Therefore ourscheme improves environment understanding abilities for driving safety. Special thanks toDr.Franke Uwe and Dr.Stefan Heinrich for providing the video frames in Figure 4(a)(b)-(d)(e), courtesy of DaimlerChrysler Research and Technology.

References

[1] The Intelligent Vehicle Initiative: Advancing “Human-Centered” Smart Vehicles. Available:http://www.tfhrc.gov/pubrds/pr97-10/p18.htm.

[2] NHTSA 2020 Report. People Saving People – On the Road to a Healthier Future. Available:http://www.nhtsa.gov/nhtsa/whatis/planning/2020Report/2020report.html

[3] 2006 FARS/GES Traffic Safety Facts Annual Report (Early Edition). Available:http://www.nhtsa.gov/portal/nhtsa static file downloader.jsp?file=staticfiles/DOT/NHTSA/NCSA/Content/TSF/2006/810809.pdf

[4] 2006 Traffic Safety Fact Sheets – older population. Available:http://www.nhtsa.gov/portal/nhtsa static file downloader.jsp?file=/staticfiles/DOT/NHTSA/NCSA/Content/TSF/2006/810808.pdf

[5] Nebraska highway safety program and the lincoln-lancaster county health department (2001,July). Available: http://nncf.unl.edu/eldercare/info/seniordriving/nightdrive.html.

[6] Senior drivers. Available: http://www.nysgtsc.state.ny.us/senr-ndx.htm

[7] Asia - New Hotbed for Consumer Automotive Electronics. Available:http://www.technewsworld.com/story/52539.html

[8] Frank Cremer, Wim de Jong, Klamer Schutte. Fusion of Polarimetric Infrared Features andGPR Features for Lanmine Detection, 2nd International Workshop on Advanced GPR, 14-16May, 2003, Delft, The Netherlands.

[9] Silvia Ferrari, Alberto Vaghi, “Demining Sensor Modeling and Feature-Level Fusion byBayesian Networks”, IEEE Sensors Journal, Vol.6, No.2, Apr.2006.

11

[10] Peter Haapaniemi. “Smart Vehicles Have Minds of Their Own”, Safety + Health, November1996, p. 64.

[11] Bergendahl Jason. A computationally efficient stereo vision algorithm for adaptive cruise

control, MIT Master thesis, 1997.

[12] Yajun Fang. Information fusion for static and dynamic obstacle detection in ITS, MIT Ph.D.thesis, 2008.

[13] M.Bertozzi, A.Broggi, A.Fascioli, S.Nichele. “Stereo Vision-based Vehicle Detection,” Proc.of IEEE Conf. on Intelligent Transportation System, 1997,pp.717-722.

[14] Thomas Meier, King N. Ngan. Automatic Segmentation of Moving Objects for Video Object

Plane Generation, IEEE Transactions on Circuits and Systems For Video Technology, Vol.8,No.5, September, 1998, pp.525-538.

[15] D. Willersinn, W.Enkelmann. Robust Obstacle Detection and Tracking by Motion Analysis,

Proc. of IEEE Conf. on Intelligent Transportation System, 1997,pp.717-722.

[16] Chris Stauffer, W.E.L. Grimson. Adaptive background mixture methods for real-time tracking,IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, 1999,p.246-252.

[17] IVI: 8 Major Problems. Available: http://www.its.dot.gov/ivi/8MPA.html#SI

[18] Yajun Fang, Berthold Horn, Ichiro Masaki, “Distance Range Based Segmentation in Intelli-gent Transportation Systems: Fusion of Radar and Binocular Stereo,” Proceedings of IEEE

Intelligent Vehicles Symposium 2001, 2001, pp.171-176.

[19] Yajun Fang, Berthold Horn, Ichiro Masaki, “Distance/Motion Based Segmentation underHeavy Background Noise,” Proceedings of IEEE Intelligent Vehicles Symposium, 2002, pp.483-488.

[20] Yajun Fang, Yoshiki Ninomiya, Ichiro Masaki, “Intelligent Transportation Systems - Chal-lenges and Opportunities,” the 2nd International Symposium on Multimedia Mediation Sys-

tems, 2002, pp.72-77.

[21] Yajun Fang, Berthold Horn, Ichiro Masaki, “Depth-Based Target Segmentation for IntelligentVehicles: Fusion of Radar and Binocular Stereo,” IEEE Transactions on Intelligent Trans-

portation Systems, Vol.3, No.3, Sept. 2002, pp.196 -202.

[22] Yajun Fang, Keiichi Yamada, Yoshiki Ninomiya, Berthold Horn, Ichiro Masaki, “Comparisonbetween Infrared-image-based and Visible-image-based Approaches for Pedestrian Detection,”Proceedings of IEEE Intelligent Vehicles Symposium, 2003, pp.505-510.

[23] Yajun Fang, Keiichi Yamada, Yoshiki Ninomiya, Berthold Horn, Ichiro Masaki,“A Shape-Independent-Method for Pedestrian Detection with Far Infrared-images,” Special Issue on

”In-Vehicle Computer Vision Systems” of the IEEE Transactions on Vehicular Technology,Vol.53, No.6, Nov. 2004 (pp.1679-1697).

[24] Berthold Horn, Yajun Fang, Ichiro Masaki, “Time to Contact Relative to a Planar Surface,”Proceedings of IEEE Intelligent Vehicles Symposium, 2007.

12