1

Drive Analysis using Vehicle Dynamics andVision-based Lane Semantics

Ravi Kumar Satzoda, Member, IEEE and Mohan M. Trivedi, Fellow, IEEE

Abstract—Naturalistic driving studies (NDS) capture largevolumes of drive data from multiple sensor modalities, whichare analyzed for critical information about driver behavior anddriving characteristics that lead to crashes and near-crashes. Oneof the key steps in such studies is data reduction, which is definedas a process by which “trained employees” review segments ofdriving video and record a taxonomy of variables that providesinformation regarding the sequence of events leading to crashes.Given the volume of sensor data in NDS, such manual analysis ofthe drive data can be time consuming. In this paper, we introduce“drive analysis” as one of the first steps towards automating theprocess of extracting mid-level semantic information from rawsensor data. Techniques are proposed to analyze sensor datafrom multiple modalities, and extract a set of 23 semantics aboutlane positions, vehicle localization within lanes, vehicle speed,traffic density and road curvature. The proposed techniquesare demonstrated using real-world test drives comprising over150,000 frames of visual data, that is also accompanied by vehicledynamics which are captured from the in-vehicle CAN bus,inertial motion unit (IMU) and global positioning system (GPS).

Index Terms—naturalistic driving studies, automatic driveanalysis, lane characteristics, lane change detection, speed vi-olation detection, traffic scenario detection

I. INTRODUCTION

Naturalistic Driving Studies (NDS) capture typical day-to-day driving sessions without artificial features that areintroduced by controlled experimental studies [1]. NDS suchas the 100-car study [2] and the more recent Strategic HighwayResearch Program (SHRP2) [1], [3] investigate the influenceof different contributing factors that lead to specific kindsof driver behaviors and driving styles, which could resultin crashes or near-crashes. Such studies not only help todetermine the reasons for crashes and accidents but also aid indetermining the effectiveness of specific inputs from advanceddriver assistance systems (ADAS) to the driver while drivingso as to prevent any crashes [4].

In order to conduct such detailed studies, large amountsof naturalistic driving data is collected using hundreds orthousands of instrumented vehicles that are fitted with sen-sor suites comprising multiple cameras, radars, sensors thatcapture vehicle dynamics etc. [1], [2], [5]. Data collection andintegration using multiple sensors itself is a major effort inmany such studies involving accident analysis [5], [6]. In the100-car study that was completed recently, 2,000,000 vehiclemiles, and nearly 43,000 hours of data with over 240 driverswas collected during a 12-13 month duration. In the currentSHRP2 study, data from nearly 3100 drivers is being collectedover a 3-year duration in six different places across USA [7].After collecting this data, the next step is to reduce this dataand generate the epoch files [7], [8], which will be analyzed

by different study groups for further behavioral studies [8].The data involved in NDS can be organized in the hierarchylevels as shown in Fig. 1. The first level is the raw sensordata that is collected from different sensors in the vehicle. Thesecond level is the reduced information (as it is called in [5],[7], [9], [10]) that is extracted from the raw sensor data. Wecall this information drive analysis data in this paper becauseit is obtained by analyzing the data that is captured by thesensors during the drive. This includes semantic informationabout the lane positions, vehicle position, lane change events,speed analysis etc. The third and the topmost level of datahierarchy in Fig. 1 comprises the higher level inferences suchas the behavior of the drivers based on the speeding andbraking of the vehicle [9], aggressive driving versus safedriving characteristics [11] etc.

Fig. 1. Hierarchy of data in NDS: Sensor data is at the lowest level followedby drive analysis data obtained from data reduction process.

As seen in the pyramid in Fig. 1, the amount of data reducesas we go from the bottom to the top of the pyramid. Thesensor data at the bottom is dependent on the sensor suite,and this raw data is not directly useful or understandableby the users of the top of the pyramid such as accidentalanalysis, cognitive sciences or human factors researchers.Therefore, this necessitates the data reduction step, which notonly reduces the amount of data but also generates informationat a mid-level of abstraction. NDS such as [4], [5], [7], [10],[12] discuss about data reduction as a process by which trainedemployees review segments of driving video, and record ataxonomy of variables that provides information regardingthe sequence of events leading to crashes or near crashes.

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

2

Given the scale and volume of data that is collected in suchstudies, manual data reduction can be a time-consuming task,and is also subjective to interpretation of the manual subjectsperforming the data reduction. More recent NDS such as[8], [13] suggest automatic tools that can perform this datareduction step resulting in improved efficiency and accuracyof the NDS.

In this paper, we introduce techniques to analyze the nat-uralistic driving data, particularly from highway drives, andgenerate a report of mid-level information of specific eventsand/or observations that can be further analyzed for behavioralstudies. We term this process as Drive Analysis, which wasfirst introduced in [14]. The proposed techniques in this paperautomatically generate a drive analysis report with a list ofsemantic information about the drive by fusing multiple mid-level semantics, which are extracted from multiple sensors inthe vehicle.

In the next section (Section II), we identify some of thecritical parameters or mid-level information that are beingconsidered in related NDS. The scope of the paper and the datacollection process is described in Section III. In Section IV, wefirst describe the lane feature extraction process which is usedin Section V to extract lane-based semantics from the visualand other sensor information. Section VI describes ways toextract semantics from vehicle dynamics obtained from vehicleCAN bus, IMU and GPS. In Section VII, the different kinds ofinformation obtained from the proposed techniques are fusedtogether to generate a list of semantics related to highway drivedata before finally motivating future directions in Section VIII.

II. RELATED WORK & MOTIVATION

In this section, we survey recent naturalistic studies to mo-tivate the objectives and scope of the proposed drive analysispresented in this paper. The final objective of most NDS isto identify driver and driving behaviors, and characteristicsthat can be correlated to crashes or near-crashes [2], [5], [9],[15]. In addition to NDS, works such as [11], [16]–[18] alsodetermine behaviors such as aggressive driving, tactical driverbehavior etc. A survey of the NDS [2], [4], [5], [7], [9], [12]and related studies [11], [16]–[18] aids in finding the mid-levelsemantics that are used to determine such behaviors, some ofwhich will motive the work in this paper.

Lanes in the road scene are used to extract critical in-formation about the drive in most naturalistic studies [2],[4], [5], [9], [10]. In [9], lane change events were used todetermine the driving characteristics leading to crashes ornear-crashes. In addition, the behavior of vehicles in adjacentlanes was also considered as a contributing factor for driverattentiveness in [9]. Lane type information can be useful indetermining the behavior of such vehicles in adjacent lanes.Lane change maneuvers were studied in significant detail inthe Phase-II report of 100-car NDS [5] to identify contributingfactors during a lane change maneuver that lead to crashesand incidents. Identifying lane changes of the subject vehicleand the road conditions under which such lane changes haveoccurred (such as traffic density etc.) was one of the key stepsdiscussed in [5]. In [19], the potential of collision avoidance

systems in avoiding rear-end crashes was investigated, andvehicle localization in the lane was studied as one of themain semantic information to study rear-end collisions. Lanesplayed a pivotal role in the evaluation of road departure crashwarning systems (RDCWS) on naturalistic driving behaviorin [4]. Similar lane keeping studies were also presented in arecent work in [10]. In [12], [20], lane variability was one ofkey metrics that was used to identify levels of driver fatigueand drowsiness.

In addition to information related to lanes, events andinformation derived from vehicle dynamics and traffic condi-tions are commonly used in extracting behavioral information[5], [9]–[11], [21]. In [4], the road curvature and vehiclespeed are analyzed together to understand the risk posed byincorrect curvature estimates and vehicle speed alerts duringcurved sections of the road. In [22], the effects of visualand cognitive demand on driving performance and driver statewere investigated using simulated and field drives, where theinformation from vehicle CAN bus and additional mechanicalsensors on the steering wheel were used. In [21], [23], lateraland longitudinal accelerations, velocity and yaw rate are usedto screen naturalistic driving data based on metrics such astime to collision (TTC) etc.

Traffic density is another semantic that is considered asa major contributing factor in studies such as [5], [7], [9],[17], [18]. Crashes during lane change maneuvers and hightraffic density have been correlated in [5], [24]. In [19],the effectiveness of collision avoidance systems is studiedunder varying traffic conditions. Similarly, traffic density isconsidered as one of the factors influencing lane keepingcapability in [10]. The effect of traffic density on the cognitivecapabilities and the attentiveness of the driver is investigatedin [25], [26].

Given that NDS are detailed studies covering a number ofaspects of driving characteristics and driver behaviors, listingall the details of these works in this paper is not feasible.From the above survey, lane related semantics, such as lanepositions, vehicle localization in lanes, lane types and lanechange events, have been used extensively in most NDS andother related work. Additionally, information about the trafficand critical vehicle dynamics such as speed characteristics alsoplay significant role in most NDS. It was seen in most currentstudies that the data reduction step to obtain these semanticsis usually a manual step, with little or no automated meansto reduce the large volume of sensor data to manageableand useful semantics. In the forthcoming sections, we willdemonstrate automated techniques that aid in data reductionand drive analysis using the visual lane information andvehicle dynamics.

III. DRIVE ANALYSIS SEMANTICS FOR DATA REDUCTION& SCOPE OF STUDY

We have instrumented a testbed called LISA-Q2 with asimilar set of sensors that are being deployed in on-going NDScalled SHRP2. LISA-Q2 is equipped with a forward lookingcamera that continuously captures the forward-view of theego-vehicle. The visual information from this camera is the

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

3

primary source of information to understand the environmentin front of the vehicle. In addition to the camera, the vehicledynamics such as velocity, braking pressure, acceleration etc.are captured by the in-vehicle sensors and recorded using thevehicle Controller Area Network (CAN) bus. The testbed isalso equipped with an inertial motion unit (IMU) comprisingaccelerometer and magnetometer, that captures the state of thevehicle with respect to the world coordinate system (WCS)using accelerations in all three directions, yaw, pitch, roll,altitude etc. Additionally, we have a global position system(GPS) which continuously records the geo-physical positionof the ego-vehicle.

TABLE ILIST OF SEMANTICS DERIVED FROM COMBINATION OF DIFFERENT

PARAMETERS

Drive Analysis Semantics 1 2 3 4 5Lane changes - number and type XDistance of the vehicle from the lane center XPosition of veh. in left/middle/right lane XLane change violations X XSpeed related violations/info. XMean speed during different lane positions X XMean deviation in different lanes X XRight and left turns XAverage speed during left and right curves X XAverage deviation during left/right curves X XPercentage high traffic time XAverage speed during high traffic X1-Lane Change Events, 2-Vehicle localization, 3-Lane marker types, 4-Speed, 5-Curvature

Before presenting techniques to reduce NDS data, we firstintroduce the list of semantics that will be analyzed in thispaper. Data reductionists in NDS such as SHRP2 and 100-car study use a reference dictionary manual (such as [27])to identify the events in raw data. In this paper, we limitthe scope of the study to specific events that are related tosemantics based on lanes and some vehicle dynamics as listedin the first column of Table I. Considering that each NDSvariable or event can be associated with a number of mid-level information, both visual and non-visual, in this paper wewill focus on extracting the following: (1) Lane change events,(2) Vehicle localization, (3) Types of lanes (solid or dashed),(4) speed of the vehicle, and (5) vehicle trajectory curvature.Items (1) to (3) are determined using lane information thatis extracted from the visual information captured during thedrive, whereas items (4) and (5) are related to vehicle dy-namics. It can be seen from Table I that combining differentinformation helps to get critical information about the driveas described by the semantics. It is to be noted that this listis not exhaustive and can be expanded with more items byintroducing data from other sensors. This report in Table I isone of the first steps in generating automated reports for NDS.The list of semantics in Table I also defines the scope of thestudy in this paper. Additionally, it is to be noted that theproposed study is catered towards highway driving scenarios,and further extensions need to be presented to address urbanand city-limit roads in future studies.

IV. LANE FEATURE EXTRACTION

It can be seen from Table I that more than half of thedrive analysis semantics are related to lane related informa-tion. Therefore, it is necessary to introduce the lane featureextraction method that will be employed to extract the mid-level information related to lanes in the forthcoming sections.We employ an extension of the recently published selectiveregion based lane analysis method (LASeR) described in [28],[29] for the drive analysis process in this paper. However, it isto be noted that deploying the positions of the lane features fordrive analysis is the focus of the paper, and the not lane featureextraction process itself. We describe LASeR in this sectionfor completeness but more details about it can be obtainedfrom [28], [29].

In LASeR, selected bands, and not the entire region ofinterest (RoI), are used to detect lane features. An image Iis first converted into its inverse perspective mapping (IPM)[30] image IW resulting in the top view of the road scene. Inthis IPM image, Ns scan bands (denoted by B0 and B1 in Fig.2), each of height hB pixels, are selected along the verticaly-axis, i.e. along the road from the ego-vehicle. Each band Biis then convolved with a vertical filter that is derived fromsteerable filters [31]. The vertical filter is represented by thefollowing equation:

G0◦(x,y) =∂

∂xG(x,y) =

−2xσ2 e−

x2+y2

σ2 (1)

where G(x,y) = e−x2+y2

σ2 is a 2D Gaussian function. Therefore,the filter response BS

i is given by

BSi = Bi ∗G0◦(x,y) (2)

where ∗ represents convolution operation. The significance ofthe 2D Gaussian function, the implications of the standarddeviation σ and the convolution operations are explained in[31]. BS

i from each band is then analyzed for lane featuresby thresholding using two thresholds to generate two binarymaps E+ and E− for each band. Vertical projections p+ andp− are computed using E+ and E−, which are then used todetect lane features using shift-and-match operation (SMO) inthe following way:

K = (p− << δ )�p+ (3)

where � represents point-wise multiplication, δ is the amountof left shift (denoted by << above) that the vector p−undergoes. The above equation for SMO can be explainedusing Fig. 2 in the following way. Peaks in p+ and p−represent dark to light and light to dark transitions in theband respectively. Therefore, shifting p− by δ pixels to theleft (denoted by << in (3)) and then multiplying the resultingvector with p+ will result in detecting adjacent light to darkand dark to light transitions, which characterize lane features.This is shown by KBi in Fig. 2, wherein the peaks correspondto lane features only. On applying a suitable threshold on Kand road model, we get the positions of left and right lanes inthe Bi-th scan band denoted by

PBi = {PL(xL,yi),PR(xR,yi)} (4)

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

4

where yi denotes the y-coordinate of the Bi-th scan band, xLand xR denote the x-coordinates of the left and right lanemarkings of the ego-lane in the Bi-th scan band. These lanemarker positions will be used in the forthcoming sections toextract lane related mid-level information for drive analysis.Fig. 2 shows the algorithmic steps of LASeR. More detailsabout LASeR are explained in [28], [29].

Fig. 2. Block diagram illustrating lane analysis using selected regions(LASeR).

V. DRIVE ANALYSIS USING LANE SEMANTICS

In this section, we will employ the lane feature extractionalgorithm described above to extract lane related mid-levelinformation for drive analysis semantics listed in Table I. Asdiscussed in Section III, lane change events, vehicle localiza-tion and lane type detection are three lane related informationthat will now be discussed in detail.

A. Lane Change Event Detection

In order to detect lane changes during the drive, the scanbands in the near view of the ego-vehicle are processed. Thisis because a lane change is characterized by the position of theego-vehicle with respect to the lane positions in the near-viewof the vehicle. We consider the lane feature positions, obtainedin the NL-th band in the near-view of the vehicle to detect thisevent. The lane feature positions of the left and right lanes inthe NL-th band are given by PNL = {PL(xL,yNL),PR(xR,yNL)}.Considering that the lanes are usually dashed, this band willnot capture the lane marking in every single frame. Therefore,tracking is necessary to predict the lane position in this band.This will also help to eliminate outliers, if any, during lanefeature extraction step.

We will now introduce the information from other sensorssuch as CAN and IMU in addition to the camera to track thelane markings using extended Kalman filter [31]. In order todo this, we initialize two trackers to track the left and rightlane markings. The general state space models used for thistracking are given by the following equations:

X ti+1|ti = AX ti|ti and Y ti = MX ti (5)

where X , A and M are defined as follows:

X =

[φ

tanθ

]A =

[1 v∆t0 1

]M =

[1 00 1

](6)

where φ is the lateral drift of the lane marking position that isbeing tracked. tanθ in X is used to track the yaw rate changeof the vehicle, which is used to determine the lateral drift inthe vehicle due to yaw rate change. Considering that we havetwo different trackers for the left and right lane markings, wewill have φl and φr for the lane markings respectively. In theabove equations v is the speed of the vehicle obtained fromthe CAN bus and θ is the yaw angle obtained from the IMU.A constant velocity is assumed in the above model betweentwo time instances, i.e ∆t. The predicted deviations φr and φlare used to track the lane positions in the NL scan band inthe current frame at ti using straight lane model. A straightlane model is sufficient because we can assume that the roadin the near-view is straight especially in the case of freewaysand highways where the lane change events are being detected.Therefore, the positions of the left and right lane markings atti can be tracked using the following equations:

xtiL = xti−1

L +φl +θyNL (7)

xtiR = xti−1

R +φr +θyNL (8)

These left and right lane positions PtiL (x

tiL,yNL) and Pti

R (xtiR,yNL)

at ti are then used to determine the left and right lane changes.The position of the left lane marking appears to move

towards the right as the vehicle makes the left lane change.Similarly, during a right lane change, the right lane markermoves to the left with respect to the vehicle. These observa-tions are used to detect the lane change events in the followingway:

LCtiL =

{1 if xti

L > T max

0 otherwiseLCti

R =

{1 if xti

R < T min

0 otherwise(9)

where LCtiL = 1 and LCti

R = 1 indicate a left and right lanechange event respectively. The time instances when the lanechanges occur are also recorded in:

T LE =

{tLi}

and T RE =

{tRi}

(10)

Fig. 3 (a) illustrates the above lane change event detection ona video sequence of a drive comprising 10,000 frames. Thex-axis denotes the frame number or the time instance, andthe y-axis gives the lane position in pixel coordinates alongthe x-axis of the IPM image. The lane change events that aredetected using the above formulations are also indicated inFig. 3 (a).

Fig. 3. (a) Receiver operating curve (ROC) for left lane change events, (b)ROC for right lane change events.

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

5

1) Evaluation of Lane Change Event Detection: LASeRalgorithm is evaluated in detail in [36]. In this paper, weevaluate the lane change event detection using the followingmethod. A dataset of 5 input videos with a frame rate ofnearly 25 frames per second were considered resulting in atotal of 66,540 frames. The videos were manually evaluatedto determine the frame numbers where a lane change eventoccurs. A left lane change event is said to have occurred whenthe left lane moves to the right and crosses the mid point of theframe. Similarly, a right lane change event is detected usingthe right lane. This is considered as the ground truth, whichis denoted by:

T LGT =

{tGT Li}

and T RGT =

{tGT Ri

}(11)

where tGT Li and tGT R

i are the time instances where a left ora right lane change event occurs in ground truth respectively.We then consider a window of size 2wt + 1 frames aroundthe ground truth to determine if the lane change eventsdetected by the proposed method (that are recorded in T L

Eand T R

E ) occur within this window. Therefore, if we considerthe left lane change events detected by the above techniqueat time instances tL

j ∈ T LE , they will be considered as true

positive (TP) for the lane change event occurring at tGT Lk

if they occur within the 2wt + 1 window around tGT Lk , i.e.

tGT Lk −wt ≤ tL

j ≤ tGT Lk +wt . Otherwise, a false positive (FP) is

considered to have been detected. Similarly true negatives andfalse negatives are defined. Fig. 3 (b)-(c) show the receiveroperating curves (ROCs) for the left and right lane changeevents. It is to be noted that the x-axis shows false positiverate (FPR) for every 1000 frames. The curves are plotted fordifferent window sizes, which are translated into seconds using25 frames per second. It is evident that increasing the windowsize increases the accuracy as shown in Fig. 3 (b)-(c). Witha window size of 5 seconds around the ground truth event, itcan be seen that a true positive rate of nearly 82% is achievedwith a false positive rate of one in every thousand frames.This validation setup helps to evaluate the lane change eventdetection process for NDS.

Fig. 4. Deviation of the center of the vehicle with respect to the center ofthe lane.

It is to be noted that the evaluation for lane change eventdetection presented in the paper is catered for analysis ofrecorded naturalistic driving data, and for the purpose of re-ducing large amounts of driving data to specific time instances.This is unlike the lane change event detection in active safetysystems where high accuracy is required. Therefore, basedon the ROCs in Fig. 3 (b)-(c), by increasing the accuracy

rates to near 100% for a 5 second window, we see that thefalse positive rate is about 3 in 1000 frames for every 1 truepositive. Therefore, having such accuracy percentages alsowould reduce the total amount of time to analyze long drivingvideos significantly as compared to manual data reductionprocess which is being followed currently. Additionally, to thebest knowledge of the authors, the evaluation presented in thispaper is the first of its kind and there are no previously pub-lished benchmarks or requirements for analyzing NDS data.Therefore, the proposed evaluations can serve as benchmarksfor future studies.

B. Vehicle Localization in Lanes

The lane positions PtiL (x

tiL,yNL) and Pti

R (xtiR,yNL) are used to

determine the localization of the vehicle in the lane. This isespecially significant during the time periods in the drive whenthe vehicle is not changing lanes. The x coordinate of thecenter of the vehicle with respect to the center of the lane isdetermined by the following equation at every time instant ti(given that the camera is placed at the center of the vehicle):

xtiV =

xtiL + xti

R2

(12)

Fig. 4 shows the deviation of the center of the vehicle withrespect to the center of the lane. This information can be usedto determine the average deviation of the vehicle from thecenter of the lane during the drive. Given the time instancesT L

E and T RE when lane change events have occurred (from the

previous section), the average absolute deviation of the vehicleDC from the center of the lane can be determined using thefollowing equation for the entire drive:

DC =

N f

∑i=1|dti |−

NLCE∑

k=1

jk∑

i=ik|dti |

N f −NLCE∑

k=1( jk− ik +1)

(13)

where N f is the total number of frames, dti is the scaleddeviation of the vehicle from center of the lane (obtainedfrom (12)), and ik and jk are the indices of the time instancesbetween which the k-th lane change event occurs. For thevehicle position deviation shown in Fig. 4, DC = 13 cm if theabove equation is used along with the lane change informationas indicated in Fig. 4. This information will be fused with otherdrive analysis information later in this paper to derive specificobservations about the drive.

C. Lane Type Detection: Solid or Dashed

We now analyze the type of lane markings to the left andright of ego-vehicle during the drive as solid or dashed lanemarkings. The knowledge of the type of lane markings can aidin finding drive characteristics such as a slow driving in theinnermost lane of the vehicle or vice versa. A straightforwardway to determine if the lane marking is solid or dashed isto check the frequency of occurrence of lane marking in agiven region [32]. We propose a technique using LASeR todetect the lane type as solid or dashed by monitoring the lanemarkings appearing in the NL-th band. During the lane feature

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

6

detection step, we generate a vector Kti at every time instanceti such that,

Kti = {k j} where k j =

1 if lane marking is foundin NL-th band

0 otherwise(14)

and i−NK ≤ j≤ i. K is a set of NK values, each correspondingto the visual presence or absence of a lane marking in the scanband in the last NK frames from the current ti-th frame. Thenumber of frames that have a lane marking in the previous NKframes is given by Np = ∑

ij=i−NK

k j, which is used to definethe ratio ψ = Np/NK . This ratio is used to define if the typeof lane marking is solid or dashed in the following way:

Ltitype =

{solid if ψ > Ttypedashed otherwise (15)

However, the values of NK and ψ are a function of the vehiclespeed vti , length of the gap between the dashed lane markingsdg, length of the lane markings dm and the frame rate of thevideo capture f . The value of NK will now be derived usingthese parameters. We first assume the largest values for dm =3.66m and dg = 10.98m for California Highways [33]. If NKand ψ satisfy these largest values, they will satisfy the lanemarkings with lesser gaps also. Given frame rate f of a cameraconfiguration in the vehicle, and assuming that the velocity vis constant over dm+dg, then the number of frames Nti

K whichshould be considered at time instant ti to determine the ratioψ ti is given by:

NtiK = b

dm +dg

vtif c (16)

Given NK , we determine K, Np and ψ at every time instant tiusing the visual data in the NL-th band. Fig. 5 shows ψL andψR plotted for the left and right lanes during a typical drivecaptured in about 1000 frames. The thumbnails from the drivecorresponding to three different segments of the drive havingdifferent types of lane markings are also shown in Fig. 5. Inthe first segment, the left lane marking is a solid lane whichis shown by ψL = 1, and the right lane is a dashed lane givenψR < 0.6. In the second segment after a right lane change,the left lane is dashed resulting in ψL < 0.6 for the rest ofthe drive in the third segment also. The right lane has twodifferent kinds of dashed lanes as shown in the segments 2and 3. The dashed lanes are more spaced out in segment 2 ascompared segment 3. Therefore ψR < 0.6 in segment 2, and itincreases marginally to 0.7 in segment 3. If we are interestedin two classes only, i.e. dashed or solid, a threshold of 0.75for ψ was found to a good classification threshold that willsatisfy all kinds of dashed lane markings found on Californiahighways.

The information about the lane types can be used to findthe periods during the drive when and how many times thevehicle was in inner lanes and outer lanes. This informationcan be critical information in studying the behavior of thedriver. Also, any lane violations during the drive can alsobe arrested using the lane change events and the lane typeinformation.

1) Note on Use of Non-Vision Sensors for Lane-relatedMid-level Information: Given that cameras are becomingpervasive in modern automobiles, and ongoing NDS suchas SHRP2 involve 5 different cameras, we chose to analyzethe visual data for extracting the lane-related mid-level infor-mation. However, it is to be noted that other sensors suchas differential GPS and advanced road maps can be usedto determine drive analysis semantics. For example, vehiclelocalization can be determined by using high accuracy differ-ential GPS. However, this may not be cost effective solutionfor deploying in thousands of test vehicles for NDS such asSHRP2. It is to be noted that differential GPS are being usedfor generating ground truths for vehicle localization but notfor deployment in NDS testbeds. Similarly, advanced digitalmaps have been used in [34] to determine vehicle localizationat lane-level. Details such as types of lane markers can alsobe determined, given accurate GPS locations of the vehicles.Considering that NDS involve analysis of data that is capturedfrom multiple locations, the availability of such enhanceddigital map data always is questionable. Additionally, theGPS locations of the drive need to be accurately recorded toextract accurate information from the digital maps. This is anadditional constraint if digital maps alone are relied upon forlane type detection.

High resolution cameras, on the other hand, are cheap,pervasive and readily available. Therefore, analyzing the visualdata for extracting such mid-level information is a moresuitable option without any uncertainties of capturing the dataitself. However, sensors such as GPS and digital maps can beused as complementary sensing and information modalitiesto cameras so that the visual data can be analyzed moreaccurately and efficiently.

VI. DRIVE ANALYSIS USING VEHICLE DYNAMICS

In the previous section, visual information from the cameraswas the primary sensor data that was being reduced to extractlane related semantics. In this section, we will focus on vehicle

Fig. 5. The ratio ψ plotted for left and right lanes of the ego-vehicle for thedrive previously shown in Fig. 4.

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

7

dynamics that are obtained from the CAN bus, IMU and GPSmodules in the testbed described in Section III.

A. Using Vehicle Speed

As shown in Section II, vehicle speed has been used in manyways in behavioral studies. In this subsection, we will extractinformation about the drive using vehicle speed from the CANbus and complemented by other sensors in the testbed.

1) Speed Violations and Average Speed: Firstly, the vehiclespeed information along with the road map information canbe used to check for any vehicle speed violations. The GPScoordinates of the vehicle can be used to determine the typeof road the vehicle is driving on, which can then be usedto check if the vehicle is violating any speed limits. Second,given the different sections of the drive, the average speed ofthe vehicle during these sections can be found. This is usefulto understand driver behavior such as aggressive driving.

2) Traffic Conditions using Speed Information: Given thespeed information and the GPS coordinates of the drive,the state of the traffic is determined, especially in freewayscenario. This is illustrated using the example shown in Fig.6, which shows the speed profiles of two different drives thatwere conducted on a freeway. Assuming that a vehicle issupposed to maintain the speed of the traffic on the freeway, atevery time instant, if the speed is less than a minimum speedvmin, it can be deduced that the traffic is high.

Fig. 6. Speed profiles for two different drives with thumbnails showing thevisual information of the scene during the drive. Distribution of vehicle speedshowing two different probability density functions P1 and P2 correspondingto high and low traffic respectively.

Fig. 6 shows the distribution of speeds that are taken fromover 10000 frames involving both low and high traffic. Thedistribution is used to define two probability density functionsP1(TH |F,v) and P2(TL|F,v), where P1(TH |F,v) denotes theprobability that the traffic is high given that the vehicle is ona freeway F and parameterized by vehicle speed v. Similarly,P2(TL|F,v) denotes the conditional probability for low traffic.Therefore, given the vehicle speed in any drive, we can useP1 and P2 to determine the state of the traffic T ti

S using thefollowing rule:

T tiS =

{High if P1(TH |F,v)> P2(TL|F,v)Low otherwise (17)

B. Vehicle Trajectory Curvature

In this section, we will determine the curvature of thevehicle trajectory. The curvature of the vehicle trajectory

Fig. 7. Curvature profile obtained from vehicle dynamics data with thumbnailsfrom the video showing the instances when a road curvature can be correlatedwith the vehicle trajectory curvature.

at different points in time can be found using the vehicledynamics information. Curvature C at any time instance ti isdefined by the following:

Cti =dθ ti

dsti(18)

where dθ is the change in angle of the tangential motion vectorof the vehicle and ds is the distance traversed by the vehiclealong the curve. We determine dθ using the yaw angle fromthe IMU. The distance traveled in ∆t is given by ds, whichis computed using the speed and time information from theCAN bus (assuming constant velocity during ∆t).

The significance of vehicle trajectory curvature is shownin various studies such as [35] to understand reasons forcrashes. A haphazard vehicle trajectory will be reflected in arandom vehicle trajectory curvature, whereas a vehicle movingstraight will show a near zero values for C over a period oftime. Similarly, if the vehicle trajectory is curving, it will bereflected in C being positive or negative over an interval oftime during the drive.

Fig. 8. 65-km route map on highways on which the test drives wereconducted.

In this paper, we limit the scope of the use of vehicletrajectory curvature to determine the road curvature. Giventhat the proposed study in this paper is related to highwayscenarios, the curvature of the vehicle trajectory can be relatedto the road curvature itself in most cases. This is because ofthe more streamlined vehicle movement in highway scenariosas compared to urban or city-limit roadways. Fig. 7 shows thevehicle trajectory curvature (in degrees/m) that was coveredduring a particular drive. It can be seen from the thumbnails att2 and t3 in Fig. 7 that if C is greater than a positive thresholdT+, then the road is also curving to the right. Similarly if

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

8

Fig. 9. Drive related semantics extracted from (A) drive 1, and (B) drive 2. Legend of curves (top to bottom in (A) and (B)): 1. Lane positions and lanechange events. Left lane change in spikes, right lane change in spikes, 2. Ego-vehicle position (in meters) with respect to the center of the lane, 3. Lanemarker type information - left lane marker in and right lane marker in , |y|= 0.5 indicates dashed lane marking, |y|= 1 indicates solid lane marking,4. Vehicle speed information in miles/hr, 5. Vehicle trajectory curvature in (degrees/meter).

C < T− where T− < 0, it indicates a left curve. Therefore,the curvature of the vehicle trajectory can be directly relatedto road curvature in highway drive analysis. However, whilet2 and t3 in Fig. 7 indicate the correct detection of left andright curvatures of the road based on the formulation above,t1 represents a right lane change and not a right road curvature.This is possible because the yaw angle varies similarly duringthe vehicle trajectories of both a right lane change event and aright turn. Therefore, using the vehicle dynamics alone fromthe IMU and CAN bus may result in false positives (such as t1in Fig. 7). Combining the curvature information with the lanechange events allows to remove such false positives. Also, thevisual information about the lanes can also be used to detectthe presence of road curvatures more accurately. For example,the curvature subtended by the lane features extracted fromthe farther scan bands in LASeR can be used to detect curvedlanes ahead of the ego-vehicle. Such vision-based techniquescan be explored further for detection of road curvature inconjunction with the vehicle trajectory curvature. Additionally,digital maps if available could be used as a complementarysource of information in addition to the in-vehicle sensors todetermine the curvature of the road.

VII. DRIVE ANALYSIS BY COMBINING VEHICLEDYNAMICS & LANE SEMANTICS

As shown in Table I, the different mid-level informationfrom speed, curvature and lanes can be combined to performa detailed drive analysis of naturalistic driving data. In thissection, we will illustrate the generation of drive analysisreports on two test drives (drive 1 and drive 2) that wereconducted by two different drivers on the 65-km test routeon highways. As indicated previously, being one of the firstcontributions in this specific area of research on NDS, the

Fig. 10. ROCs for lane change event detection the two test drives (a) drive1, (b) drive 2.

scope of this study is limited to highways in this paper. Thetechniques extracting mid-level information based on lanes andvehicle dynamics both need to be further extended to cater tourban and city-limit roads. This is because, as compared tohighways, city-limit roads are lesser constrained in terms oflane geometry, infrastructural requirements, and regulations onvehicle movement and speed. Therefore, analyzing drives onsuch roads requires additional requirements, which are out ofscope of this paper.

Fig. 8 shows the 65-km test route on highways for the drives1 and 2. For each drive, Fig. 9 plots the 5 main componentsof drive analysis information that are extracted from thesensors, i.e. lane position and lane change information, vehiclelocalization, lane marker types, speed and curvature, whichare all time synchronized with each other. This information isfurther used to determine the different drive analysis semanticsthat were listed in Table I. Firstly, the lane change eventdetection was performed to generate the ROCs shown in Fig.

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

9

TABLE IIDRIVE ANALYSIS RESULTS

Drive Analysis Semantics Drive 1 Drive 2Lane changes 45 69Types of lane changes - left/right 20/25 26/43Vehicle from the lane center - distance 0.21 m 0.20 mVehicle on middle lane - time spent 57% 56%Vehicle on left most lane - time spent 8% 19%Vehicle on right most lane - time spent 23% 10%Lane change violations Nil NilSpeed violations - percentage time 58% 68%Mean speed during speed violations 69.68 mph 70.73 mphMean speed during middle lanes 68.97 mph 65.65 mphMean speed during leftmost lanes 65.84 mph 65.88 mphMean speed during rightmost lanes 64.67 mph 48.05 mphMean veh. dev. during middle lanes 0.2 m 0.21 mMean veh. dev. during leftmost lanes 0.19 m 0.18 mAverage veh. dev. during rightmost lanes 0.23 m 0.31 mRight turns 3 3Left turns 4 4Mean speed during left curves (C <−0.05) 68.46 mph 61.56 mphMean speed during right curves (C > 0.05) 53.31 mph 51.42 mphMean deviation during left curves 0.62 m 0.31 mMean deviation during right curves 0.18 m 0.14 mHigh traffic time (percentage time) 1.95% 12.5%Mean speed during high traffic 43.17 mph 32.98 mph

10. Nearly 90% of the lane changes were detected accuratelyin both drives at a false positive rate of 1 false positive inevery 1000 frames. The lane marker positions detected byLASeR algorithm were then used to determine the ego-vehiclelocalization, i.e. the mean deviation of the ego-vehicle fromthe center of the lane during the drive. The information aboutthe lanes is fused with speed of the vehicle and curvature ofthe road to determine 23 different semantics shown in TableII for the two drives.

It can be seen from Table II that drive 2 involved morelane changes than drive 1. The ego-vehicle is deviated fromthe center of the lane in a similar way in both drives. Bothdrivers spent majority of the time on the middle lanes (i.e.left and right lane markers are both dashed). However, driver2 seems to have spent more time on the left most lane ascompared to driver 1. From the speed profiles, it can also beestablished that both drivers exceeded the speed limit of 65mph for most part of their drives by about 5 mph. Combiningthe lane marker type information with the speed informationshows that driver 2 drove at less than 50 mph when he wason the right most lane. This is in contrast to driver 1 whorecorded a speed of 65.6 mph when he was on the right mostlane. It can also be seen that the average vehicle deviation ishigher in both drivers when they are on the rightmost lanes ascompared to leftmost lanes. Another semantic of interest is thevehicle speed and deviation during the different curvatures. Itcan be seen that driver 2 is more cautious than driver 1 becausehe records lower speeds during the curvatures. It can also beseen that the average vehicle deviation during such curvaturesis lesser for driver 2 as compared to driver 1. This can beattributed to the lower speeds of driver 2. Additionally, thespeed profiles can also be used to determine that 12.5% ofdrive 2 experienced high traffic, when the speed was reduced

to 33 mph.

VIII. CONCLUSIONS & FUTURE DIRECTIONS

In this paper, we investigated the need for automated driveanalysis so as to reduce the time and effort needed for datareduction in NDS. The proposed drive analysis techniques areshown to synergistically fuse data from a forward lookingcamera, in-vehicle CAN bus, IMU and GPS to extract lane po-sitions, vehicle localization in the lane, types of lane markers,lane change events, speed information and vehicle trajectorycurvature information. These mid-level semantics were furtherfused in different ways to automatically determine a set of23 semantics about the drive. The drive analysis process isdemonstrated on naturalistic driving data involving multipledrivers over a 65-km test route involving more than 150,000frames of data and accompanying vehicle dynamics. However,such NDS involve a number of semantics that are of interestto specific behavioral studies. Therefore, there is a need togenerate more techniques such as those presented in this paperto cater to the different data needs and requirements of NDS.Future work [37] includes developing more such techniquesand toolboxes to extract mid-level semantics, and also analyzeit further for behavioral investigations.

ACKNOWLEDGMENTS

The authors would like to thank the associate editor and thereviewers for their constructive suggestions and comments. We wouldalso like to thank our sponsors, Toyota’s Collaborative Safety Re-search Center (CSRC), particularly Dr. Pujitha Gunaratne, and UCDiscovery Program. Finally, we thank our colleagues Ms. SujithaMartin, Mr. Minh Van Ly, and Mr. Greg Dawe from Laboratory forIntelligent and Safe Automobiles (LISA).

REFERENCES

[1] L. N. Boyle, S. Hallmark, J. D. Lee, D. V. McGehee, D. M. Neyens,and N. J. Ward, “Integration of Analysis Methods and Development ofAnalysis Plan - SHRP2 Safety Research,” Transportation Research Boardof the National Academies, Tech. Rep., 2012.

[2] V. L. Neale, T. A. Dingus, S. G. Klauer, and M. Goodman, “An Overviewof the 100-Car Naturalistic Study and Findings,” VTTI, Tech. Rep., 2002.

[3] G. Davis and J. Hourdos, “Development of Analysis Methods UsingRecent Data: SHRP2 Safety Research,” Transportation Research Boardof the National Academies, Tech. Rep., 2012.

[4] D. LeBlanc, J. Sayer, C. Winkler, R. Ervin, S. Bogard, J. M. Devnoshire,M. Hagan, Z. Bareket, R. Goodsell, and T. Gordon, “Road DepartureCrash Warning System Field Operational Test : Methodology and Re-sults,” The University of Michigan Transportation Research Institute,Tech. Rep. June, 2006.

[5] T. A. Dingus, S. G. Klauer, V. L. Neale, V. L. Petersen, S. E. Lee,J. Sudweeks, M. A. Perez, J. M. Hankey, D. Ramsey, S. Gupta, C. Bucher,Z. R. Doerzaph, J. Jermeland, and R. R. Knipling, “The 100-Car Natural-istic Driving Study Phase II Results of the 100-Car Field Experiment,”National Highway Traffic Safety Administration (NHSTA), Tech. Rep.April, 2006.

[6] K. Takeda, J. Hansen, P. Boyraz, C. Miyajima, and H. Abut, “InternationalLarge-Scale Vehicle Corpora for Research on Driver Behavior on theRoad,” IEEE Trans. on Intell. Transp. Syst., vol. 12, no. 4, pp. 1609–1623, 2011.

[7] A. M. Brach, “SHRP 2 Naturalistic Driving Study What We HaveLearned So Far,” in Third International Symposium on NaturalisticDriving Research, 2012, pp. 1–12.

[8] K. Ebe, “Development of Distracted Driving Database,” in Third Inter-national Symposium on Naturalistic Driving Research, 2012, pp. 1–17.

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear

10

[9] S. G. Klauer, T. A. Dingus, V. L. Neale, J. Sudweeks, and D. Ramsey,“Comparing Real-World Behaviors of Drivers With High versus LowRates of Crashes and Near-Crashes,” National Highway Traffic SafetyAdministration (NHSTA), Tech. Rep. February, 2009.

[10] Y. Peng, L. N. Boyle, and S. L. Hallmark, “Driver’s lane keeping abilitywith eyes off road: Insights from a naturalistic study.” Accident; analysisand prevention, vol. 50, pp. 628–34, Jan. 2013.

[11] D. A. Johnson and M. M. Trivedi, “Driving style recognition using asmartphone as a sensor platform,” 2011 14th International IEEE Conf.on Intell. Transp. Syst. (ITSC), pp. 1609–1615, Oct. 2011.

[12] M. Sallinen, “Metrics for Identifying Fatigued Driving Rationale fordeveloping metrics of driver fatigue Valid metrics of driver fatigue ofdriver fatigue,” in Third International Symposium on Naturalistic DrivingResearch, 2012, pp. 1–11.

[13] P. P. Jovanis, V. Shankar, J. Aguero-Valverde, K.-F. Wu, and A. Green-stein, “Analysis of Existing Data : Prospective Views on MethodologicalParadigms,” Tansportation Research Board of the National Academies,Tech. Rep., 2012.

[14] R. K. Satzoda, S. Martin, M. V. Ly, P. Gunaratne, and M. M. Trivedi,“Towards Automated Drive Analysis : A Multimodal Synergistic Ap-proach,” in IEEE Intell. Transp. Syst. Conf., 2013, pp. 1912-1916. appear.

[15] E. Ohn-Bar, S. Martin, and M. M. Trivedi, “Driver hand activity analysisin naturalistic driving studies: challenges, algorithms, and experimentalstudies,” Journal of Electronic Imaging, vol. 22, no. 4, pp. 041119–1–10,Oct. 2013.

[16] A. Doshi, B. T. Morris, and M. M. Trivedi, “On-road prediction ofdriver’s intent with multimodal sensory cues,” IEEE Pervasive Computing,vol. 10, no. 3, pp. 22–34, 2011.

[17] A. Doshi and M. M. Trivedi, “Examining the impact of driving styleon the predictability and responsiveness of the driver: Real-world andsimulator analysis,” 2010 IEEE Intelligent Vehicles Symposium, pp. 232–237, June 2010.

[18] ——, “Tactical driver behavior prediction and intent inference: Areview,” in 2011 14th International IEEE Intell. Transp. Syst. Conf.(ITSC), Oct. 2011, pp. 1892–1897.

[19] S. B. McLaughlin, J. M. Hankey, T. A. Dingus, and S. G. Klauer,“Development of an FCW Algorithm Evaluation Methodology WithEvaluation of Three Alert Algorithms,” National Highway Traffic SafetyAdministration, Tech. Rep. June, 2009.

[20] P. M. Forsman, B. J. Vila, R. A. Short, C. G. Mott, and H. P. A.Van Dongen, “Efficient driver drowsiness detection at moderate levelsof drowsiness.” Accident; analysis and prevention, vol. 50, pp. 341–50,Jan. 2013.

[21] K.-F. Wu and P. P. Jovanis, “Screening Naturalistic Driving Study Datafor Safety-Critical Events,” Transportation Research Record: Journal ofthe Transportation Research Board, vol. 2386, pp. 137–146, Dec. 2013.

[22] J. Engstrom, E. Johansson, and J. Ostlund, “Effects of visual andcognitive load in real and simulated motorway driving,” TransportationResearch Part F: Traffic Psychology and Behaviour, vol. 8, no. 2, pp.97–120, Mar. 2005.

[23] K.-F. Wu and P. P. Jovanis, “Defining and screening crash surrogateevents using naturalistic driving data.” Accident analysis and prevention,vol. 61, pp. 10–22, Dec. 2013.

[24] S. E. Lee, E. C. B. Olsen, and W. W. Wierwille, “A ComprehensiveExamination of Naturalistic Lane-Changes,” National Highway TrafficSafety Administration, Tech. Rep., 2004.

[25] S. G. Klauer, T. A. Dingus, V. L. Neale, J. Sudweeks, and D. Ramsey,“The Impact of Driver Inattention on Near-Crash/Crash Risk: An AnalysisUsing the 100-Car Naturalistic Driving Study Data,” NHSTA, Tech. Rep.,April, 2006.

[26] J. Stutts, J. Feaganes, D. Reinfurt, E. Rodgman, C. Hamlett, K. Gish,and L. Staplin, “Driver’s exposure to distractions in their natural drivingenvironment.” Accident; analysis and prevention, vol. 37, no. 6, pp. 1093–101, Nov. 2005.

[27] “Researcher Dictionary for Video Reduction Data,” VTTI, Tech. Rep.,2012.

[28] R. K. Satzoda and M. M. Trivedi, “Selective Salient Feature based LaneAnalysis,” in IEEE Intell. Transp. Syst. Conf., 2013, pp. 1906–1911.

[29] ——, “Vision-based Lane Analysis : Exploration of Issues and Ap-proaches for Embedded Realization,” in IEEE Conf. on CVPR Workshopson Embedded Vision, 2013, pp. 604–609.

[30] M. Bertozzi and A. Broggi, “GOLD: a parallel real-time stereo visionsystem for generic obstacle and lane detection.” IEEE Trans. on ImageProcess., vol. 7, no. 1, pp. 62–81, Jan. 1998.

[31] J. McCall and M. Trivedi, “Video-Based Lane Estimation and Trackingfor Driver Assistance: Survey, System, and Evaluation,” IEEE Trans. onIntell. Transp. Syst., vol. 7, no. 1, pp. 20–37, Mar. 2006.

[32] S. Nedevschi, V. Popescu, R. Danescu, T. Marita, and F. Oniga, “Accu-rate Ego-Vehicle Global Localization at Intersections Through Alignmentof Visual Data with Digital Map,” IEEE Trans. on Intell. Transp. Syst.,pp. 1–15, 2013.

[33] Caltrans, “Markings,” in Traffic Manual, 1996, ch. 6, pp. 1–76.[34] D. Betaille and R. Toledo-Moreo, “Creating Enhanced Maps for Lane-

Level Vehicle Navigation,” IEEE Trans. on Intell. Transp. Syst., vol. 11,no. 4, pp. 786–798, Dec. 2010.

[35] R. Talbot, H. Fagerlind, and A. Morris, “Exploring inattention anddistraction in the SafetyNet Accident Causation Database.” Accident;analysis and prevention, vol. 60, pp. 445–55, Nov. 2013.

[36] R. K. Satzoda, and M. M. Trivedi, “On Performance Evaluation Metricsfor Lane Estimation,” Intl. Conf. on Pattern Recog. (ICPR), 2014 ToAppear.

[37] R. K. Satzoda, P. Gunaratne, and M. M. Trivedi, “Drive Analysis usingLane Semantics for Data Reduction in Naturalistic Driving Studies,” inIEEE Intell. Veh. Symp., 2014, pp. 293–298.

Ravi Kumar Satzoda received his B.Eng. (withFirst Class Honors) degree in electronics fromNanyang Technological University (NTU), Singa-pore in 2004 under Accelerated Bachelors Program.He completed his M.Eng. (by Research) from NTUin 2007 with a Master’s dissertation on VLSI ar-chitectures for cryptosystems. He received his PhDin 2013 from NTU with a dissertation on efficientcomputing architectures for Advanced Driver As-sistance Systems (ADAS). He is currently a PostDoctoral Fellow at University of California San

Diego (UCSD), and associated with Laboratory for Intelligent and SafeAutomobiles. His research interests include computer vision, embedded visionsystems and intelligent vehicles. He is the recipient of the prestigious SIA-NOL (Singapore) scholarship for his under-graduation (2000-2004). He iscurrently serving as a reviewer for various IEEE journals and conferencesincluding IEEE Transactions on Computers, IEEE Transactions on IntelligentTransportation Systems, and IEEE Transactions on Industrial Electronics.

Mohan Manubhai Trivedi (F’ 08) received theB.E. (with honors) degree in electronics from BirlaInstitute of Technology and Science, Pilani, India, in1974 and the M.S. and Ph.D. degrees in electricalengineering from Utah State University, Logan, UT,USA, in 1976 and 1979, respectively. He is currentlya Professor of electrical and computer engineeringand he is the Founding Director of the ComputerVision and Robotics Research Laboratory, Univer-sity of California San Diego (UCSD), La Jolla, CA,USA. He has also established the Laboratory for

Intelligent and Safe Automobiles, Computer Vision and Robotics ResearchLaboratory, UCSD, where he and his team are currently pursuing researchin machine and human perception, machine learning, human-centered mul-timodal interfaces, intelligent transportation, driver assistance, active safetysystems and Naturalistic Driving Study (NDS) analytics. He regularly servesas a Consultant to industry and government agencies in the United States,Europe, and Asia. He has given over 70 Keynote/Plenary talks at majorconferences. He has coauthored a number of papers winning Best Paperawards.

Prof. Trivedi is a Fellow of the International Association of PatternRecognition (for contributions to vision systems for situational awareness andhuman-centered vehicle safety) and the Society for Optical Engineering (forcontributions to the field of optical engineering). He received the IEEE Intel-ligent Transportation Systems Societys highest honor, Outstanding ResearchAward in 2013, the Pioneer Award (Technical Activities) and the MeritoriousService Award of the IEEE Computer Society, and the Distinguished AlumniAward from Utah State University, Logan, UT, USA. He serves on the Boardof Governors of IEEE ITS Society and on the Editorial advisory board of theIEEE Trans on Intelligent Transportation Systems.

IEEE Transactions on Intelligent Transportation Systems, 2014 - To Appear