GesturerecognitionwithKinect

JoakimLarsson

Outline

• Taskdescription• Kinectdescription• AdaBoost• Buildingadatabase• Evaluation

TaskDescription

• ThetaskwastoimplementgesturedetectionforsomegesturesusingaKinectsensor.

• Thegesturechosenwasiftheuserwaspointinginadirection.

Kinect

• Kinectisamotionsensinginputdevice.• Itrecordsthedistancebetweenthesensorandanyobjectsinitswayasapointcloud.

• Fromthesepointsthereexistssoftwaretoextractahumanbodyfromthepointcloudlive.

Kinect

Kinect

Firstsolution

• Thefirstsolutionusedwastoextractthejointsanddetermineifthehand-,wrist-,elbow-joints’positionforbotharmswasclosetoformingaline.

• Insuchascenariotheuserwouldbeconsideredtobepointing.

• ThissolutionrequiredtheusertoholdtheirarmsVERYstraighthowever.

• ThemethodeventuallychosenusesthemetaalgorithmAdaBoost.

AdaBoost

• AdaBooststandsforAdaptiveboosting.• ItwontheGödelPrizein2003.• It’sanalgorithmwithwhichamachinecanbetaughtwhatclassifiersaresignificantwhendeterminingwhetheradiscreteeventhasoccurredornot.

AdaBoostexample

• Imaginewewishtoclassifyafournumberstringonanunknowncriterion.

• Theonlythingweknowofthecriterionistheinformationwecangatherfromasetoftrainingexamples.

AdaBoostExample

• Inordertoclassifythetrainingsetsomesimpleclassifiersareneeded.

• Inthisexamplecasetheclassifiersareoftwonatures:– Treatingthestringsasintegersandaskingwhethertheyareaboveorbelowacertainthresholdvalue.

– Countinghowmanyoftheintegersinthestringthatareofacertaintype.

STRING CLASS0001 FALSE0012 FALSE0415 FALSE0881 FALSE0888 TRUE1234 FALSE1235 FALSE1299 TRUE1515 FALSE1559 FALSE7654 TRUE7771 TRUE7777 TRUE7779 TRUE7780 TRUE8337 TRUE8502 FALSE9001 FALSE9039 TRUE9999 TRUE

AdaBoostExample

• Examinationshowsthatinthisexample7650isthebestthresholdvalue.

STRING CLASS >76500001 FALSE FALSE0012 FALSE FALSE0415 FALSE FALSE0881 FALSE FALSE0888 TRUE FALSE1234 FALSE FALSE1235 FALSE FALSE1299 TRUE FALSE1515 FALSE FALSE1559 FALSE FALSE7654 TRUE TRUE7771 TRUE TRUE7777 TRUE TRUE7779 TRUE TRUE7780 TRUE TRUE8337 TRUE TRUE8502 FALSE TRUE9001 FALSE TRUE9039 TRUE TRUE9999 TRUE TRUE

AdaBoostExample

• Examinationshowsthatinthisexample7650isthebestthresholdvalue.

• Thislabels80%oftheexamplescorrectly.

STRING CLASS >76500001 FALSE FALSE0012 FALSE FALSE0415 FALSE FALSE0881 FALSE FALSE0888 TRUE FALSE1234 FALSE FALSE1235 FALSE FALSE1299 TRUE FALSE1515 FALSE FALSE1559 FALSE FALSE7654 TRUE TRUE7771 TRUE TRUE7777 TRUE TRUE7779 TRUE TRUE7780 TRUE TRUE8337 TRUE TRUE8502 FALSE TRUE9001 FALSE TRUE9039 TRUE TRUE9999 TRUE TRUE

STRING CLASS >7650 CORRECT0001 FALSE FALSE TRUE0012 FALSE FALSE TRUE0415 FALSE FALSE TRUE0881 FALSE FALSE TRUE0888 TRUE FALSE FALSE1234 FALSE FALSE TRUE1235 FALSE FALSE TRUE1299 TRUE FALSE FALSE1515 FALSE FALSE TRUE1559 FALSE FALSE TRUE7654 TRUE TRUE TRUE7771 TRUE TRUE TRUE7777 TRUE TRUE TRUE7779 TRUE TRUE TRUE7780 TRUE TRUE TRUE8337 TRUE TRUE TRUE8502 FALSE TRUE FALSE9001 FALSE TRUE FALSE9039 TRUE TRUE TRUE9999 TRUE TRUE TRUE

0,80,2

AdaBoostExample• Thenextcriterionwouldbetoseehowmanyzeroesareinthestring.

• Afterthatones.• Andsoon.

STRING CLASS >7650 CORRECT #0>0 CORRECT0001 FALSE FALSE TRUE TRUE FALSE0012 FALSE FALSE TRUE TRUE FALSE0415 FALSE FALSE TRUE TRUE FALSE0881 FALSE FALSE TRUE TRUE FALSE0888 TRUE FALSE FALSE TRUE TRUE1234 FALSE FALSE TRUE FALSE TRUE1235 FALSE FALSE TRUE FALSE TRUE1299 TRUE FALSE FALSE FALSE FALSE1515 FALSE FALSE TRUE FALSE TRUE1559 FALSE FALSE TRUE FALSE TRUE7654 TRUE TRUE TRUE FALSE FALSE7771 TRUE TRUE TRUE FALSE FALSE7777 TRUE TRUE TRUE FALSE FALSE7779 TRUE TRUE TRUE FALSE FALSE7780 TRUE TRUE TRUE TRUE TRUE8337 TRUE TRUE TRUE FALSE FALSE8502 FALSE TRUE FALSE TRUE FALSE9001 FALSE TRUE FALSE TRUE FALSE9039 TRUE TRUE TRUE TRUE TRUE9999 TRUE TRUE TRUE FALSE FALSE

0,8 0,350,2 0,65

AdaBoostExampleSTRING CLASS Weight >7650 CORRECT #0>0 CORRECT #0>1 CORRECT #1>0 CORRECT0001 FALSE 0,05 FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE0012 FALSE 0,05 FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE0415 FALSE 0,05 FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE0881 FALSE 0,05 FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE0888 TRUE 0,05 FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE1234 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE1235 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE1299 TRUE 0,05 FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE1515 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE1559 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE7654 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE7771 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE7777 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE7779 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE7780 TRUE 0,05 TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE8337 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE8502 FALSE 0,05 TRUE FALSE TRUE FALSE FALSE TRUE FALSE TRUE9001 FALSE 0,05 TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE9039 TRUE 0,05 TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE9999 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

1 0,8 0,35 0,35 0,150,2 0,65 0,65 0,85

AdaBoostExample

• Theclassifierthatgivethesmallesterror,i.e.theonethatforthetrainingsetreturnsthecorrectclassificationmostoftenischosenasourprimaryclassifier.

AdaBoostExample

• Intheexampleusing#1=0asaclassifiernets3/20incorrectlabelings.

STRING CLASS Weight >7650 CORRECT #0>0 CORRECT #0>1 CORRECT #1>0 CORRECT0001 FALSE 0,05 FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE0012 FALSE 0,05 FALSE TRUE TRUE FALSE TRUE FALSE TRUE FALSE0415 FALSE 0,05 FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE0881 FALSE 0,05 FALSE TRUE TRUE FALSE FALSE TRUE TRUE FALSE0888 TRUE 0,05 FALSE FALSE TRUE TRUE FALSE FALSE FALSE FALSE1234 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE1235 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE1299 TRUE 0,05 FALSE FALSE FALSE FALSE FALSE FALSE TRUE TRUE1515 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE1559 FALSE 0,05 FALSE TRUE FALSE TRUE FALSE TRUE TRUE FALSE7654 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE7771 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE TRUE TRUE7777 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE7779 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE7780 TRUE 0,05 TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE8337 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE8502 FALSE 0,05 TRUE FALSE TRUE FALSE FALSE TRUE FALSE TRUE9001 FALSE 0,05 TRUE FALSE TRUE FALSE TRUE FALSE TRUE FALSE9039 TRUE 0,05 TRUE TRUE TRUE TRUE FALSE FALSE FALSE FALSE9999 TRUE 0,05 TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE

1 0,8 0,35 0,35 0,150,2 0,65 0,65 0,85

AdaBoostExample

• Inordertodeterminehowwelltheclassifierworkedweassignitan α-valuedependentonitserrorrate.

• 𝛼" =$%& ln $)*+

*+

AdaBoostExample

• Wenowwanttofindadditionalclassifiers• Sinceourcurrentclassifierclassifiessomeexamplesquitewell,butothersnotsomuchwewantournextclassifiertoprimarilyclassifythoseincorrectlylabeledcorrectly.

• Therelativeweightofthedifferentexamplesarethusmultipliedby𝑒α iftheclassifierclassifiedincorrectlyand𝑒−α otherwise.

AdaBoostExampleSTRING CLASS Weight0001 FALSE 0,0150012 FALSE 0,0150415 FALSE 0,0150881 FALSE 0,0150888 TRUE 0,0151234 FALSE 0,0151235 FALSE 0,0151299 TRUE 0,0851515 FALSE 0,0151559 FALSE 0,0157654 TRUE 0,0157771 TRUE 0,0857777 TRUE 0,0157779 TRUE 0,0157780 TRUE 0,0158337 TRUE 0,0158502 FALSE 0,0859001 FALSE 0,0159039 TRUE 0,0159999 TRUE 0,015

AdaBoostExample

• Wethencontinuebypickingoutthecriterionthatminimizestheweightederrors.

• Afterwards,wecalculatethecorrespondingα-value,andcalculatethenewweightsforourtrainingset.

• Afterwards,thebestweightedcriterioniscalculated.

• Thiscontinuesuntilwehaveasufficientlygoodcombinedclassifier.

AdaBoostExample

• Intheexampleusing#0>0asaclassifiernetsabetterweightedvalue,despitethenumericalthresholdingcorrectlyclassifyingmoreexamples.

STRING CLASS Weight >1240 CORRECT #0>0 CORRECT #0>1 CORRECT0001 FALSE 0,015 FALSE TRUE TRUE FALSE TRUE FALSE0012 FALSE 0,015 FALSE TRUE TRUE FALSE TRUE FALSE0415 FALSE 0,015 FALSE TRUE TRUE FALSE FALSE TRUE0881 FALSE 0,015 FALSE TRUE TRUE FALSE FALSE TRUE0888 TRUE 0,015 FALSE FALSE TRUE TRUE FALSE FALSE1234 FALSE 0,015 FALSE TRUE FALSE TRUE FALSE TRUE1235 FALSE 0,015 FALSE TRUE FALSE TRUE FALSE TRUE1299 TRUE 0,085 TRUE TRUE FALSE FALSE FALSE FALSE1515 FALSE 0,015 TRUE FALSE FALSE TRUE FALSE TRUE1559 FALSE 0,015 TRUE FALSE FALSE TRUE FALSE TRUE7654 TRUE 0,015 TRUE TRUE FALSE FALSE FALSE FALSE7771 TRUE 0,085 TRUE TRUE FALSE FALSE FALSE FALSE7777 TRUE 0,015 TRUE TRUE FALSE FALSE FALSE FALSE7779 TRUE 0,015 TRUE TRUE FALSE FALSE FALSE FALSE7780 TRUE 0,015 TRUE TRUE TRUE TRUE FALSE FALSE8337 TRUE 0,015 TRUE TRUE FALSE FALSE FALSE FALSE8502 FALSE 0,085 TRUE FALSE TRUE FALSE FALSE TRUE9001 FALSE 0,015 TRUE FALSE TRUE FALSE TRUE FALSE9039 TRUE 0,015 TRUE TRUE TRUE TRUE FALSE FALSE9999 TRUE 0,015 TRUE TRUE FALSE FALSE FALSE FALSE

1 0,715686 0,205882 0,3431370,284314 0,794118 0,656863

AdaBoostExample

• Thefinalclassifierforexamples𝑥 outsidethetrainingsetisthengivenbythesignof:

• ∑ 𝛼0 & ℎ0 𝑥02340

• whereℎ0 𝑥 isthe𝑘:thclassifierreturning+1forpassingclassificationand-1otherwise.

AdaBoostwithKinect

• ThisisthealgorithmthatisinuseinthegesturedetectionsoftwarethatcomeswiththeKinectSDK.

• Whiletheexampleusedstringsofintegers,theKinectSDKusesthereconstructedskeletonfromthepointcloudastrainingdata.

• Fromthisitextractsfeatures,suchasangles,differenceinposition,speed,etc.

UsingtheKinectSDK

• TheKinectSDKcomeswithawealthofsoftwaretocreateadatabase.

• FirstisKinectStudio,withwhichonecancapturefootagefromtheKinect.

• SecondisVisualGestureBuilder,withwhichonecantrainaclassifier.

• ThirdisVisualGestureBuilderViewer,withwhichonecanverifyhowwelltheclassifierworks.

KinectStudio

KinectStudio

VisualGestureBuilder

• Inthebuildereachcapturedframeoffootageisconsideredoneexampleinthetrainingset.

• Theseframesarethentaggedasfulfilling/notfulfillingthecriterion.

• TheBuilderisthenusedtobuildadatabase,i.e.determinewhatclassifiersbestdeterminewhetheragestureisactiveornot.

VisualGestureBuilderViewer

VisualGestureBuilder

VisualGestureBuilderViewer

• TheVisualGestureBuilderVieweristhenusedtoexaminehowwellthedatabaseclassifiesgestures.

Approachtotheproblem

• Theapproachusedwasthus:– CapturefootagewithKinectStudiowherethegestureswereperformed.

– Tagtheframesofthefootageduringwhichthegestureswereperformed.

– LettheGestureBuilderextracttheappropriatefeaturesandbuildadatabaseusingAdaBoost.

– EvaluateitsaccuracywiththeGestureBuilderViewer.

– Repeatuntiltheendofterm.

Approach

• ThegesturetobecapturedwasiftheuserispointinginoneofthefourdirectionsUp,Down,LeftorRight.Additionallytherewasonegenericgesturejustforpointing.

• Thefirstclipwasofmepointingstraightinthesedirectionssixtimes,threeperhand.

Featuresafterfirstclip

• Afterthefirstclip,the#1featuresdiscoveredineachdirectionwas:– Up:Angles(WristLeft,Head,WristRight)>=180– Down:Angles(ThumbRight,HandRight,WristRight)>=130

– Left:MuscleTorqueZ(SpineShoulder)>=2– Right:Angles(Head,ShoulderRight,SpineBase)<126

• Icontinuedaddingclipsinthisfashion,sothatthefeaturesusedwouldbemorecorrectlyused.

• AtthispointIdiscoveredthatitwasphysicallyexhaustingtostandinthepositionstheclassifierdemanded.

• Ithereforewentbackthroughthealreadytaggedclipstoextendthetimethearmwouldbeconsideredpointing.

Intheaboveuppointingmovement,themiddleonesusedtobethestartandfinishoftheframestaggedpointStraightUp.

• Thiscontinueduntilthetermended.• Inthefinalformtherearetwodatabases.– Onewhichhasbeentrainedtoregardaclosedfistasnotpointing.

– Onewhichhasnot.• ThisisduetoKinect’sinbuilthandStatebeingquiteuncertainwhentestingitonline.

Evaluation

• Thefinaldatabaseis,accordingtomypersonalevaluations,quiteaccurateinrealizingthattheuserispointingandinwhatdirection.

• Itislesscertaininsomedirections.• Inthedatabasethatissupposedtoignoreaclosedfistthedatabaseoftenfailwiththatverytask,sofurthertrainingwouldbenecessary.

• Howsuccessfulthedatabaseiscanvary.

Pointdown

Pointdown

Pointleft

Pointleft

Pointright

Pointright

Pointup

Pointup

Doublepointing

Fingerdown