Post on 13-Apr-2017
A Short Introduction toComputer Vision-based
Marker Tracking
Dr. Jens Grubertjg@jensgrubert.de
Partly based on materials by Dieter Schmalstieg, Mark Billinghurst, Hartmut Seichter, Daniel Wagner
Dr. Jens Grubert | jg@jensgrubert.de
SpatialRegistrationDefiningRelativePositionofEachElements ofaSceneusingCoordinateSystems(CS)
WorldCS
Camera/SensorCS
Initialposeestimation: RegistrationTemporalupdates givenaninitialpose:Tracking
Imagecourtesy
bySesameWorkshop
http://w
ww.sesam
eworkshop.org/our-b
log/tag/new-technology/
Dr. Jens Grubert | jg@jensgrubert.de
OverviewTrackingMagneticTracker InertialTracker UltrasonicTrackerVision Tracker
Marker-BasedTracking MarkerlessTracking
SpecializedTracking
RecursiveTracking TrackingbyDetection
Edge-BasedTracking Template-BasedTracking
InterestPointTracking
MechanicalTracker
MonocularTracking
DepthBased
Dr. Jens Grubert | jg@jensgrubert.de
MarkerTracking
• Doneformorethan15years• Severalopensourcesolutionsexist• Fairlysimpletoimplement
• Standardcomputervisionmethods
• Arectangularmarkerprovides4cornerpoints• Ú enoughforposeestimation
Dr. Jens Grubert | jg@jensgrubert.de
MarkerBasedTracking– Examples
4. Marker types and identification
90
�
© jamesmarsh.com
Figure 60. Examples of visual designs: On the upper left: QR code used in Red Hot Chilli Peppers’ ‘I’m with you’ album campaign. On the upper right: artwork by James Marsh, ‘Barcode butterfly’ © jamesmarsh.com (Image courtesy of James Marsh). On the bottom: QR codes designed by JESS3 [134] (images courtesy of JESS3).
If the visual appearance plays a very high role in application, it might be a good idea to spend some effort on professional design (see Figure 60).
4.4.5 General marker detection application
Besides augmented reality, markers are widely used for other purposes as well. Our mobile phone application [3, 11, 17] automatically launched the appropriate application depending on the marker content. If a marker contained a phone num-ber, it launched a phone application. Likewise, it launched a web browser if a marker contained a URL. We also proposed GPS as one type of data: users could get the GPS location from a marker. This would enable location-based AR brows-ers on mobile devices that are not equipped with GPS and at indoor locations where GPS is inoperative.
In a general AR browser scenario, we would like to attach hyperlinks to objects (cf. Internet of Things). However, the space of all possible objects is far too big to handle. We see several solutions to overcome this issue. For example, one could limit the browser to one type of data at a time. Current location-based browsers have different environments. In practice, the “bookstore application” would recog-
Dr. Jens Grubert | jg@jensgrubert.de
MarkerTrackingPipelineOverview
0:Input 1:Threshold 2:Contours
3:MarkerCandidates4:PatternChecking5,6:CornerRefinement,Poseestimation
Dr. Jens Grubert | jg@jensgrubert.de
9
1-3:Fiducial Detection1. Thresholdthewholeimage2. Contours
1. Searchscanline perscanline foredges(whitetoblacksteps)
2. Followedgesuntileither:BacktostartingpixelorImageborder
3. Contourapproximationusingpolygons,sanitychecks(4corners,size)
Dr. Jens Grubert | jg@jensgrubert.de
10
4:Patternchecking
• Calculatehomography usingthe4cornerpoints• DirectLinearTransformalgorithm• Mapsnormalized coordinatestomarker coordinates(simpleperspective projection,nocameramodel)
• Extractpatternbysampling• Checkpattern
• Id(implicitencoding)• Template (normalized crosscorrelation)
• 42D-3Dcorrespondences~poseestimation
Dr. Jens Grubert | jg@jensgrubert.de
5:CornerRefinement
• Refinecornercoordinates• Criticalforhighqualitytracking• Remember:4pointsisthebareminimum!• Sothese4pointsshouldbetterbeaccurate…
• Detectsub-pixelcoordinates• E.g.,Harriscornerdetector• Specializedmethodcanbefasterandmoreaccurate• Stronglyreducesjitter!
• Undistortcornercoordinates• Removeradialdistortionfromlens
Dr. Jens Grubert | jg@jensgrubert.de
6:PoseEstimation
• Calculatesmarkerposition/rotationrelativetocamera
• PnPalgorithmwith42D-3Dcorrespondences• Alternative:UseHomography
• Directestimationfrom42D- 2Dcorrespondences• Veryfast,butcoarse• Jittersalot…
Dr. Jens Grubert | jg@jensgrubert.de
Now it‘s your turn
• Downloadmarkertrackerexamplefromhttps://bitbucket.org/jens_grubert/markertracker• Built the code using CMAKE(https://cmake.org/)• RunmarkerTrackerTestwithparameters:
markerTrackerTest <cameraId> <camera.xml> <markerSize>
e.g.:
markerTrackerTest 0 lc910.yml 9.2
Dr. Jens Grubert | jg@jensgrubert.de
MainTrackingMethodbool MarkerDetectorBW::findMarkers(const cv::Mat& frame, std::vector<MarkerBW>& detectedMarkers)
{
// Convert the image to grayscale
prepareImage(frame, m_grayscaleImage);
// Make it binary
performThreshold(m_grayscaleImage, m_thresholdImg);
// Detect contours
findContours(m_thresholdImg, m_contours, m_grayscaleImage.cols / 5);
// Find closed contours that can be approximated with 4 points
findCandidates(m_contours, detectedMarkers);
// Find is them are markers
recognizeMarkers(m_grayscaleImage, detectedMarkers);
// Calculate their poses
estimatePosition(detectedMarkers);
//sort by id
std::sort(detectedMarkers.begin(), detectedMarkers.end());
return false;
}Dr. Jens Grubert | jg@jensgrubert.de