Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk,...
-
date post
19-Dec-2015 -
Category
Documents
-
view
223 -
download
2
Transcript of Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk,...
![Page 1: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/1.jpg)
Low latency rendering for mobile augmented reality
W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen
Delft University of TechnologyUbiCom – http://www.ubicom.tudelft.nl
![Page 2: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/2.jpg)
System sketch
![Page 3: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/3.jpg)
Way finding
![Page 4: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/4.jpg)
![Page 5: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/5.jpg)
Maintenance, assistance
![Page 6: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/6.jpg)
How to do this?
1. System overview
2. Why low latency3. Low-latency rendering4. Low-latency tracking
5. Status of project
![Page 7: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/7.jpg)
1. System overview
InternetBasestationBackboneMobile
unitBase
stationMass
storageComputepower
Mobile unit off-loads processes to backbone
![Page 8: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/8.jpg)
Camera serves 2 purposesViewing through camera of another user Tracking the position of the user
Mobile unit hardware17GHz, IR
TransmissionCompressAnalysisvideo outTrackingDecom-press
Render
![Page 9: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/9.jpg)
2. Latency requirements
Accurate positioning requires low latencyMost demanding situation: head rotation
Alignment error depends on lag and rotational speed
See-throughdisplayReal objectMisalignedvirtual object
![Page 10: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/10.jpg)
Humans can detect <8ms
But hardly any literature on relation with task.
Estimated for mentioned tasks:Max. error 0.5˚ @ 50˚/s latency required ≤ 10 ms
![Page 11: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/11.jpg)
VR: several sources of latency
Source of latency typical3D Rendering 30-200 msTracking 15-40 msDisplay @50Hz 20 ms
Fastest reported: 17 (33)ms latency
Our goal: 10 msRendering + Display 8msTracking 2ms
![Page 12: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/12.jpg)
Std. Voodoo 3D game card
Render just ahead of raster beam
4 partitionsgive latency5-10 ms
20msDisplaying here
Rendering hereTo be cleared and rendered
3. Low-latency rendering
![Page 13: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/13.jpg)
~ hundred texture-mapped polygons Simplify virtual world
In backbone, using LOD polygon reduction polygon-to-texture techniques
Feasible: relatively small #virtual objects
![Page 14: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/14.jpg)
ApplicationdatabasesNetworkresourcesComputestationsimplified sceneBasestation
![Page 15: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/15.jpg)
Latency-layered system
complexvirtualobjects
simplevirtualobjects
displaylistDisplay>200HzHeadsetBackbone~10Hz<1HzPositionfrom inertial tracker
Position from camera and prediction
Usermovement~10ms~100ms~1s
![Page 16: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/16.jpg)
PrototypeKURT Realtime Linux to• schedule renderer at 200Hz• in sync with displayLatency measured: 4.2-8.8ms (6.35 avg)
TrackerSimilar latency-layered techniques as rendering, using• inertial tracker in low-latency loop• camera feature tracker in outer loop
![Page 17: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/17.jpg)
Prototypeoff-the-shelf componentsNot yet wireless (tracker & power cables)
Rendering system• “VRML2” scene graphs• changed animation:
no 200Hz scene graph traversal-> compile @10Hz to display list @ 200Hz-> tags in displaylist for last-minute updates
5. Status of project
![Page 18: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/18.jpg)
Tracking• Proto: Intersense IS600 (inertial+acoustic)• Working on optical tracking with artificial feducials• Later: natural feducials
Future• Fast & simple hardware for rendering, combining well with video transmission• Task - latency relation • Applications• Incremental scene simplification
![Page 19: Low latency rendering for mobile augmented reality W. Pasman, A. van der Schaaf, R.L. Lagendijk, F.W. Jansen Delft University of Technology UbiCom – .](https://reader036.fdocuments.us/reader036/viewer/2022081512/56649d3f5503460f94a1880b/html5/thumbnails/19.jpg)
ConclusionAccurate alignment for ARLatency-layered system architectureFirst-ever <10ms rendering (no prediction)
www.ubicom.tudelft.nl