Lab 5: Vuforia + Unity - web.cecs.pdx.edu
Transcript of Lab 5: Vuforia + Unity - web.cecs.pdx.edu
Lab 5: Vuforia + Unity
CS410/510: Virtual RealityEhsan Aryafar [email protected] Shippey [email protected]
1
Review from last week● Vuforia: What/why● Go to https://developer.vuforia.com/● Sign up for an account● Upload an image under “target manager”● When choosing an image, select for:
○ Strong lines○ Good lighting○ <2Mb size
2
2
Review from last week● Key AR concepts
○ Marker-based AR vs Markerless AR○ A really rough view of how object detection works○ Know what SLAM means and why it’s important
3
3
Review from last week● Key AR concepts
○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,
within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from
there by adding things to the scene.○ A really rough view of how object detection works○ Know what SLAM means and why it’s important
4
4
Review from last week● Key AR concepts
○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,
within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from
there by adding things to the scene.○ A really rough view of how object detection works
■ Classification, bounding boxes, etc. Limited on phones, but works better than you would think.
○ Know what SLAM means and why it’s important
5
5
Review from last week● Key AR concepts
○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,
within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from
there by adding things to the scene.○ Know what SLAM means and why it’s important
■ Simultaneous Localization And Mapping. Depth maps, orientation, etc.
○ A really rough view of how object detection works
6
6
Review from last week● Key AR concepts
○ Marker-based AR vs Markerless AR■ Marker-based AR finds a “marker”, usually an image or pattern,
within a scene. Everything is defined in reference to that marker.■ Markerless AR tries to understand an environment, and works from
there by adding things to the scene.○ Know what SLAM means and why it’s important
■ Simultaneous Localization And Mapping. Depth maps, orientation, etc.
○ A really rough view of how object detection works■ Classification, bounding boxes, etc. Limited on phones, but works
better than you would think.7
7
● Limits to environmental understanding
Addendum to last week
8
8
● Limits to environmental understanding○ Object recognition and SLAM are hard
Addendum to last week
9
9
● Limits to environmental understanding○ Object recognition and SLAM are hard○ Phones have limited hardware and poor thermals
Addendum to last week
10
10
● Limits to environmental understanding○ Object recognition and SLAM are hard○ Phones have limited hardware and poor thermals○ Functionality for cutting edge software isn’t always particularly
user-friendly.
Addendum to last week
11
11
● Although neural approaches to object detection, SLAM, and other AR related tasks such as human segmentation are still hard, they’re now doable, even on phones.
● Even over the web.
Addendum to last week
12
12
Addendum to last week● Some examples from Tensorflow’s javascript API:
○ https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection
○ https://github.com/tensorflow/tfjs-models/tree/master/posenet
13Facemesh Posenet
13
Onto Unity!● Open the Unityhub and we’ll get to work building a super
basic AR application.
14
14
Editor settings● If you plan to build and
deploy this, make sure your platform’s build support is checked in the editor you installed.
● To check, go to Installs in the UnityHub.
15
15
What to do if this doesn’t build● This might or might not build for you● Why this happens
○ As previously discussed, iOS doesn’t necessarily play nice with Windows or Linux to the point where cross compilation is impossible.
○ Sometimes Android is annoying.○ Oftentimes Unity is annoying.
● What to do○ Rejoice! Vuforia can simulate some of this.○ If you really, really want to deploy quickly, we have other
technologies.16
16
Getting started● Create project
17
17
Getting started● Setup editor to work correctly● We want to be able to see:
18
18
Getting started● Setup editor to work correctly● We want to be able to see:
○ Package manager: Window->Package manager
The package manager lets us add packages, which are distinct from Assets in that they somehow extend not just your project’s functionality, but Unity’s as well, both the editor and the engine.
19
19
Getting started● Setup editor to work
correctly● We want to be able to see:
○ Package manager○ Game windows
These should be visible by default, but it’s easier if both the preview and the scene view are both turned on. 20
20
Adding Vuforia support● In the package manager,
find “Vuforia Engine AR”
21
21
Adding Vuforia support● In the package manager,
find “Vuforia Engine AR” ● Click “Install”
22
22
Adding Vuforia support● In the package manager,
find “Vuforia Engine AR” ● Click “Install”● Wait
23
23
Vuforia Camera● AR Cameras are different!● Even with marker-based AR, we still have to do 6dof
tracking.● Vuforia implements its own camera for this reason exactly.● Remember: It matters remarkably little where this camera
is. All positions are computed relative to the marker rather than the camera.
24
24
Vuforia Camera● Add by going to
“GameObjects”● -> Vuforia Engine● -> AR Camera● Delete the main camera.● Position somewhere you’ll
remember it.● Should look basically the
same as the default scene. 25
25
Adding the Vuforia marker● Time to add the markers we made last week through
Vuforia’s developer portal.
26
26
Adding the Vuforia marker● Time to add the markers we made last week through
Vuforia’s developer portal.● Go to the Vuforia developer portal
27
27
Adding the Vuforia marker● Time to add the markers we made last week through
Vuforia’s developer portal.● Go to the Vuforia developer portal● Target manager
28
28
Adding the Vuforia marker● Time to add the markers we made last week through
Vuforia’s developer portal.● Go to the Vuforia developer portal● Target manager● Find the target you created in lab 4
29
29
Adding the Vuforia marker● Time to add the markers we made last week through
Vuforia’s developer portal.● Go to the Vuforia developer portal● Target manager● Find the target you created in lab 4● Download it as a .unitypackage.
30
30
Adding the Vuforia Marker● Back in the Unity editor, go to● GameObject● -> Vuforia Engine● -> Image
31
31
Adding the Vuforia Marker● You should now have an image in the scene!
32
32
Adding the Vuforia Marker● You should now have an image in the scene!● Note: It may be extremely tiny because of the scale. Make
sure it’s there by zooming in.
33
33
Adding the Vuforia Marker● You should now have an image in the scene!● Note: It may be extremely tiny because of the scale. Make
sure it’s there by zooming in.● Time to add our image. Go to● Assets● -> Right click● -> Import package…● -> Custom package…● Find the .unitypackage you downloaded earlier
34
34
Adding the Vuforia Marker● Now, we want to configure the image we
have to match our database.● Click the image on the scene (Or
click+drag to select it)
35
35
Adding the Vuforia Marker● Now, we want to configure the image we
have to match our database.● Click the image on the scene (Or
click+drag to select it)● Examine the object in the inspector.
36
36
Adding the Vuforia Marker● Now, we want to configure the image we
have to match our database.● Click the image on the scene (Or
click+drag to select it)● Examine the object in the inspector.● Change the “database” and “image
target” attributes to point to what you want them to.
37
37
The actual app part● You now have a kind of working app!● Add some assets, as children of the image target.
38
38
The actual app part● You now have a kind of working app!● Add some assets, as children of the image target.● I’m uncreative, so I’ve put a cube here.
39
39
The actual app part● You can now work as if this was a more or less normal
app in Unity.● This part is the part you’re familiar with
40
40
Deployment, further reading● Deployment is intentionally left out of the close
instructions as otherwise it would exclude most of the iOS users.
● As such, as part of the homework, Android users are encouraged (but not required) to work out how to deploy to Android.
● This isn’t required because the sheer volume of software needed to do it makes it borderline unfair.
41
41
Deployment, further reading● Guide for mobile deployment
○ https://learn.unity.com/tutorial/building-for-mobile#5c7f8528edbc2a002053b4a1
● Tips for mobile deployment○ Make sure the phone is connected in the right way according to
the software. My phone is set, by default, to act something like a USB stick. This doesn’t go so great for sending apps to it. Make sure USB Debugging is enabled under developer settings, etc.
○ If your phone gets a bit hot testing, that’s unfortunately kind of normal.
42
42
Deployment, further reading● Deployment is intentionally left out of the close
instructions as otherwise it would exclude most of the iOS users.
● The homework○ Add some more things to your app! Assets, physics, etc. Knock
yourself out.○ Package it up and send it to us.
43
43
Deployment, further reading
44
● Guide for mobile deployment○ https://learn.unity.com/tutorial/building-for-
mobile#5c7f8528edbc2a002053b4a1● Tips for mobile Unity usage
○ Unity has support for mobile input, you can read about that here: https://learn.unity.com/tutorial/touch-input-for-mobile-scripting
○ You might want to minimize the number and complexity of elements on the screen, particularly in regard to physics.
44
Deployment, further reading● Deployment is intentionally left out of the close
instructions as otherwise it would exclude most of the iOS users.
● The homework○ Add some more things to your app! Assets, physics, etc. Knock
yourself out.○ Package it up and send it to us.○ How to do that: Find where it’s installed (Right click Assets->Open
in Containing Folder…), go up a level, package the folder your project is in in a zip, tarball, 500mb git repository, etc.
45
45
Deployment, further reading● Guide for mobile deployment
○ https://learn.unity.com/tutorial/building-for-mobile#5c7f8528edbc2a002053b4a1
● Tips for mobile deployment○ Make sure the phone is connected in the right way according to
the software. My phone is set, by default, to act something like a USB stick. This doesn’t go so great for sending apps to it. Make sure USB Debugging is enabled under developer settings, etc.
46
46
Further reading● Vuforia is easy to use, but certainly not the only AR
platform!● If you’d like to experiment more with Marker based AR,
consider using Three.JS + AR.JS, AWS Sumerian, or any one of around a dozen constantly shifting proprietary solutions.
47
47
Further reading● Vuforia is easy to use, but certainly not the only AR
platform!● If you’d like to experiment more with Marker based AR,
consider using Three.JS + AR.JS, A-Frame, AWS Sumerian, or any one of around a dozen constantly shifting proprietary solutions.
● If you’d like to experiment with markerless AR, try the same set of things, plus Unity’s AR Foundation.
48
48
Let’s experiment● I’ll sit around here to answer more questions if you want to
continue adding things while we’re here.● Feel free to leave and bug me on Slack later, or stick
around and try to debug things with this OR with your project now.
49
49