Hololens developing: How to use Holotoolkit and Understanding HoloToolkit (5), How Cursor works

Now let's look at Cursor, how it works
Click BasicCursor and notice the ObjectCursor script

Double click the ObjectCursor.cs file, and Click F12 on Cursor parent class

keep finding its void Start() method

haha, what I highlighted here is what I was using in my previous post, GazeManager.Instance, yes, Gaze is using GazeManager for detecting the outer world!

let keep finding void Update() method

they are using gazeManager.HitObject which is what I was using in previous post

and finally they using transform.location, rotation to relocate the Cursor's position and its status

this is how cursor works in Holotoolkit, i skipped lots of detail, but just give out a hint how Gaze and GazeManger, InputManager are related together

next post, let use gesture to rotate the Cube。

Hololens developing: How to use Holotoolkit and Understanding HoloToolkit (4), Show Gaze info on the Text

In last post, we put a text message on top-left corner of screen, now we are going to add a c# script onto the text, letting it show what the user is looking at

First add a tag to the cube, name is as "cool cube"

Create a "Scripts" folder and add a c# script ShowObjectName.cs

Edit the script in Visual Studio,

Save and assign this code to the Text

play the scene in Unity, you can see the text is now showing what the cursor is aiming at

now let's explain these codes:

GameManager.Instance, if you add an InputManager into the Hierarchy, there will be a GazeManger static instance, it is a Singleton pattern, so in one scene, there will be only one GazeManager existing

when GazeManager.HitObject is not null, try to read the gameObject's tag and assign it to the Text we are showing. otherwise, showing none

Hololens developing: How to use Holotoolkit and Understanding HoloToolkit (3), Add a Text on the screen

Before we introduce Gaze, I want to put a text string on the top-left corner of the screen in order to show any message, for example, when the are looking at the cube, we want to show the user the cube's name from the text message.

Steps:

1)      Add a text to the Hierarchy

=> 

let's talk a little bit about this Canvas, if you click Canvas and check its properties:

you can find the "Render Mode" is set to "Screen Space - Overlay" which means, the Canvas and its children(like Text) will always fit whatever the user see from screen, here it's our hololens view

The "Render Mode" could be set to other values, but we are not interested for now, if you want more detail, please check unity website

http://docs.unity3d.com/Manual/UICanvas.html 

Click Text and change the properties to be like this:

change the parameters, make sure you can see the text message on the top-left corner of the screen

done, next chapter we will make a c# script to let the message show what the user is looking at

Hololens developing: How to use Holotoolkit and Understanding HoloToolkit (2), GazeManager.HitInfo

In last post, we created a simple Holotoolkit project, created a small Cube, when the user looking at (gaze aiming at) the cube, a cursor will appear on the surface of the cube indicting where you are looking at.

How is everything happened?

let's split the problem into some layers, basic by basic

1) How Hololens can "see" some objects? 

   You can image like this: Hololen device send some "wave" from the sensor in the front of it, when the wave hit object, it will hit back to the receive sensor telling Hololens it hit something, carrying back hit angle, hit distance, and other hit informations. You are right, it is just like a Radar in the battlefield.

Although it works very similar to a radar,  actually it doesn't send any physic wave when hololens try to see anything. Hololens do have sensors to send physic waves, but that is for Spatial Mapping(we will talk about SpatialMapping in later chapters) which hololen try to map and build 3D environment data around the user, Spatial Mapping could be turned off which means hololen knows nothing about the room you are standing in, but still can see the object(like the Cube) you created in Unity;

Hololen can see the object you created because it knows the object's location, its's own location, the direction it is looking at, based on these three parameters, hololens can tell if the object is on the way you are looking at and thus return every parameters you need to react.

Introducing Raycast

Physics.Raycast(Vector3 position, Vector3 direction, out  RaycastHit)
https://docs.unity3d.com/ScriptReference/Physics.Raycast.html

like I said above, it cast a ray from (Vector3 position) in direction (Vector3 direction), if hit anything, return back what it hit (out  RaycastHit).

the method return a bool value indicting if the ray hit something or not.

the Holotoolkit, no matter how complex it is, is using Physics.Raycast to explore the  outer world.

let's see how holotoolkit use Physics.Raycast,

In Hierarchy, click the "Input Manager", see the Inspector, there is a "Gaze Manager" script component;

Open this "Gaze Manager (Script)" component in Visual Studio, find this:

You can see the hitInfo which is a RaycastHit value there, keep looking down and Find out void Update() method.

keep finding RaycastPhysics() method

you can see from the screenshot the hitInfo is updated by GazeManager through out its void Update() method, which is 60 frames per second.

Just ignore all these if..else code in the screenshot for now(we will talk these things later), just to remember, GazeManager.HitInfo property always has the latest hit information of what hololens is looking at.

Next Chapter, I will introduce how Gaze component is using this GazeManager.HitInfo to show itself to the user

Hololens developing: How to use Holotoolkit and Understanding HoloToolkit (1)

I am learning how to develop Hololens now, met lots of challenging, I am going to share some thinking of how to use Holotoolkit

1) What is Holotoolkit?
       Of course you can google it, but for me, using some very simple words to introduce Holotoolkit: it is just a set of codes made by Microsoft to help Hololen developer quickly developing their application

2) Why would I write this blog?
     Holotoolkit is a great set of helping class, but to understand it, not that easy. There are lots of videos or articles from internet teaching you how to use Holotoolkit, place this model in, place that model in, and everything works.
    But how these model works? why these model works? What cause these model works together so well? NO ANY ARTICLE  INTRODUCING ANYTHING
    So I want to write something about what is happening under the hook of Holotoolkit, that's how this blog is coming out

OK, let's begin:

To understanding how holotoolkit works, we have to first understand how Unity codes works, if you don't have basic knowledge of Unity and C# coding, I suggest you learning some basic first.

Holotoolkit or Hololens codes are Unity codes no matter how complex they are, if you create a scene and put some models like this:

From the hierarchy, you can see there are 8 GameObjects listed on the top level, when game started running, something like this will happen(I use C# as the language to simulate what happene at the very beginning)

{
    Task.Factory.StartNew(() =>  new DirectionalLight());
    Task.Factory.StartNew(() =>  new HoloLensCamera());
    Task.Factory.StartNew(() =>  new SpatialMapping());
     ......
    Task.Factory.StartNew(() =>  new Sphere());
}

all these gameObjects will be initialized at the same time, the void Start() and void Update() method of each gameobjects will be invoked one by one.

this is how Unity scripts works in my imagination(if I am wrong, please correct me)

Create a sample Holotoolkit Project

Ok, let's return to Holotoolkit, we first create a empty project "UnderStandingHolotoolKit", like this:

importing Holotoolkit package:

Choose the 2017.1.0 version, and get this result:

Delete the Main Camera and add HoloLensCamera from HoloToolkit.Input.Prefabs
Add a BasicCursor from HoloToolkit.Input.Prefabs.Cursor
Add an InputManager from HoloToolkit.Input.Prefabs

what you see in the Hierarchy should be like this:

Now we have a basic environment using Holotoolkit, normally if you want to play with Hololens, this is the minimum settlement which could not be less, and this setting contains nearly everything you need to continue programming

let's continue;
Save above as a scene(panel1)
the settings above is just a basic frame, you can not see anything if you run this in device or Simulator, let's add a cube, like this:

Try to play in the Unity Editor or device or Emulator, you will see the cursor appearing on the surface of the Cube

Wow, this is cool, we didn't code a single line, and the Holotoolkit help us adding Cursor on the object we created, but how is this working? what happened under the hook?

I will show you the answer on next post