Sergiy Baydachnyy

Blog about technologies

Archive for the ‘Kinect’ Category

Try to use Kinect with Unity 5

leave a comment »

During Global Developer Conference there were many announcements like Unity 5 availability for free. Of course, Unity has been offering a free version of the editor for many years but it was an editor without many important features. For example it was not possible to use external libraries in free version of the editor. It was the main stopper for many educational projects like projects related to Kinect. From one side we have a very cool solution like Kinect, which researches can use in many interesting projects but from other side, it’s very hard to develop something using DirectX directly. Unity might save the situation but Kinect SDK worked with Pro version only.

But Unity decided to change business model and announced that since Unity 5 there is Personal Edition, which contains all features.

image

Of course, I decided to return to my projects with Kinect and tried to rebuild some of them using the Unity 5 Personal. And there is great news – everything works fine! So the last stopper was broken and today you can try to use Kinect and Unity for free.

If you are interested in the topic I would recommend to use my previous article about Kinect as an instruction how to start.

Written by Sergiy Baydachnyy

03/11/2015 at 6:11 AM

Posted in Kinect, Unity 3D

Tagged with

Kinect 2 and Unity 3D: How to…

with 9 comments

In one of my earlier posts I already told about my first experience in Kinect and Unity integration. But at that time I was able to download only the beta version of the package, which didn’t support Face, Face HD and Fusion APIs there. Today, we have access to a release version of Kinect for Windows SDK 2.0 as well as access to Unity Pro packages and if you want to download them right now, you can find all needed links here http://www.microsoft.com/en-us/kinectforwindows/develop/downloads-docs.aspx.

Of course, you still need a Unity Pro for Kinect functionality but if you want to test some features right now, it’s easy to activate 30-days trial version of Unity Pro. It should be enough in order to understand some features of the Kinect as well as to decide if you want to start a business right now.

So, if you already downloaded the Unity Pro package, you could notice that it contains three packages inside (.unitypackage files). The first file Kinect.2.0.1410.19000.unitypackage contains base functionality of Kinect SDK for Unity. It will allow to track bodies, leans, colors and so on. But if you want to use functionality, which relates to face (emotions, face HD tracking etc.) you will need the second package – Kinect.Face.2.0.1410.19000.unitypackage. The last package contains API which will help to use data from Visual Gesture Builder in order to simplify a way to understand predefined gestures.

Before I start to create some code, I want to drag your attention to the hardware part. In order to start working with Kinect 2 SDK you need a Kinect 2 sensor. I know that the last sentence looks stupid but it’s realityJ I just found in Microsoft Store that Kinect 2 for Windows costs around 200 dollars – it’s very expensive, if you just want to test something. But Microsoft announced a solution of this problem as well. If you have Kinect for Xbox One, you will able to buy a special Kinect adapter for Windows, which will allow you to connect existing sensor to PC. The adapter costs around 50 dollars, which is much cheaper than a new Kinect sensor. Because I already have Xbox One, I decided to buy adapter only.

clip_image002

The adapter has a pretty big box, because it’s not just connector to USB. In general, Kinect requires more power than USB can provide, so the adapter allows to connect Kinect to power as well.

Because we already discussed some hardware questions I want to point to some additional requirements there. In order to build something with Kinect you will need to have Windows 8 (x64) operation system, USB 3.0 host controller and DirectX 11 capable graphics adapter.

Finally, we finished with hardware, so let’s create some code.

Let’s start with simple example where we will make some manipulation with cube using the base API. In order to do it I created a new Unity Pro project and imported Kinect.2.0.1410.19000.unitypackage there. Let’s put a cube in front of camera and create a script, which should be associated with the cube. We are going to do all work inside that script.

First of all we need to create some data fields in our class:

private KinectSensor _Sensor;
private BodyFrameReader _Reader;
private Body[] _Data = null;

We are going to use KinectSensor in order to access to Kinect. KinectSensor class provides some properties, which allows us to get sources’ references. Kinect supports several sources:

· BodyFrameSource – provides basic information about tracked people (up to 6) like skeleton information, leans, hand states etc.;

· AudioSource – allows to track a sound source from a specific direction;

· BodyIndexFrameSource – shows if a particular pixel relates to body or to background;

· ColorFrameSource – gets a copy of video image, which Kinect got from the camera;

· DepthFrameSource – each pixel of this frame represents a distance between Kinect and tracked objects (up to 4.5 meters);

· InfraredFrameSource – supports black and white frame, which looks good with any sources of lights;

· LongExposureInfraredFrameSource – looks like infrared frame but it supports better quality with less noise. It requires a longer period of time in order to get data;

All of these properties support OpenReader method, which returns the appropriate reader. KinectSensor also supports OpenMultiSourceFrameReader, which allows to get data from several readers just using a single line of code.

Since we need to know basic body movements only, we will use just BodyFrameReader. Additionally, we need an array of Body class in order to store current information about the body.

Right now we are ready to write some initialization methods. Unity supports Start method for MonoBehaviour classes in order to provide a place for initialization. We will use this method to prepare out Kinect and reader.

void Start () 
{
_Sensor = KinectSensor.GetDefault();

if (_Sensor != null)
{
_Reader = _Sensor.BodyFrameSource.OpenReader();

if (!_Sensor.IsOpen)
{
_Sensor.Open();
}
}
}

In the previous post about Kinect I created bad code and forgot to dispose my objects. It worked fine there but if you make the same mistake here you will be able to get data from Kinect just for the first launch of your game inside Unity. After it you will not able to get any data as well as exceptions or something like it. So, in order to avoid having to restart Unity every time when you launch your application inside, we should include OnApplicationQuit method as well. We will call Dispose method for our objects there.

void OnApplicationQuit()
{
if (_Reader != null)
{
_Reader.Dispose();
_Reader = null;
}

if (_Sensor != null)
{
if (_Sensor.IsOpen)
{
_Sensor.Close();
}
_Sensor = null;
}
}

Right now we are ready to implement Update method, which will be called on each frame update. This method will contain less Kinect related code. We need to get last frame using AcquireLatestFrame method and, in case if frame exists, we need to initialize Body array. Because Kinect supports up to 6 bodies, we need to create array based on this number but BodyFrameSource supports BodyCount property as well.

Please, don’t forget to dispose frame right after initialization of Body array.

Additionally, we should understand if at least one body was tracked. In order to do it we may use IsTracked property in Body class. If at least one body is tracked we will use its index in order to start moving our cube. Here is my version of the Update method:

void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();

if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}

frame.GetAndRefreshBodyData(_Data);

frame.Dispose();
frame = null;

int idx = -1;
for (int i = 0; i < _Sensor.BodyFrameSource.BodyCount; i++)
{
if (_Data[i].IsTracked)
{
idx = i;
}
}
if (idx>-1)
{
if (_Data[idx].HandRightState != HandState.Closed)
{
horizontal =
(float)(_Data[idx].Joints[JointType.HandRight].Position.X
* 0.1);
vertical =
(float)(_Data[idx].Joints[JointType.HandRight].Position.Y
* 0.1);

if (firstdeep == -1)
{
firstdeep =
(float)(_Data[idx].Joints[JointType.HandRight].Position.Z
* 0.1);
}
deep =
(float)(_Data[idx].Joints[JointType.HandRight].Position.Z
* 0.1) - firstdeep;

this.gameObject.transform.position = new Vector3(
this.gameObject.transform.position.x + horizontal,
this.gameObject.transform.position.y + vertical,
this.transform.position.z + deep);
}
if (_Data[idx].HandLeftState != HandState.Closed)
{
angley =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.X );
anglex =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.Y);
anglez =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.Z);

this.gameObject.transform.rotation =
Quaternion.Euler(
this.gameObject.transform.rotation.x+anglex * 100,
this.gameObject.transform.rotation.y+angley * 100,
this.gameObject.transform.rotation.z+anglez * 100);
}
}
}
}
}

As you can see, I used HandLeftState and HandRightState in order to change properties of cube. User will be able to “close” his hand in order to avoid cube movement. In order to use z axis I initialize deep variable in current position of user’s hand by z because the axis shows distance between Kinect and user’s hand. But thanks to that initialization I am able to move cube backward or forward by z.

Next time we will create more advanced examples based on DepthFrameSource and AudioSource.

Written by Sergiy Baydachnyy

11/20/2014 at 10:21 PM

Posted in Kinect, Unity 3D

Tagged with

Kinect, Unity 3D and Debugging

with one comment

In this article, I was planning to make additional investigation of Kinect 2.0 SDK but I found a better theme – Debugging of Unity 3D applications using Visual Studio. Because I am not a professional developer in Unity 3D yet, I never thought about debugging of Unity 3D applications. But in the last article, I wrote about my first experience in Kinect and Unity integration and, frankly speaking, I had a headache during preparation of my demo. It happened due to the lack of ability to debug my demo scripts. Of course, Kinect 2 SDK was new for me, so I used many Debug.Log methods in order to understand what was happening with my variables.

I know that Mono Develop allows to debug Unity scripts but I don’t like this tool. So, I did not debug my scripts for some time and returned to this idea only two days ago. I was very surprised that there is an easy way to enable Debugging tools in Visual Studio. In order to do it you need to download and install Visual Studio Tools for Unity.

These tools were created by SyntaxTree team, which was bought by Microsoft in June-July this year. At once Microsoft announced availability of the tools for all developers for free and the first Microsoft version of them was presented in the end of July. The tools are integrated with Visual Studio 2010, 2012 and 2013 versions, so you can use them with older version of Visual Studio.

I have Visual Studio 2013, so I simply installed the package and enabled debugging in my project.

In order to enable debugging in an existing project you need to import package to it. If you install the tools successfully, you will find a new menu item there.

clip_image002

If you create a new project you will be able to add the package right in Create New Project dialog.

clip_image003

The package will add some scripts to your project, which will extend your standard menu. Additionally, the package contains some components that will support communication channel between Visual Studio and Unity.

clip_image004

In my case, I imported the package successfully, so I opened project in Visual Studio in order to start debugging. It was pretty simple, I just put some breakpoints there and clicked the F5 hot key in order to start debugging. Visual Studio switched to the Debug mode and I returned to Unity in order to start my application there. After that, I just clicked Start button in Unity and, finally, I was moved to Visual Studio in order to see a fired breakpoint.

clip_image006

You will be able to click and unclick Play (Pause) button as many time as you want. It will restart your game or just pause it but it will not affect Visual Studio which will stay in the Debug mode. If you want to exit the Debug mode in Visual Studio, you should just click Stop Debugging there.

To summarize, I spent a minute but I got a very powerful tool for my future Unity projects.

Written by Sergiy Baydachnyy

10/05/2014 at 10:30 AM

Kinect 2 and Unity 3D: My first experience

with 6 comments

Several days ago, I got my first Kinect 2 for Windows devices in order to prepare my presentation for Developer Day event in Ukraine together with Dmytro Mindra. Right now, I already have some experience about Kinect and I am going to share it in my blog.

Before we create some code, we will need to download Kinect for Windows SDK 2.0. This SDK is in Public Preview now but it allows to create Windows Store applications (previous version didn’t allow it), so I was happy to install this version of SDK. I found it here: http://www.microsoft.com/en-us/download/details.aspx?id=43661.

Once SDK was installed I was able to review some examples and tools. I would recommend to use SDK Browser, that can be found in the following directory C:\Program Files\Microsoft SDKs\Kinect\v2.0-PublicPreview1409\Tools\SDKBrowser. This is a very useful tool, which will show you the entire SDK structure like tools, examples and link to docs.

clip_image002

I would recommend to use the Body Basic-XAML example in order to understand that your Kinect works fine or you may use Kinect Studio in order to do some experiments with the sensorJ Because my Kinect was working fine I could start to think about some demos.

Kinect is a very powerful device and you can use it in many ways as a sensor for your home robot or as a part of your security system but the most popular way of Kinect usage is for entertainment. It allows to create many good games and many of such games are already available for Xbox. At same time I don’t have any desire to create a game in C++/DirectX and I was trying to find a solution for Unity 3d because it is one of the best game engines and I like Unity 3d.

Some time ago, I heard information about an add-in for Unity 3d but I had to spend much time in order to find it. I used several search systems, then I called Unity 3d guys and finally I found it in a very unusual place – on the Kinect for Windows SDK 2.0 download page. This link is collapsed by default and you need to expand System Requirements section in order to find it.

clip_image004

Pay attention that this add-in will work with Unity PRO version only, because the free version of Unity doesn’t support add-ins. Probably, there is a way to fix it for Windows Store applications because Non-Pro version allows to use add-ins for this type of applications but I will check it later and for now I use a Pro version.

If you download the add-in successfully, you will be able to find the unitypackage file as well as two examples inside of it. We already have a small game, which allows us to kill “friendly green men” with our battle stellar ship and we decided to implement some Kinect ready activities to make killing process funnier.

Finally, we can implement three steps, which will allow us to use our body during the game in killing process.

On the first step, we found our MonoBehaviour that was responsible for movement of our ship and we added some new code to it:

private KinectSensor _Sensor;

private BodyFrameReader _Reader;

private Body[] _Data = null;

This code should be clear but I will describe it. In order to get access to sensor we needed KinectSensor class, which allows to initialize our Kinect and provides needed sources. In our case, we needed access to BodyFrame source, so we created BodyFrameReader variable that we used in order to read data from the source. Finally, we created Body array in order to store our data from the source.

On the next step, we needed to initialize Kinect and open our reader. That’s why we implemented the following code:

void Start()

{

    _Sensor = KinectSensor.GetDefault();

    if (_Sensor != null)

    {

        _Reader = _Sensor.BodyFrameSource.OpenReader();

        if (!_Sensor.IsOpen)

        {

            _Sensor.Open();

        }

    }

}

We initialized the Kinect inside the very familiar method for Unity 3D developers and opened our reader there.

Finally, we realized a method, which was called from Update method of our Behaviour class. This method collected data from the sensor and calculated new position for the ship as well as fire ability:

horizontal = 0;

vertical = 0;

fireButton = false;

if (_Reader != null) {

       var frame = _Reader.AcquireLatestFrame ();

       if (frame != null) {

              if (_Data == null) {

                     _Data = new Body[_Sensor.BodyFrameSource.BodyCount];

              }

              frame.GetAndRefreshBodyData (_Data);

              int idx = -1;

              for (var i = 0; i < 6; i++) {

                     if (_Data [i].LeanTrackingState !=

                       TrackingState.NotTracked) {

                           idx = i;

                     }

              }

              if (idx > -1) {     

                     fireButton =

                        _Data [idx].Joints [JointType.HandLeft].Position.Y > 0;

                     horizontal = _Data [idx].Lean.X;

                     vertical = _Data [idx].Lean.Y;

              }

       }

       frame.Dispose();

}

This code is more complex and Dima said that I should not show it because it’s not optimal but, probably, you will create the same code for your first Kinect application for Unity 3DJ

So, in this code, we used the Reader to grab current (latest) body frame that was needed in order to found bodies data inside. Kinect 2 supports up to 6 bodies, so _Sensor.BodyFrameSource.BodyCount should return 6. Using the latest frame we filled Body array in order to aggregate data for each body in our frame. At this point we finished work with Kinect sensor and tried to analyze bodies data. But, if you have just one ship (in our case we had two) and you don’t have any specific interface for Kinect initialization, you will not know about location of real body in the array. That’s why we were trying to find a specific index of the body of a pilot. Because we used leans as a main control mechanism for our ship we checked every body and we were trying to find LeanTrackingState status. If we find a tracked body we fixed index of this body and recalculated position of the ship as well as the fire ability.

I think that we spent about 30 minutes for this example and our presentation was very funny. If you know Russian, you can find our video there: http://devday.in.ua. And I just cut a short video from our presentation there are two guys (me and Dima) were doing a strange things on the stageJ

In the next article I am planning to integrate Kinect with simple Windows 8 application and will discuss different sources of data.

Written by Sergiy Baydachnyy

10/03/2014 at 10:18 PM

Posted in Kinect, Windows 8

Tagged with ,