Kinect 2 and Unity 3D: How to…
In one of my earlier posts I already told about my first experience in Kinect and Unity integration. But at that time I was able to download only the beta version of the package, which didn’t support Face, Face HD and Fusion APIs there. Today, we have access to a release version of Kinect for Windows SDK 2.0 as well as access to Unity Pro packages and if you want to download them right now, you can find all needed links here http://www.microsoft.com/en-us/kinectforwindows/develop/downloads-docs.aspx.
Of course, you still need a Unity Pro for Kinect functionality but if you want to test some features right now, it’s easy to activate 30-days trial version of Unity Pro. It should be enough in order to understand some features of the Kinect as well as to decide if you want to start a business right now.
So, if you already downloaded the Unity Pro package, you could notice that it contains three packages inside (.unitypackage files). The first file Kinect.2.0.1410.19000.unitypackage contains base functionality of Kinect SDK for Unity. It will allow to track bodies, leans, colors and so on. But if you want to use functionality, which relates to face (emotions, face HD tracking etc.) you will need the second package – Kinect.Face.2.0.1410.19000.unitypackage. The last package contains API which will help to use data from Visual Gesture Builder in order to simplify a way to understand predefined gestures.
Before I start to create some code, I want to drag your attention to the hardware part. In order to start working with Kinect 2 SDK you need a Kinect 2 sensor. I know that the last sentence looks stupid but it’s realityJ I just found in Microsoft Store that Kinect 2 for Windows costs around 200 dollars – it’s very expensive, if you just want to test something. But Microsoft announced a solution of this problem as well. If you have Kinect for Xbox One, you will able to buy a special Kinect adapter for Windows, which will allow you to connect existing sensor to PC. The adapter costs around 50 dollars, which is much cheaper than a new Kinect sensor. Because I already have Xbox One, I decided to buy adapter only.
The adapter has a pretty big box, because it’s not just connector to USB. In general, Kinect requires more power than USB can provide, so the adapter allows to connect Kinect to power as well.
Because we already discussed some hardware questions I want to point to some additional requirements there. In order to build something with Kinect you will need to have Windows 8 (x64) operation system, USB 3.0 host controller and DirectX 11 capable graphics adapter.
Finally, we finished with hardware, so let’s create some code.
Let’s start with simple example where we will make some manipulation with cube using the base API. In order to do it I created a new Unity Pro project and imported Kinect.2.0.1410.19000.unitypackage there. Let’s put a cube in front of camera and create a script, which should be associated with the cube. We are going to do all work inside that script.
First of all we need to create some data fields in our class:
private KinectSensor _Sensor;
private BodyFrameReader _Reader;
private Body[] _Data = null;
We are going to use KinectSensor in order to access to Kinect. KinectSensor class provides some properties, which allows us to get sources’ references. Kinect supports several sources:
· BodyFrameSource – provides basic information about tracked people (up to 6) like skeleton information, leans, hand states etc.;
· AudioSource – allows to track a sound source from a specific direction;
· BodyIndexFrameSource – shows if a particular pixel relates to body or to background;
· ColorFrameSource – gets a copy of video image, which Kinect got from the camera;
· DepthFrameSource – each pixel of this frame represents a distance between Kinect and tracked objects (up to 4.5 meters);
· InfraredFrameSource – supports black and white frame, which looks good with any sources of lights;
· LongExposureInfraredFrameSource – looks like infrared frame but it supports better quality with less noise. It requires a longer period of time in order to get data;
All of these properties support OpenReader method, which returns the appropriate reader. KinectSensor also supports OpenMultiSourceFrameReader, which allows to get data from several readers just using a single line of code.
Since we need to know basic body movements only, we will use just BodyFrameReader. Additionally, we need an array of Body class in order to store current information about the body.
Right now we are ready to write some initialization methods. Unity supports Start method for MonoBehaviour classes in order to provide a place for initialization. We will use this method to prepare out Kinect and reader.
void Start ()
{
_Sensor = KinectSensor.GetDefault();
if (_Sensor != null)
{
_Reader = _Sensor.BodyFrameSource.OpenReader();
if (!_Sensor.IsOpen)
{
_Sensor.Open();
}
}
}
In the previous post about Kinect I created bad code and forgot to dispose my objects. It worked fine there but if you make the same mistake here you will be able to get data from Kinect just for the first launch of your game inside Unity. After it you will not able to get any data as well as exceptions or something like it. So, in order to avoid having to restart Unity every time when you launch your application inside, we should include OnApplicationQuit method as well. We will call Dispose method for our objects there.
void OnApplicationQuit()
{
if (_Reader != null)
{
_Reader.Dispose();
_Reader = null;
}
if (_Sensor != null)
{
if (_Sensor.IsOpen)
{
_Sensor.Close();
}
_Sensor = null;
}
}
Right now we are ready to implement Update method, which will be called on each frame update. This method will contain less Kinect related code. We need to get last frame using AcquireLatestFrame method and, in case if frame exists, we need to initialize Body array. Because Kinect supports up to 6 bodies, we need to create array based on this number but BodyFrameSource supports BodyCount property as well.
Please, don’t forget to dispose frame right after initialization of Body array.
Additionally, we should understand if at least one body was tracked. In order to do it we may use IsTracked property in Body class. If at least one body is tracked we will use its index in order to start moving our cube. Here is my version of the Update method:
void Update()
{
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
frame.GetAndRefreshBodyData(_Data);
frame.Dispose();
frame = null;
int idx = -1;
for (int i = 0; i < _Sensor.BodyFrameSource.BodyCount; i++)
{
if (_Data[i].IsTracked)
{
idx = i;
}
}
if (idx>-1)
{
if (_Data[idx].HandRightState != HandState.Closed)
{
horizontal =
(float)(_Data[idx].Joints[JointType.HandRight].Position.X
* 0.1);
vertical =
(float)(_Data[idx].Joints[JointType.HandRight].Position.Y
* 0.1);
if (firstdeep == -1)
{
firstdeep =
(float)(_Data[idx].Joints[JointType.HandRight].Position.Z
* 0.1);
}
deep =
(float)(_Data[idx].Joints[JointType.HandRight].Position.Z
* 0.1) - firstdeep;
this.gameObject.transform.position = new Vector3(
this.gameObject.transform.position.x + horizontal,
this.gameObject.transform.position.y + vertical,
this.transform.position.z + deep);
}
if (_Data[idx].HandLeftState != HandState.Closed)
{
angley =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.X );
anglex =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.Y);
anglez =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.Z);
this.gameObject.transform.rotation =
Quaternion.Euler(
this.gameObject.transform.rotation.x+anglex * 100,
this.gameObject.transform.rotation.y+angley * 100,
this.gameObject.transform.rotation.z+anglez * 100);
}
}
}
}
}
As you can see, I used HandLeftState and HandRightState in order to change properties of cube. User will be able to “close” his hand in order to avoid cube movement. In order to use z axis I initialize deep variable in current position of user’s hand by z because the axis shows distance between Kinect and user’s hand. But thanks to that initialization I am able to move cube backward or forward by z.
Next time we will create more advanced examples based on DepthFrameSource and AudioSource.
Sergii Baidachnyi 
Hello,
I started earlier this month to develop with KinectV2 & Unity3D. I succeeded to use the Face API and did everything i wanted on Unity. However, when came the moment to build the project and put the target as a Windows Store App (Windows 8.1), i couldn’t start the application with the Kinect running.
I did everything i should :
– Enabled the Microphone and Webcam capabilities
– Change the platform to x64.
I’ve spent a lot of time searching the internet about solutions on this problem, but couldn’t find anything useful..
Either i get the references aren’t good, or the app just craches on load.
Have you been able to run a Windows Store App made with Unity using the Face API ?
Thanks
Kévin LANCIEN
11/24/2014 at 7:44 PM
Could you send a direct email to sbaydach@microsoft.com?
Sergiy Baydachnyy
11/27/2014 at 2:58 AM
Sorry, i add the error i get to complete my previous post:
Error 1 The command “echo UnityInstallationDir ‘C:\Program Files (x86)\Unity\Editor’
echo UnityProjectDir ‘C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth’
echo Copying assemblies…
copy /Y “C:\Users\Kev\Documents\UnityProjects\Earth\w8\Earth\Unprocessed\*” “C:\Users\Kev\Documents\UnityProjects\Earth\w8\Earth\”
copy /Y “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Unprocessed\Assembly-CSharp.dll” “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Assembly-CSharp.dll”
copy /Y “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Unprocessed\Assembly-CSharp-firstpass.dll” “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Assembly-CSharp-firstpass.dll”
if exist “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Unprocessed\Assembly-CSharp.pdb” copy /Y “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Unprocessed\Assembly-CSharp.pdb” “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Assembly-CSharp.pdb”
if exist “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Unprocessed\Assembly-CSharp-firstpass.pdb” copy /Y “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Unprocessed\Assembly-CSharp-firstpass.pdb” “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Assembly-CSharp-firstpass.pdb”
echo Running AssemblyConverter…
“C:\Program Files (x86)\Unity\Editor\Data\PlaybackEngines\MetroSupport\Tools\AssemblyConverter.exe” -platform=wsa81 “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Assembly-CSharp.dll” “C:\Users\pdaukin\Documents\Visual Studio 2013\Projects\Unity\Earth\bin\Store 8.1\x86\Debug\Assembly-CSharp-firstpass.dll” “C:\Users\Kev\Documents\UnityProjects\Earth\w8\Earth\\Assembly-UnityScript-firstpass.dll” “C:\Users\Kev\Documents\UnityProjects\Earth\w8\Earth\\Boo.Lang.dll” “C:\Users\Kev\Documents\UnityProjects\Earth\w8\Earth\\UnityEngine.dll” “C:\Users\Kev\Documents\UnityProjects\Earth\w8\Earth\\WinRTLegacy.dll”
echo AssemblyConverter done.
” exited with code 1. Earth
Thank you in advance for any help you could provide !
Kévin LANCIEN
11/24/2014 at 8:11 PM
[…] you are interested in the topic I would recommend to use my previous article about Kinect as an instruction how to […]
Try to use Kinect with Unity 5 | Sergiy Baydachnyy
03/11/2015 at 6:11 AM
Hi Sergiy, im newbie using kinect v2 + Unity 5, Actually I’m trying to use your script, but i dont see where you declare these horizontal, vertical, deep, firstdeep, angley,anglex,anglez. I tried to declare global like a float but it still doesn’t works…Could someone help me? Thanks in advance
Adria
04/23/2015 at 12:26 PM
Let’s start with this code:
public class CubeB : MonoBehaviour {
private KinectSensor _Sensor;
private BodyFrameReader _Reader;
private Body[] _Data = null;
// Use this for initialization
void Start () {
_Sensor = KinectSensor.GetDefault();
if (_Sensor != null)
{
_Reader = _Sensor.BodyFrameSource.OpenReader();
if (!_Sensor.IsOpen)
{
_Sensor.Open();
}
}
}
void OnApplicationQuit()
{
if (_Reader != null)
{
_Reader.Dispose();
_Reader = null;
}
if (_Sensor != null)
{
if (_Sensor.IsOpen)
{
_Sensor.Close();
}
_Sensor = null;
}
}
// Update is called once per frame
void Update () {
if (_Reader != null)
{
var frame = _Reader.AcquireLatestFrame();
if (frame != null)
{
if (_Data == null)
{
_Data = new Body[_Sensor.BodyFrameSource.BodyCount];
}
frame.GetAndRefreshBodyData(_Data);
frame.Dispose();
frame = null;
int idx = -1;
for (int i = 0; i -1)
{
if (_Data[idx].HandLeftState != HandState.Closed)
{
float angley =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.X);
float anglex =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.Y);
float anglez =
(float)(_Data[idx].Joints[JointType.HandLeft].Position.Z);
this.gameObject.transform.rotation =
Quaternion.Euler(
this.gameObject.transform.rotation.x + anglex * 100,
this.gameObject.transform.rotation.y + angley * 100,
this.gameObject.transform.rotation.z + anglez * 100);
}
}
}
}
}
}
Sergiy Baydachnyy
04/23/2015 at 8:21 PM
Hi,
I’m also a newbie using both Kinect 2 and Unity, and I wanted to try out both codes (the one from the post and the one from the comment) but I always get the same error at some point:
Error CS0029 Cannot implicitly convert type ‘int’ to ‘bool’
In the case of the code from the comment both errors I get (the one presented above and CS1002 ; expected ) concern this line:
for (int i = 0; i-1)
I will go through your previous posts, cause this is the first time I’m here, but maybe someone had similar problems.
All the best and keep up the good work!
KAT
11/02/2015 at 5:13 PM
hello! I am geeta I try your code which you provide on this link.
I am getting error ‘can not convert bool to int explicit’ on for(I=0;i-1) how can I solve that please help.
Geeta
04/07/2017 at 9:07 AM
Hi Sergiy.
is there any full documentation for kinect 2 unity pacage?
i want to learn about coordinate mapping and how to fit my avatar to colormap.
please help me to find right documents thanks.
mahdi
02/12/2016 at 12:59 PM