Sergii Baidachnyi

Several days ago, I got my first Kinect 2 for Windows devices in order to prepare my presentation for Developer Day event in Ukraine together with Dmytro Mindra. Right now, I already have some experience about Kinect and I am going to share it in my blog.

Before we create some code, we will need to download Kinect for Windows SDK 2.0. This SDK is in Public Preview now but it allows to create Windows Store applications (previous version didn’t allow it), so I was happy to install this version of SDK. I found it here: http://www.microsoft.com/en-us/download/details.aspx?id=43661.

Once SDK was installed I was able to review some examples and tools. I would recommend to use SDK Browser, that can be found in the following directory C:\Program Files\Microsoft SDKs\Kinect\v2.0-PublicPreview1409\Tools\SDKBrowser. This is a very useful tool, which will show you the entire SDK structure like tools, examples and link to docs.

I would recommend to use the Body Basic-XAML example in order to understand that your Kinect works fine or you may use Kinect Studio in order to do some experiments with the sensorJ Because my Kinect was working fine I could start to think about some demos.

Kinect is a very powerful device and you can use it in many ways as a sensor for your home robot or as a part of your security system but the most popular way of Kinect usage is for entertainment. It allows to create many good games and many of such games are already available for Xbox. At same time I don’t have any desire to create a game in C++/DirectX and I was trying to find a solution for Unity 3d because it is one of the best game engines and I like Unity 3d.

Some time ago, I heard information about an add-in for Unity 3d but I had to spend much time in order to find it. I used several search systems, then I called Unity 3d guys and finally I found it in a very unusual place – on the Kinect for Windows SDK 2.0 download page. This link is collapsed by default and you need to expand System Requirements section in order to find it.

Pay attention that this add-in will work with Unity PRO version only, because the free version of Unity doesn’t support add-ins. Probably, there is a way to fix it for Windows Store applications because Non-Pro version allows to use add-ins for this type of applications but I will check it later and for now I use a Pro version.

If you download the add-in successfully, you will be able to find the unitypackage file as well as two examples inside of it. We already have a small game, which allows us to kill “friendly green men” with our battle stellar ship and we decided to implement some Kinect ready activities to make killing process funnier.

Finally, we can implement three steps, which will allow us to use our body during the game in killing process.

On the first step, we found our MonoBehaviour that was responsible for movement of our ship and we added some new code to it:

private KinectSensor _Sensor;

private BodyFrameReader _Reader;

private Body[] _Data = null;

This code should be clear but I will describe it. In order to get access to sensor we needed KinectSensor class, which allows to initialize our Kinect and provides needed sources. In our case, we needed access to BodyFrame source, so we created BodyFrameReader variable that we used in order to read data from the source. Finally, we created Body array in order to store our data from the source.

On the next step, we needed to initialize Kinect and open our reader. That’s why we implemented the following code:

void Start()

{

    _Sensor = KinectSensor.GetDefault();

    if (_Sensor != null)

    {

        _Reader = _Sensor.BodyFrameSource.OpenReader();

        if (!_Sensor.IsOpen)

        {

            _Sensor.Open();

        }

    }

}

We initialized the Kinect inside the very familiar method for Unity 3D developers and opened our reader there.

Finally, we realized a method, which was called from Update method of our Behaviour class. This method collected data from the sensor and calculated new position for the ship as well as fire ability:

horizontal = 0;

vertical = 0;

fireButton = false;

if (_Reader != null) {

       var frame = _Reader.AcquireLatestFrame ();

       if (frame != null) {

              if (_Data == null) {

                     _Data = new Body[_Sensor.BodyFrameSource.BodyCount];

              }

              frame.GetAndRefreshBodyData (_Data);

              int idx = -1;

              for (var i = 0; i < 6; i++) {

                     if (_Data [i].LeanTrackingState !=

                       TrackingState.NotTracked) {

                           idx = i;

                     }

              }

              if (idx > -1) {     

                     fireButton =

                        _Data [idx].Joints [JointType.HandLeft].Position.Y > 0;

                     horizontal = _Data [idx].Lean.X;

                     vertical = _Data [idx].Lean.Y;

              }

       }

       frame.Dispose();

}

This code is more complex and Dima said that I should not show it because it’s not optimal but, probably, you will create the same code for your first Kinect application for Unity 3DJ

So, in this code, we used the Reader to grab current (latest) body frame that was needed in order to found bodies data inside. Kinect 2 supports up to 6 bodies, so _Sensor.BodyFrameSource.BodyCount should return 6. Using the latest frame we filled Body array in order to aggregate data for each body in our frame. At this point we finished work with Kinect sensor and tried to analyze bodies data. But, if you have just one ship (in our case we had two) and you don’t have any specific interface for Kinect initialization, you will not know about location of real body in the array. That’s why we were trying to find a specific index of the body of a pilot. Because we used leans as a main control mechanism for our ship we checked every body and we were trying to find LeanTrackingState status. If we find a tracked body we fixed index of this body and recalculated position of the ship as well as the fire ability.

I think that we spent about 30 minutes for this example and our presentation was very funny. If you know Russian, you can find our video there: http://devday.in.ua. And I just cut a short video from our presentation there are two guys (me and Dima) were doing a strange things on the stageJ

In the next article I am planning to integrate Kinect with simple Windows 8 application and will discuss different sources of data.