Skeleton Tracking with the Kinect

From Learning

Jump to: navigation, search

This tutorial will explain how to track human skeletons using the Kinect. The OpenNI library can identify the position of key joints on the human body such as the hands, elbows, knees, head and so on. These points form a representation we call the 'skeleton'.

Contents

Enabling Skeleton Tracking

Let us start with the code that we had by the end of the tutorial called Drawing Depth with the Kinect: 

import SimpleOpenNI.*;
 
SimpleOpenNI  context;
 
void setup()
{
  // instantiate a new context
  context = new SimpleOpenNI(this);
 
  // enable depth image generation 
  context.enableDepth();
 
  // create a window the size of the depth information
  size(context.depthWidth(), context.depthHeight()); 
}
 
void draw()
{
  // update the camera
  context.update();
 
  // draw depth image
  image(context.depthImage(),0,0); 
}


First we must tell the OpenNI library to determine joint positions by enabling the skeleton tracking functionality . We do this by adding the following to the setup() function, just after the line that calls context.enableDepth();: 

// enable skeleton generation for all joints
 context.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);


We are also going to draw the skeleton with lines, so in anticipation of that lets set the background, stroke colour and stroke weight by adding the following to setup(): 

background(200,0,0);
stroke(0,0,255);
strokeWeight(3);
smooth();


Your full setup() function should now look like this: 

void setup()
{
  // instantiate a new context
  context = new SimpleOpenNI(this);
 
  // enable depthMap generation 
  context.enableDepth();
 
  // enable skeleton generation for all joints
  context.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
 
  background(200,0,0);
  stroke(0,0,255);
  strokeWeight(3);
  smooth();
 
  // create a window the size of the depth information
  size(context.depthWidth(), context.depthHeight()); 
}


Detecting When the Skeleton is Being Tracked

The SimpleOpenNI wrapper for OpenNI has a function which tells us when a skeleton is successfully being tracked. We can call the function isTrackingSkeleton(int userID) which returns true if we are tracking the skeleton of the specified user. We can check if a single skeleton is being tracked by adding the following to our draw() function: 

// for all users from 1 to 10
int i;
for (i=1; i<=10; i++)
{
  // check if the skeleton is being tracked
  if(context.isTrackingSkeleton(i))
  {
 
  }
}

Now any code we place in the if statement will be called whenever a skeleton is being tracked. This is where we want to draw lines between joints.


To define that functionality, let's define a new function, drawSkeleton(): 

// draw the skeleton with the selected joints
void drawSkeleton(int userId)
{
 
}


and place a call to drawSkeleton() for the ith user in the if statement in the draw() function. Your entire draw() function should look like: 

void draw()
{
  // update the camera
  context.update();
 
  // draw depth image
  image(context.depthImage(),0,0); 
 
  // for all users from 1 to 10
  int i;
  for (i=1; i<=10; i++)
  {
    // check if the skeleton is being tracked
    if(context.isTrackingSkeleton(i))
    {
      drawSkeleton(i);  // draw the skeleton
    }
  }
}

Drawing the Skeleton

Now we will use our drawSkeleton() function to draw lines between joints.

Each joint has an identifier (just a reference to a simple integer) and there are 15 joints in all. They are:

SimpleOpenNI.SKEL_HEAD
SimpleOpenNI.SKEL_NECK
SimpleOpenNI.SKEL_LEFT_SHOULDER
SimpleOpenNI.SKEL_LEFT_ELBOW
SimpleOpenNI.SKEL_LEFT_HAND
SimpleOpenNI.SKEL_RIGHT_SHOULDER
SimpleOpenNI.SKEL_RIGHT_ELBOW
SimpleOpenNI.SKEL_RIGHT_HAND
SimpleOpenNI.SKEL_TORSO
SimpleOpenNI.SKEL_LEFT_HIP
SimpleOpenNI.SKEL_LEFT_KNEE
SimpleOpenNI.SKEL_LEFT_FOOT
SimpleOpenNI.SKEL_RIGHT_HIP
SimpleOpenNI.SKEL_RIGHT_KNEE
SimpleOpenNI.SKEL_RIGHT_FOOT


The SimpleOpenNI wrapper has a function for drawing a line between joints called drawLimb(). It takes the userID of the skeleton being tracked, the joint to draw the line from and the joint to the line draw to. For example, the following draws a line from the left elbow to the left hand: 

 context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_ELBOW, SimpleOpenNI.SKEL_LEFT_HAND);


To draw the skeleton in full, add commands to draw limbs to our drawSkeleton() function so that it looks like: 

// draw the skeleton with the selected joints
void drawSkeleton(int userId)
{  
  context.drawLimb(userId, SimpleOpenNI.SKEL_HEAD, SimpleOpenNI.SKEL_NECK);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_NECK, SimpleOpenNI.SKEL_LEFT_SHOULDER);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER, SimpleOpenNI.SKEL_LEFT_ELBOW);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_ELBOW, SimpleOpenNI.SKEL_LEFT_HAND);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_NECK, SimpleOpenNI.SKEL_RIGHT_SHOULDER);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, SimpleOpenNI.SKEL_RIGHT_ELBOW);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_ELBOW, SimpleOpenNI.SKEL_RIGHT_HAND);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER, SimpleOpenNI.SKEL_TORSO);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, SimpleOpenNI.SKEL_TORSO);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_LEFT_HIP);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_HIP, SimpleOpenNI.SKEL_LEFT_KNEE);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_KNEE, SimpleOpenNI.SKEL_LEFT_FOOT);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_RIGHT_HIP);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_HIP, SimpleOpenNI.SKEL_RIGHT_KNEE);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_KNEE, SimpleOpenNI.SKEL_RIGHT_FOOT);  
}


Skeleton Tracking Event-based Methods

There are several event-based functions necessary for skeleton tracking.

Detecting New Users and Losing Users

Firstly, we must define methods which are called when a new person ('user') enters or leaves the field of view: 

// when a person ('user') enters the field of view
void onNewUser(int userId)
{
  println("New User Detected - userId: " + userId);
 
 // start pose detection
  context.startPoseDetection("Psi", userId);
}
 
// when a person ('user') leaves the field of view 
void onLostUser(int userId)
{
  println("User Lost - userId: " + userId);
}

The function onNewUser() is called when a new user is detected in the field of view. When this happens we print "New User Detected" and the user ID to the screen. We then call the startPoseDetection() function to tell the software to begin looking for calibration poses. The function onLostUser() is called when a person leaves the field of view (or is otherwise considered 'lost' by the library). In this case we simply print "User Lost" and the user ID to the screen.


Detecting When A User Starts a Calibration Pose

Next we must define a function called onStartPose() that is called when the library recognises that a user is beginning a calibration pose: 

// when a user begins a pose
void onStartPose(String pose,int userId)
{
  println("Start of Pose Detected  - userId: " + userId + ", pose: " + pose);
 
  // stop pose detection
  context.stopPoseDetection(userId); 
 
  // start attempting to calibrate the skeleton
  context.requestCalibrationSkeleton(userId, true); 
}

In this case we print "Start of Pose Detected" and the user ID to the screen. We then tell the library to stop attempting to recognise a pose. Finally, we request that the library attempts to calibrate the user's skeleton (recognise the joint positions on the user).


Starting and Ending the Calibration Process

Finally, we must define functions that are called when the skeleton calibration process begins and ends: 

// when calibration begins
void onStartCalibration(int userId)
{
  println("Beginning Calibration - userId: " + userId);
}
 
// when calibaration ends - successfully or unsucessfully 
void onEndCalibration(int userId, boolean successfull)
{
  println("Calibration of userId: " + userId + ", successfull: " + successfull);
 
  if (successfull) 
  { 
    println("  User calibrated !!!");
 
    // begin skeleton tracking
    context.startTrackingSkeleton(userId); 
  } 
  else 
  { 
    println("  Failed to calibrate user !!!");
 
    // Start pose detection
    context.startPoseDetection("Psi", userId);
  }
}


The function onStartCalibration() is called when the calibration process starts. In this case we simply print "Beginning Calibration" and the user ID to the screen.

The function onEndCalibration() is called at the end of the calibration process. It is passed a boolean argument ('successful') which indicates whether the calibration was successful or not. In this situation we print to the screen the user ID and whether or not the calibration was successful. Then, if the calibration was successful we begin skeleton tracking. If not, we re-start the pose detection process.

Skeleton Tracking in Practice

Now try running the sketch. Stand in front of the Kinect with your arms out like the pose below:

Pose.png


After a while it should start tracking your skeleton and it should be drawn on the screen, looking something like this:

SkeletonTracked.png


Full Source

The full source code of your sketch should look like this: 

import SimpleOpenNI.*;
 
SimpleOpenNI  context;
 
void setup()
{
  // instantiate a new context
  context = new SimpleOpenNI(this);
 
  // enable depthMap generation 
  context.enableDepth();
 
  // enable skeleton generation for all joints
  context.enableUser(SimpleOpenNI.SKEL_PROFILE_ALL);
 
  background(200,0,0);
  stroke(0,0,255);
  strokeWeight(3);
  smooth();
 
  // create a window the size of the depth information
  size(context.depthWidth(), context.depthHeight()); 
}
 
void draw()
{
  // update the camera
  context.update();
 
  // draw depth image
  image(context.depthImage(),0,0); 
 
  // for all users from 1 to 10
  int i;
  for (i=1; i<=10; i++)
  {
    // check if the skeleton is being tracked
    if(context.isTrackingSkeleton(i))
    {
      drawSkeleton(i);  // draw the skeleton
    }
  }
}
 
// draw the skeleton with the selected joints
void drawSkeleton(int userId)
{  
  context.drawLimb(userId, SimpleOpenNI.SKEL_HEAD, SimpleOpenNI.SKEL_NECK);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_NECK, SimpleOpenNI.SKEL_LEFT_SHOULDER);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER, SimpleOpenNI.SKEL_LEFT_ELBOW);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_ELBOW, SimpleOpenNI.SKEL_LEFT_HAND);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_NECK, SimpleOpenNI.SKEL_RIGHT_SHOULDER);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, SimpleOpenNI.SKEL_RIGHT_ELBOW);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_ELBOW, SimpleOpenNI.SKEL_RIGHT_HAND);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_SHOULDER, SimpleOpenNI.SKEL_TORSO);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_SHOULDER, SimpleOpenNI.SKEL_TORSO);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_LEFT_HIP);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_HIP, SimpleOpenNI.SKEL_LEFT_KNEE);
  context.drawLimb(userId, SimpleOpenNI.SKEL_LEFT_KNEE, SimpleOpenNI.SKEL_LEFT_FOOT);
 
  context.drawLimb(userId, SimpleOpenNI.SKEL_TORSO, SimpleOpenNI.SKEL_RIGHT_HIP);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_HIP, SimpleOpenNI.SKEL_RIGHT_KNEE);
  context.drawLimb(userId, SimpleOpenNI.SKEL_RIGHT_KNEE, SimpleOpenNI.SKEL_RIGHT_FOOT);  
}
 
// Event-based Methods
 
// when a person ('user') enters the field of view
void onNewUser(int userId)
{
  println("New User Detected - userId: " + userId);
 
 // start pose detection
  context.startPoseDetection("Psi",userId);
}
 
// when a person ('user') leaves the field of view 
void onLostUser(int userId)
{
  println("User Lost - userId: " + userId);
}
 
// when a user begins a pose
void onStartPose(String pose,int userId)
{
  println("Start of Pose Detected  - userId: " + userId + ", pose: " + pose);
 
  // stop pose detection
  context.stopPoseDetection(userId); 
 
  // start attempting to calibrate the skeleton
  context.requestCalibrationSkeleton(userId, true); 
}
 
// when calibration begins
void onStartCalibration(int userId)
{
  println("Beginning Calibration - userId: " + userId);
}
 
// when calibaration ends - successfully or unsucessfully 
void onEndCalibration(int userId, boolean successfull)
{
  println("Calibration of userId: " + userId + ", successfull: " + successfull);
 
  if (successfull) 
  { 
    println("  User calibrated !!!");
 
    // begin skeleton tracking
    context.startTrackingSkeleton(userId); 
  } 
  else 
  { 
    println("  Failed to calibrate user !!!");
 
    // Start pose detection
    context.startPoseDetection("Psi",userId);
  }
}
Retrieved from "http://learning.codasign.com/index.php?title=Skeleton_Tracking_with_the_Kinect"
  • This page was last modified on 13 November 2011, at 22:17.
  • This page has been accessed 15,614 times.