Before we start, The libraries that is required for example SimpleOpenNi is incompatible with Processing 3+ versions. Hence, we are using an older version of Processing ie. 2.2.1 for this exercise.
>> You can download the older version here under stable releases: https://processing.org/download/
>> You can work on multiple versions of processing on your desktop, However, do not run different versions at the same time, you may end up with something very whacky! :)
Let’s begin. As we are working with SimpleOpenNi, which is no longer available for the newer Processing versions, we need to download some dependencies in order for this exercise to work successfully.
These should be installed on the class laptops, but just in case the instructions are included here.
Installing Dependencies
Processing
Download Processing >= 2.2.1 for your platform and install it
Go to the menu: Sketch-> Import Library...-> Add Library...
Select and install SimpleOpenNI
Select and install Kinect4WinSDK
Select and install OpenCV for processing
Select and install Open Kinect for processing
or Download as file: Download SimpleOpenNI
Download the first SimpleOpenNi 1.96 zip folder
Copy the extracted folder in your Processing library folder * Windows: C:\Users\'your username'\Documents\Processing\libraries *
Install Kinect SDK Download Kinect SDK from the Developer Page Start the Kinect SDK Installer
If everything worked out, you should see the plugged camera in your Device Manager(under 'Kinect for Windows'). In case you have an error when you startup a processing sketch with SimpleOpenNI, try to install the Runtime Libraries from Microsoft.
Now if you’re set up is installed correctly, let’s start with the code.
The following 4 functions are implemented in the code below. All images at this moment are 640 x 480.
Obtain the RGB image from the Kinect camera.
Obtain the depth image from the Kinect camera.
Align the RGB image with the depth image.
Obtain player and skeleton information.
API description
GetImage() returns a 640 x 480 ARGB PImage.
GetDepth() returns a 640 x 480 ARGB PImage. The image is, however, grey scale only. It resolution is also reduced from the original 13 bits to 8 bits for compatibility with the 256 grey scale image.
GetMask() returns a 640 x 480 ARGB PImage. The image is transparent in the background using the alpha channel. Only those areas with players are opaque with the aligned RGB images of the players.
Skeleton tracking is a bit complicated. The library will expect 3 event handlers in your Processing sketch. Each event handler uses one or two arguments of type SkeletonData (to be explained later). Each SkeletonData represents a human figure that appears, disappears or moves in front of the Kinect camera.
appearEvent – it is triggered whenever a new figure appears in front of the Kinect camera. The SkeletonData keeps the id and position information of the new figure.
disappearEvent – it is triggered whenever a tracked figure disappears from the screen. The SkeletonData keeps the id and position information of the left figure.
moveEvent – it is triggered whenever a tracked figure stays within the screen and may move around. The first SkeletonData keeps the old position information and the second SkeletonData maintains the new position information of the moving figure.
Please note that a new figure may not represent a real new human player. An existing player goes off screen and comes back may be considered as new.
Run this code and get some cool shots:
import kinect4WinSDK.Kinect;
import kinect4WinSDK.SkeletonData;
Kinect kinect;
ArrayList <SkeletonData> bodies;
void setup()
{
size(640, 480);
background(0);
kinect = new Kinect(this);
smooth();
bodies = new ArrayList<SkeletonData>();
}
void draw()
{
background(0);
image(kinect.GetImage(), 320, 0, 320, 240);
image(kinect.GetDepth(), 320, 240, 320, 240);
image(kinect.GetMask(), 0, 240, 320, 240);
for (int i=0; i<bodies.size (); i++)
{
drawSkeleton(bodies.get(i));
drawPosition(bodies.get(i));
}
}
void drawPosition(SkeletonData _s)
{
noStroke();
fill(0, 100, 255);
String s1 = str(_s.dwTrackingID);
text(s1, _s.position.x*width/2, _s.position.y*height/2);
}
void drawSkeleton(SkeletonData _s)
{
// Body
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_HEAD,
Kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER,
Kinect.NUI_SKELETON_POSITION_SHOULDER_LEFT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER,
Kinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER,
Kinect.NUI_SKELETON_POSITION_SPINE);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_LEFT,
Kinect.NUI_SKELETON_POSITION_SPINE);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT,
Kinect.NUI_SKELETON_POSITION_SPINE);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SPINE,
Kinect.NUI_SKELETON_POSITION_HIP_CENTER);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_HIP_CENTER,
Kinect.NUI_SKELETON_POSITION_HIP_LEFT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_HIP_CENTER,
Kinect.NUI_SKELETON_POSITION_HIP_RIGHT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_HIP_LEFT,
Kinect.NUI_SKELETON_POSITION_HIP_RIGHT);
// Left Arm
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_LEFT,
Kinect.NUI_SKELETON_POSITION_ELBOW_LEFT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_ELBOW_LEFT,
Kinect.NUI_SKELETON_POSITION_WRIST_LEFT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_WRIST_LEFT,
Kinect.NUI_SKELETON_POSITION_HAND_LEFT);
// Right Arm
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT,
Kinect.NUI_SKELETON_POSITION_ELBOW_RIGHT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_ELBOW_RIGHT,
Kinect.NUI_SKELETON_POSITION_WRIST_RIGHT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_WRIST_RIGHT,
Kinect.NUI_SKELETON_POSITION_HAND_RIGHT);
// Left Leg
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_HIP_LEFT,
Kinect.NUI_SKELETON_POSITION_KNEE_LEFT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_KNEE_LEFT,
Kinect.NUI_SKELETON_POSITION_ANKLE_LEFT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_ANKLE_LEFT,
Kinect.NUI_SKELETON_POSITION_FOOT_LEFT);
// Right Leg
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_HIP_RIGHT,
Kinect.NUI_SKELETON_POSITION_KNEE_RIGHT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_KNEE_RIGHT,
Kinect.NUI_SKELETON_POSITION_ANKLE_RIGHT);
DrawBone(_s,
Kinect.NUI_SKELETON_POSITION_ANKLE_RIGHT,
Kinect.NUI_SKELETON_POSITION_FOOT_RIGHT);
}
void DrawBone(SkeletonData _s, int _j1, int _j2)
{
noFill();
stroke(255, 255, 0);
if (_s.skeletonPositionTrackingState[_j1] != Kinect.NUI_SKELETON_POSITION_NOT_TRACKED &&
_s.skeletonPositionTrackingState[_j2] != Kinect.NUI_SKELETON_POSITION_NOT_TRACKED) {
line(_s.skeletonPositions[_j1].x*width/2,
_s.skeletonPositions[_j1].y*height/2,
_s.skeletonPositions[_j2].x*width/2,
_s.skeletonPositions[_j2].y*height/2);
}
}
void appearEvent(SkeletonData _s)
{
if (_s.trackingState == Kinect.NUI_SKELETON_NOT_TRACKED)
{
return;
}
synchronized(bodies) {
bodies.add(_s);
}
}
void disappearEvent(SkeletonData _s)
{
synchronized(bodies) {
for (int i=bodies.size ()-1; i>=0; i--)
{
if (_s.dwTrackingID == bodies.get(i).dwTrackingID)
{
bodies.remove(i);
}
}
}
}
void moveEvent(SkeletonData _b, SkeletonData _a)
{
if (_a.trackingState == Kinect.NUI_SKELETON_NOT_TRACKED)
{
return;
}
synchronized(bodies) {
for (int i=bodies.size ()-1; i>=0; i--)
{
if (_b.dwTrackingID == bodies.get(i).dwTrackingID)
{
bodies.get(i).copy(_a);
break;
}
}
}
}
Once you have your skeleton tracking output, let’s convert the frames into a video.
You need to change the ‘Draw’ function in order to save the frame. So let’s do that. Locate the draw function and add this line of code at the end:
saveFrame("Frame-######.png");
Your modified draw function should look like this:
void draw()
{
background(0);
image(kinect.GetImage(), 320, 0, 320, 240);
image(kinect.GetDepth(), 320, 240, 320, 240);
image(kinect.GetMask(), 0, 240, 320, 240);
for (int i=0; i<bodies.size (); i++)
{
drawSkeleton(bodies.get(i));
drawPosition(bodies.get(i));
}
saveFrame("Frame-######.png");
}
Run this code, Go to Sketch Folder and check whether frames are getting saved. If yes, then the above code was successful.
Now, go to Tools>MovieMaker to turn the frames into a Quicktime video.
The frames from your kinect would be saved under a specific sketch folder.
Once you have opened Movie Maker, Click on ‘Choose’ and browse through the folders to add the ‘Sketch Folder’.
Usually, it should be under this path: C:\Users\robot\AppData\Local\Temp
Save the video under any desired folder and Voila! Your movie is ready!
This is a very simple simple particle systems in processing. It's intended for you to expand upon…
Step 1: Open Processing > Paste this code below. Do not run the code yet.
class Particle{
PVector location;
PVector velocity;
PVector acceleration;
float lifespan;
Particle(PVector l){
location = l.get();
velocity = new PVector(random(-1,1),random(-2.0));
acceleration = new PVector(0,0.05);
lifespan = 255;
}
void run(){
update();
display();
}
void update(){
location.add(velocity);
velocity.add(acceleration);
}
void display(){
stroke(0,lifespan);
fill(175,200,lifespan);
ellipse(location.x, location.y,8,8);
}
boolean isDead(){
if(lifespan<0){
return true;
} else {
return false;
}
}
}
Step 2: Now open another Sketch tab in Processing and paste the code below:
import SimpleOpenNI.*;
SimpleOpenNI kinect;
ArrayList<Particle> particles;
int closestValue;
int closestX;
int closestY;
void setup(){
size(640,360);
kinect = new SimpleOpenNI(this);
kinect.enableDepth();
particles = new ArrayList<Particle>();
}
void draw(){
closestValue = 4000;
kinect.update();
int[] depthValues = kinect.depthMap();
for(int y = 0;y<360; y++){
for(int x = 0;x<640; x++){
int i = x + y*640;
int currentDepthValue = depthValues[i];
if(currentDepthValue>0 && currentDepthValue < closestValue){
closestValue = currentDepthValue;
closestX = x;
closestY = y;
}
}
}
image(kinect.depthImage(),0,0);
particles.add(new Particle(new PVector(closestX, closestY)));
for(int i=0; i<particles.size(); i++){
Particle p = particles.get(i);
p.run();
if(p.isDead()){
particles.remove(i);
}
}
//image(kinect.depthImage(),0,0);
// fill(155,150,200);
// ellipse(closestX, closestY,25,25); //You can change the values of the ellipse and check the output. You should also try changing the shape of the particle in various shapes and sizes.
}
Step 3: Once we have both the code in place, let’s run the sketches. Run both the sketches simultaneously and view the output from the Kinect. You should be able to see Depth Image from the Kinect. Move around to create a whacky effect.
Now Let’s move onto Step 4:
Now that you can view your output, let’s move onto the last step. Converting the frames into a video just like we did for the earlier exercise.
You need to change the ‘Draw’ function in order to save the frames. Locate the draw function and append this line of code at the end: saveFrame("Frame-######.png");
Your modified draw function should look like this:
void draw(){
closestValue = 4000;
kinect.update();
int[] depthValues = kinect.depthMap();
for(int y = 0;y<360; y++){
for(int x = 0;x<640; x++){
int i = x + y*640;
int currentDepthValue = depthValues[i];
if(currentDepthValue>0 && currentDepthValue < closestValue){
closestValue = currentDepthValue;
closestX = x;
closestY = y;
}
}
saveFrame("Frame-######.png");
}
Run this code, Go to Sketch Folder and check whether frames are getting saved. If yes, then the above code was successful.
Now, go to Tools>MovieMaker to turn the frames into a Quicktime video.
The frames from your kinect would be saved under a specific sketch folder.
Once you have opened Movie Maker, Click on ‘Choose’ and browse through the folders to add the ‘Sketch Folder’.
Usually, it should be under this path: C:\Users\robot\AppData\Local\Temp
Save the video under any desired folder and Voila! Your movie is ready!
Saving Pointclouds
Last week, we looked at the point clouds produced by the Kinect and how to merge them with the color data of the RGB image, and everyone managed to get a video out of that. One of the big ideas in using those point clouds in animation, or 3D object scanning, or to create a video of the same scene from a new angle, is how to save the point clouds. One of the major file formats associated with saving pointclouds is the PLY file, developed by Stanford computer vision researchers.
A PLY file isn't too difficult to write; it has a standard format that starts with a header to identify the file type and define the sorts of data being stored about each point, and then has a long list of the individual points. (I don't normally recommend Wikipedia as the definitive sources for these things, but it has a nice introductory summary to the PLY file if you'd like to read more.)
The first thing we'll do is create a PLY file as output. Let's start with this chunk of code:
import SimpleOpenNI.*; import processing.opengl.*; SimpleOpenNI kinect; float rotation = 0; boolean writeFile = false; PVector[] depthPoints; PImage rgbImage; void setup() { size(1024,768,OPENGL); kinect = new SimpleOpenNI(this); kinect.enableDepth(); kinect.enableRGB(); kinect.alternativeViewPointDepthToImage(); print("Press 'p' key to preserve the point cloud in a file called 'kinect_output.ply'\n"); } void draw() { background(0); // update the cam kinect.update(); rgbImage = kinect.rgbImage(); translate(width/2, height/2, -250); rotateX(radians(180)); translate(0,0,1000); rotateY(radians(rotation)); depthPoints = kinect.depthMapRealWorld(); for (int i = 0; i < depthPoints.length; i++){ PVector currentPoint = depthPoints[i]; stroke(rgbImage.pixels[i]); point(currentPoint.x, currentPoint.y, currentPoint.z); } if (writeFile){ writeFile(); writeFile = false; } }This should look pretty familiar from last week. You'll notice that we've added some references to writeFile, a boolean variable that writes a PLY file if we press 'p'. So we need to add code to handle the keypress:
void keyPressed() { if (key == 'p'){ writeFile = true; } }That should look familiar from the key-handling you did last week. Next we'll need to write a function to actually write the file:
void writeFile(){ println("writing file"); PrintWriter output = createWriter("kinect_output.ply"); String header = "ply\n" + "format ascii 1.0\n" + "element vertex " + depthPoints.length + "\n" + "property float x\n" + "property float y\n" + "property float z\n" + "property uchar diffuse_red\n" + "property uchar diffuse_green\n" + "property uchar diffuse_blue\n"+ "end_header\n"; output.print(header); for (int i = 0; i < depthPoints.length; i++){ PVector currentPoint = depthPoints[i]; output.print(String.format("%10f", currentPoint.x) + " " + String.format("%10f", currentPoint.y) + " " + String.format("%10f", currentPoint.z) + " " + int(red(rgbImage.pixels[i])) + " " + int(green(rgbImage.pixels[i])) + " " + int(blue(rgbImage.pixels[i])) + "\n"); } output.flush(); output.close(); }So let's take a look at that for a second, because that's basically the whole deal behind storing point clouds. We're creating a file with the PrintWriter, and then inserting the file header that gives meta-information about what's in the file: the "ply" line is there to tell applications it's a ply file, there's a note that it's in a text format, a vertex count that tallies up how many points are getting stored, and then lines to declare that we're planning to store the X, Y, Z and R, G, B values for each point. So any PLY file you look at, you can read what information the person who wrote the file is storing.
Right after that, we've got the "for" loop, which is iterating across all the points in the point cloud to collect that information and store it as a line in the file. (This will look really similar to the "for" loop in your draw function, except instead of drawing every point in the array, we're creating a text representation and writing it to file.)
So give that a shot - run it, then go to the Sketch menu and select "show sketch folder" to find that PLY file. You can open it up in a text editor like gedit. The first thing you're going to notice is that the file is massive. Most of your MP3s are probably smaller than this. It's not compressing any image data, and it's storing twice as much information per pixel than regular 2D images do.
What would be really fun is storing video, like the scenes you recorded last class, as point clouds so that you could play them back and change the viewing angles every time - you could create a half a dozen videos, each from different angles, from the same recorded scene. We're moving into some pretty experimental territory here and dealing with huge amounts of data that might be very slow to move around, so this may or may not result in nice videos at the end.
We're going to start by recording pointclouds for individual frames. Let's stick a variable for a counter up at the top so we can count frames.
int fnum = 0;Now we can use that in the writeFile() function to output lots of frames that are numbered. So in writeFile(), replace the PrintWriter creation line with this:
PrintWriter output = createWriter("frames/frame" + str(fnum) + ".ply");So there you go, instead of writing one file, it's writing sequential ones. We need to make a couple of changes to writeFile gets called on every frame instead of once on a keypress. First we're going to remove the keypress handler by taking out the keyPressed() function. Then in your draw function, remove the "if(writeFile)" loop from around the writeFile() call.
So there ya go. Run that, and that should create a directory in your sketch folder called "frames" and fill it with PLY files for each frame.
Reading the pointclouds in to be displayed is in theory just reversing that, but in practice it’s kind of slow for these files, so we’re going to switch over to a different set of pointclouds - look on your Desktop for a folder called LIDAR.