Studio 5

Saving Pointclouds

Last week, we looked at the point clouds produced by the Kinect and how to merge them with the color data of the RGB image, and everyone managed to get a video out of that. One of the big ideas in using those point clouds in animation, or 3D object scanning, or to create a video of the same scene from a new angle, is how to save the point clouds. One of the major file formats associated with saving pointclouds is the PLY file, developed by Stanford computer vision researchers.

A PLY file isn't too difficult to write; it has a standard format that starts with a header to identify the file type and define the sorts of data being stored about each point, and then has a long list of the individual points. (I don't normally recommend Wikipedia as the definitive sources for these things, but it has a nice introductory summary to the PLY file if you'd like to read more.)

The first thing we'll do is create a PLY file as output. Let's start with this chunk of code:

import SimpleOpenNI.*;
import processing.opengl.*;

SimpleOpenNI  kinect;
float rotation = 0;
boolean writeFile = false;
PVector[] depthPoints;
PImage rgbImage;

void setup()
{
  size(1024,768,OPENGL);
  kinect = new SimpleOpenNI(this);
   
  kinect.enableDepth();
  kinect.enableRGB();
  kinect.alternativeViewPointDepthToImage();
 
  print("Press 'p' key to preserve the point cloud in a file called 'kinect_output.ply'\n");

}

void draw()
{
  background(0);
  // update the cam
  kinect.update();
 
  rgbImage = kinect.rgbImage();
 
  translate(width/2, height/2, -250);
  rotateX(radians(180));
  translate(0,0,1000);
  rotateY(radians(rotation));
 
  depthPoints = kinect.depthMapRealWorld();
  for (int i = 0; i < depthPoints.length; i++){
 PVector currentPoint = depthPoints[i];
 stroke(rgbImage.pixels[i]);
 point(currentPoint.x, currentPoint.y, currentPoint.z);
  }
 
  if (writeFile){
 writeFile();
 writeFile = false;
  }
}

This should look pretty familiar from last week. You'll notice that we've added some references to writeFile, a boolean variable that writes a PLY file if we press 'p'. So we need to add code to handle the keypress:

void keyPressed() {
  if (key == 'p'){
writeFile = true;
  }
}

That should look familiar from the key-handling you did last week. Next we'll need to write a function to actually write the file:

void writeFile(){
  println("writing file");
  PrintWriter output = createWriter("kinect_output.ply");
  String header = "ply\n" +
"format ascii 1.0\n" +
"element vertex " + depthPoints.length + "\n" +
"property float x\n" +
"property float y\n" +
"property float z\n" +
"property uchar diffuse_red\n" +
"property uchar diffuse_green\n" +
"property uchar diffuse_blue\n"+
"end_header\n";
  output.print(header);
 
  for (int i = 0; i < depthPoints.length; i++){
PVector currentPoint = depthPoints[i];
output.print(String.format("%10f", currentPoint.x) +
    " " + String.format("%10f", currentPoint.y) +
    " " + String.format("%10f", currentPoint.z) +
    " " + int(red(rgbImage.pixels[i])) +
    " " + int(green(rgbImage.pixels[i])) +
    " " + int(blue(rgbImage.pixels[i])) +
    "\n");
  }
  output.flush();
  output.close();
}

So let's take a look at that for a second, because that's basically the whole deal behind storing point clouds. We're creating a file with the PrintWriter, and then inserting the file header that gives meta-information about what's in the file: the "ply" line is there to tell applications it's a ply file, there's a note that it's in a text format, a vertex count that tallies up how many points are getting stored, and then lines to declare that we're planning to store the X, Y, Z and R, G, B values for each point. So any PLY file you look at, you can read what information the person who wrote the file is storing.

Right after that, we've got the "for" loop, which is iterating across all the points in the point cloud to collect that information and store it as a line in the file. (This will look really similar to the "for" loop in your draw function, except instead of drawing every point in the array, we're creating a text representation and writing it to file.)

So give that a shot - run it, then go to the Sketch menu and select "show sketch folder" to find that PLY file. You can open it up in a text editor like gedit. The first thing you're going to notice is that the file is massive. Most of your MP3s are probably smaller than this. It's not compressing any image data, and it's storing twice as much information per pixel than regular 2D images do.

What would be really fun is storing video, like the scenes you recorded last class, as point clouds so that you could play them back and change the viewing angles every time - you could create a half a dozen videos, each from different angles, from the same recorded scene. We're moving into some pretty experimental territory here and dealing with huge amounts of data that might be very slow to move around, so this may or may not result in nice videos at the end.

We're going to start by recording pointclouds for individual frames. Let's stick a variable for a counter up at the top so we can count frames.

int fnum = 0;

Now we can use that in the writeFile() function to output lots of frames that are numbered. So in writeFile(), replace the PrintWriter creation line with this:

  PrintWriter output = createWriter("frames/frame" + str(fnum) + ".ply");

So there you go, instead of writing one file, it's writing sequential ones. We need to make a couple of changes to writeFile gets called on every frame instead of once on a keypress. First we're going to remove the keypress handler by taking out the keyPressed() function. Then in your draw function, remove the "if(writeFile)" loop from around the writeFile() call.

So there ya go. Run that, and that should create a directory in your sketch folder called "frames" and fill it with PLY files for each frame.

Reading the pointclouds in to be displayed is in theory just reversing that, but in practice it’s kind of slow for these files, so we’re going to switch over to a different set of pointclouds - look on your Desktop for a folder called LIDAR.