The Image Jacobian

Let's consider again the situation where we have a camera looking at an object.  Once again it's a ball. We attach a coordinate C to the camera and we have the coordinate of the ball expressed with respect to the camera coordinate frame. 

We're going to introduce a new symbol, which is F hat, and it is the focal length of a lens, which is in units of meters, divided by the dimensions of the pixel given by the Greek letter Rho. Now Rho has units of meters per pixel, so the ratio has got units of pixels. F hat then is another way of representing the focal length of a lens instead of expressing it in units of meters, we express it in units of pixels and this can be very useful to us shortly. 

We can project the position of the ball into the image plane, and we represent that by coordinate U, V, there is nothing new in any of this.  I'm going to simplify the notation and refer to the position of the ball in 3D space as capital X, capital Y, capital Z and now we can write the coordinates of the projection of the ball in the image plane in this fashion, using the symbols X, Y and Z and F hat, which remember is the focal length of the lens expressed in units of pixels. 

Now let's consider that the camera is moving.  We're going to strap it to a rabbit.  The camera now is able to move with a velocity, which we can express by a three vector Vx, Vy and Vz and it's also able to rotate. And we can express its rotational velocity, as it's angular velocity with components Omega X, Omega Y and Omega Z. 

Well what happens now?  Well we now need to think about the projection of the ball in the image plane having not just a location U, V but it's also got a velocity in the image plane, which we'll represent by U dot and V dot. 

Now there's a very well known expression that relates the velocity of the camera to the pixel velocity in the image plane and these two quantities are related by a Jacobian matrix and we have come across Jacobian Matrices before in the robotics MOOC. 

The Jacobian matrix here is a two by six matrix, which relates the camera velocity and this is the cameras spatial velocity which has got a translational component and a rotational component.  Multiplied by this two by six matrix, to give us the velocity in the image plane.  The velocity at which that point in the image appears to move. 

Here's our MATLAB workspace and we have already created a camera object.  I'm going to compute the image Jacobian into the workspace variable J and to do that I am going to use the visjac_p method, that's short for visual Jacobian for a point feature, it's a method of the camera object and I'm going to pass in the coordinate of the image plane point that I'm interested in and that's at 600, 600, it's got to be column vector, so I put an apostrophe there and it corresponds to a world point that's five meters away from the camera. 

So here's the visual Jacobian; it is a 2 by 6 matrix.  If I multiply the image Jacobian by a spatial velocity vector, let's consider a unit velocity in the X direction, no velocity in Y or Z and no rotation and it's a column vector, then the image plane point is going to move at 160 pixels per second in the negative U direction and it's not going to move at all in the V direction. 

Let's try a different spatial velocity.  Let's look at what happens if it moves in the world Y direction, we see that it moves at 160 pixels per second in the negative V direction on the image plane. 

Let's just look at one more example and we will move the camera at unit velocity in the Z direction; that's along the optical axis and we see that now the image plane point is moving at 17.6 pixels per second in the U direction and 17.6 pixels per second in the V direction.