Motion Controller Detail

Idea:

Threat the frame of player(s) as a texture and analyze the average direction of the contour edges of the player(s).

Assumptions:

1. Players can be distinguished from background because they are moving. (Even if they looks still, people have subtle movement other than the background.)

2. People can be seen as sticks. (They are tall, long, and most of the edges are vertical if they stand still.)

Extract Moving Things:

That is a simple trick, because the background image never moves (Assumption 1), we can simply use the difference between a frame and its previous frame.

Smooth and Get Gradient Map:

We can use a gradient convolution kernel {-1, 1} to get the result. We can see that the horizontal lines are obvious after convoluted with a Y direction gradient, and the vertical lines are obvious after X direction gradient.

Get the Interesting Points:

We can see from the gradient kernel, even if the background is subtracted, there is still a lot of noise comes from the background, that is due to contrast change from frame-to-frame brightness change.  So we need to set a threshold that means, if the edge is more obvious than something, I should pay attention to it. A simple way to measure how the edge is obvious is add the magnitude of the x-and-y-direction gradient up. (If Gx*Gx + Gy*Gy > threshold, the point in counted as a interesting point.)

Calculate the Point Directions:

Here is an example of how to calculate the local direction of edges:

The upper left matrix is for a simple edge, the upper right one is the smoothed edge with a Gaussian kernel of size 3, while the lower left is the convolution result with a x-direction gradient kernel and the lower right is the the convolution result with a y-direction gradient kernel.

As we known that if the edge is vertical, the x direction convolution response will be high and vice versa, we can get the direction of the edge by calculate the tangent value of x and y direction convolution results.

Say, we want to know direction of the point (3, 3), we can simply use atan2(-16, -16) = 45 degree.

Calculate the Average Direction of the Whole Frame

Knowing the method above, we can simply accumulate all the direction values and calculate the average of them to get the direction of the whole frame.