Representation for Scene and Place Recognition



In this project, we propose a representation, PACT (Principal Component Analysis of Census Transoform histograms), for recognizing the semantic category of natural scenes and indoor environments, e.g. forests, coasts, streets, bedrooms, living rooms, etc.

Recognizing places has been attracting many attention in both robotics and computer vision. In some cases we need to recognize the instance of a place (e.g. 搑oom 115?, and in some other cases the semantic category (e.g. 揟his is an office room.?. Making our robots having the ability to answer all such questions are key to the success of future robotic systems.
We believe that the central problem is a suitable representation ?because places/scenes are so different from objects that we cannot directly borrow representations from object recognition research. We propose PACT, Principal Component Analysis of Census Transoform histograms, as our representation.

Census Transform (CT) summaries local shape and textures by comparing a pixel with its 8 neighbors, such that any 3x3 image region is mapped to one of the 256 discrete cells.

Histogram of CT values is a summarization of local 3x3 shapes. We show that a CT histogram of an image window implicitly encodes global shape, which is essential for recognizing places and scenes.

As shown by this example, we can reconstruct the input image (the left one in each three-image group) requiring that the reconstructed image (the right one in each group). The middle one in each group is the random starting state for the reconstruction algorithm, which try to minimize the difference between CT histogram of the current state and the desired CT histogram.
By further applying PCA (Principal Component Analysis) on the CT histograms, we get the PACT representation.

In summary, PACT has:
The source code for PACT is available.

Please note a typo in the README, thanks to Henry Shu for pointing out this error.
Basically, the following lines (one for Swedish leaf and one for scene) are switched.
========================================================================
4. Running the Sweden Leaf example
* 4.2 change main.cpp -- uncomment Load_UIUC_Scenes(), and comment out SwedenLeafContour()
5. Running the scene recognition example
* 5.2 change main.cpp -- comment out Load_UIUC_Scenes(), and uncomment SwedenLeafContour()
======================================================
If you use the provided source code, please cite the following paper:
Where am I: Place instance and category recognition using spatial PACT
Jianxin Wu and James M. Rehg
CVPR 2008, Anchorage, Alaska, June 24-26, 2008.