mCENTRIST: A principled way to incorporate color into CENTRIST

      


Color contains richer information than gray-scale. In this project, we seek to incorporate the color information in a principled way into the CENTRIST visual descriptor. The result is mCENTRIST--a multi-channel version of CENTRIST. The following considerations are taken into account when designing mCENTRIST:
  1. We are working with multi-channel images, rather than color (RGB) images. Channels like NIR (near-infrared) can be very useful in image understanding. In the above, the two images (with green and red boxes inside them, respectively) are the R and the NIR channel of a 4-channel (RGB+NIR) image.
  2. We seek to incorporate the information from multiple channels simultaneously. Currently, the popular way is to extract feature vectors from each channel independently, and then concatenate them together into a longer representation. If we consider a two channel image as a two dimensional random variable (joint random variable), what these methods do are compute the marginal distributions and combine them. We argue that in many cases, like in the above example image, knowing the joint distribution provides better description of the scene. That is, we want to simultaneously extract information from both channels. For example, the tree crown is described by both curved edge structure in R and suppressed edges (at corresponding pixel locations) in NIR.
  3. We want a joint but efficient description--efficient in terms of both time and storage costs. Modelling a joint distribution is more costly than finding the marginals. We made two types of design tradeoffs (i.e., propose two techiniques) to achieve this goal.
  4. As a "free" additional channel, we propose that the Sobel image of the gray-scale channel can almost always improve scene recognition in the mCENTRIST framework.

The Matlab source code for mCENTRIST is available (also including instructions to reproduce results in our paper). For problem with the source code, please contact Dr. Yang Xiao at hustcowboy@gmail.com. For other researchers' convenience, we also included a copy of the LIBSVM software, whose license information can be found inside its folder.

Matlab source code is here.  Please read the README.txt file first before you try out the code.
If you use the provided source code, please cite the following paper:

mCENTRIST: A Multi-channel Feature Generation Mechanism for Scene Categorization
Yang Xiao, Jianxin Wu and Junsong Yuan
IEEE Transactions on Image Processing, 23(2), 2014: pp. 823-836.

mCENTRIST is based on the CENTRIST visual descriptor, which is described in the following paper:

CENTRIST: A Visual Descriptor for Scene Categorization
Jianxin Wu and James M. Rehg
IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8), 2011: pp. 1489-1501.