Joint(Cross) Bilateral Filter

I have been meaning to write about this for a while but due to two conference deadlines I did not get time. Anyway it is one of the most interesting things I read and implemented recently which helped immensely in my project. Joint bilateral filter is also referred to as Cross bilateral filter. Both mean the same. Without any further ado let me motivate its necessity and then provide some explanation with results. I will make the matlab, c++ and java code available on my website very soon.

A basic course in image processing will teach you about low pass filtering. In layman's language smoothing. An image is fundamentally composed of two types of frequency components, low and high. Low frequency components signify smooth and constant regions where as high frequency components signify edges and corners. So, a low pass filter passes low frequency components untouched but smoothes the high frequency components. Typically smoothing is a process of convolving a kernel with the image at each pixel location. It is used to reduce noise. The typical kernel is a uniform or a Gaussian kernel. These kernels work very well in general but have issues near edges/boundaries. They induce artifacts because the resulting value after smoothing at the boundary pixel comes from two different regions. The edges are not preserved. Bilateral filter is a technique that can be used to perform edge preserving smoothing. There are variants to this namely anisotropic diffusion. I will not go into details of that in this post. So, in short bilateral filter modifies the kernel based on the local content so that edges are preserved.

Taking it a step further imagine you have two sources and you want to smooth one source based on the similarity of the other source. A typical example is if you have color image and depth image and you want to smooth the color image such that color does not bleed across depth boundaries. That is when you use the joint/cross bilateral filter. Here, the kernel is a combination of weights based on the color similarity and depth similarity. Thus the filter will only smooth values with similar color and depth and keep the rest untouched. This is a simple but elegant solution that has tremendous applications. The application I am interested in and where I used this in my research is to smooth the color image as a preprocessing step to perform over-segmentation.

Below I share three images, the input image, result obtained by smoothing the color image using a Gaussian filter alone and result obtained by using the joint bilateral filter. You can clearly see that the edges are preserved but the rest of the content is smoothed out. Keep in mind that along with the color image I have the corresponding(registered) depth data. Without an additional source of information you can only apply bilateral filter, not cross bilateral filter.

I will discuss about the two papers that are relevant to this in my JiffyPaper blog.