How to Implement Image Semantic Segmentation Using CRF-RNN

Selected from GitHub

Author: Shuai Zheng et al.

Translated by: Machine Heart

Contributors: Jiang Siyuan

This GitHub project implements image semantic segmentation by combining CNN and CRF-RNN models. Readers can follow this project to implement the process using Keras/Tensorflow.

• Demo Address: http://crfasrnn.torr.vision

• Project Address: https://github.com/sadeepj/crfasrnn_keras

This GitHub project contains methods to implement “CRF-RNN” image semantic segmentation using Keras/TensorFlow. The paper “Conditional Random Fields as Recurrent Neural Networks” was first published at ICCV2015. The demonstration of this project also won the Best Demonstration Award that year. Below, we provide the original Caffe code for this project. The Keras/TensorFlow code provided by this project is basically equivalent to the Caffe version. Now, let’s follow the installation guide to achieve semantic segmentation.

Caffe Version Code: https://github.com/torrvision/crfasrnn

Installation Guide

1.1 Install Dependencies

Install TensorFlow and Keras, and then follow the installation guide for each. If we plan to use pre-trained models, we need to install Keras and HDF5/h5py. After installing these two packages, run the following command line to ensure they are installed correctly:

Install TensorFlow: https://www.tensorflow.org/install/

Install Keras: https://keras.io/#installation

# First, activate the correct Python virtualenv if you used one during Tensorflow/Keras installation
$ source /home/user/tensorflow_virtualenv/bin/activate  
$ python
>>> import tensorflow
>>> import keras

If installed correctly, we should not see any errors when importing the libraries as shown above.

1.2 Build Custom C++ Code for CRF-RNN

Check the code of this project, activate the Tensorflow/Keras virtualenv, and then run the compile.sh script in the cpp directory. That is, run the following command line:

$ git clone https://github.com/sadeepj/crfasrnn_keras.git
$ cd crfasrnn_keras/cpp
$ source /home/user/tensorflow_virtualenv/bin/activate
$ ./compile.sh

If the build is successful, we will see a new file named high_dim_filter.so (the extension may vary depending on our system). If the build fails, please check the commands in the compile.sh file, and we can also refer to the official guide for building custom operations in TensorFlow.

Official Build Guide Address: https://www.tensorflow.org/extend/adding_an_op#build_the_op_library

1.3 Download Pre-trained Model Weights

After downloading the model weights, place them in the srfasrnn_keras directory and name it crfrnn_keras_model.h5.

Pre-trained Weights Download Address: https://goo.gl/ciEYZi

1.4 Run Demo

$ cd crfasrnn_keras
$ python run_demo.py  
# Make sure that the correct virtualenv is already activated

If the above runs smoothly, we can see the semantic segmentation results in the file “labels.png”.

Limitations of the Current Version

1. Currently, some operations in the CRF-RNN layer can only run on the CPU. A GPU version of the training method will be released soon.

2. The crfrnn_keras_model.h5 model is directly converted from the Caffe model, but it is also feasible to fully train a model directly from Keras.

Paper: Conditional Random Fields as Recurrent Neural Networks

Paper Address: http://www.robots.ox.ac.uk/~szheng/papers/CRFasRNN.pdf

Pixel-level labeling tasks such as image semantic segmentation play a key role in image understanding tasks. Recent methods have been trying to apply deep learning techniques used for image recognition to pixel-level labeling tasks. The key issue with this approach is the limited ability of deep learning techniques to depict visual objects. To address this issue, we introduce a novel convolutional neural network, which is composed of Convolutional Neural Networks (CNN) and Conditional Random Field (CRF) based probabilistic graphical models. To achieve this, we formulated mean-field approximate inference for the CRF with Gaussian pairwise potential and embedded it as a Recurrent Neural Network. We call this network CRF-RNN, and then we can embed it as part of the CNN, obtaining a deep network that possesses the properties of both CNN and CRF. Importantly, our system can fully integrate CRF modeling into CNN, allowing the model to train the entire deep network end-to-end using backpropagation and avoiding offline post-processing methods for target description.

We applied the proposed method to solve the image semantic segmentation problem, achieving the best results on the Pascal VOC 2012 challenge semantic segmentation benchmark.

This article is translated by Machine Heart. Please contact this public account for authorization to reprint..

✄————————————————

Join Machine Heart (Full-time Reporter/Intern): [email protected]

Submissions or Seeking Coverage: [email protected]

Advertising & Business Cooperation: [email protected]