Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, Kevin Murphy

Abstract: We propose a method that can generate an unambiguous description (known as a referring expression) of a specific object or region in an image, and which can also comprehend or interpret such an expression to infer which object is being described. We show that our method outperforms previous methods that generate descriptions of objects without taking into account other potentially ambiguous objects in the scene. Our model is inspired by recent successes of deep learning methods for image captioning, but while image captioning is difficult to evaluate, our task allows for easy objective evaluation. We also present a new large-scale dataset for referring expressions, based on MS-COCO. We have released the dataset and a toolbox for visualization and evaluation, see https://github.com/mjhucla/Google_Refexp_toolbox.

@InProceedings{Mao_2016_CVPR,author = {Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan L. and Murphy, Kevin},title = {Generation and Comprehension of Unambiguous Object Descriptions},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Hyeonwoo Noh, Paul Hongsuck Seo, Bohyung Han

Abstract: We tackle image question answering (ImageQA) problem by learning a convolutional neural network (CNN) with a dynamic parameter layer whose weights are determined adaptively based on questions. For the adaptive parameter prediction, we employ a separate parameter prediction network, which consists of gated recurrent unit (GRU) taking a question as its input and a fully-connected layer generating a set of candidate weights as its output. However, it is challenging to construct a parameter prediction network for a large number of parameters in the fully-connected dynamic parameter layer of the CNN. We reduce the complexity of this problem by incorporating a hashing technique, where the candidate weights given by the parameter prediction network are selected using a predefined hash function to determine individual weights in the dynamic parameter layer. The proposed network---joint network with the CNN for ImageQA and the parameter prediction network---is trained end-to-end through back-propagation, where its weights are initialized using a pre-trained CNN and GRU. The proposed algorithm illustrates the state-of-the-art performance on all available public ImageQA benchmarks.

@InProceedings{Noh_2016_CVPR,author = {Noh, Hyeonwoo and Seo, Paul Hongsuck and Han, Bohyung},title = {Image Question Answering Using Convolutional Neural Network With Dynamic Parameter Prediction},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, Trevor Darrell

Abstract: While recent deep neural network models have achieved promising results on the image captioning task, they rely largely on the availability of corpora with paired image and sentence captions to describe objects in context. In this work, we propose the Deep Compositional Captioner (DCC) to address the task of generating descriptions of novel objects which are not present in paired image-sentence datasets.  Our method achieves this by leveraging large object recognition datasets and external text corpora and by transferring knowledge between semantically similar concepts.  Current deep caption models can only describe objects contained in paired image-sentence corpora, despite the fact that they are pre-trained with large object recognition datasets, namely ImageNet.  In contrast, our model can compose sentences that describe novel objects and their interactions with other objects. We demonstrate our model's ability to describe novel concepts by empirically evaluating its performance on MSCOCO and show qualitative results on ImageNet images of objects for which no paired image-caption data exist. Further, we extend our approach to generate descriptions of objects in video clips. Our results show that DCC has distinct advantages over existing image and video captioning approaches for generating descriptions of new objects in context.

@InProceedings{Hendricks_2016_CVPR,author = {Hendricks, Lisa Anne and Venugopalan, Subhashini and Rohrbach, Marcus and Mooney, Raymond and Saenko, Kate and Darrell, Trevor},title = {Deep Compositional Captioning: Describing Novel Object Categories Without Paired Training Data},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Smola

Abstract: This paper presents stacked attention networks (SANs)that learn to answer natural language questions from images. SANs use semantic representation of a question as query to search for the regions in an image that are related to the answer. We argue that image question answering (QA) often requires multiple steps of reasoning. Thus, we develop a multiple-layer SAN in which we query an image multiple times to infer the answer progressively. Experiments conducted on four image QA data sets demonstrate that the proposed SANs significantly outperform previous state-of-the-art approaches. The visualization of the attention layers illustrates the progress that the SAN locates the relevant visual clues that lead to the answer of the question layer-by-layer.

@InProceedings{Yang_2016_CVPR,author = {Yang, Zichao and He, Xiaodong and Gao, Jianfeng and Deng, Li and Smola, Alex},title = {Stacked Attention Networks for Image Question Answering},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Neural Module Networks

Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein

Abstract: Visual question answering is fundamentally compositional in nature---a question like "where is the dog?" shares substructure with questions like "what color is the dog?" and "where is the cat?" This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions.  We describe a procedure for constructing and learning _neural module networks_, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.

@InProceedings{Andreas_2016_CVPR,author = {Andreas, Jacob and Rohrbach, Marcus and Darrell, Trevor and Klein, Dan},title = {Neural Module Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Scott Reed, Zeynep Akata, Honglak Lee, Bernt Schiele

Abstract: State-of-the-art methods for zero-shot visual recognition formulate learning as a joint embedding problem of images and side information. In these formulations the current best complement to visual features are attributes: manually-encoded vectors describing shared characteristics among categories. Despite good performance, attributes have limitations: (1) finer-grained recognition requires commensurately more attributes, and (2) attributes do not provide a natural language interface. We propose to overcome these limitations by training neural language models from scratch; i.e. without pre-training and only consuming words and characters. Our proposed models train end-to-end to align with the fine-grained and category-specific content of images. Natural language provides a flexible and compact way of encoding only the salient visual aspects for distinguishing categories. By training on raw text, our model can do inference on raw text as well, providing humans a familiar mode both for annotation and retrieval. Our model achieves strong performance on zero-shot text-based image retrieval and significantly outperforms the attribute-based state-of-the-art for zero-shot classification on the Caltech-UCSD Birds 200-2011 dataset.

@InProceedings{Reed_2016_CVPR,author = {Reed, Scott and Akata, Zeynep and Lee, Honglak and Schiele, Bernt},title = {Learning Deep Representations of Fine-Grained Visual Descriptions},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Zeynep Akata, Mateusz Malinowski, Mario Fritz, Bernt Schiele

Abstract: Scaling up visual category recognition to large numbers of classes remains challenging. A promising research direction is zero-shot learning, which does not require any training data to recognize new classes, but rather relies on some form of auxiliary information describing the new classes. Ultimately, this may allow to use textbook knowledge that humans employ to learn about new classes by transferring knowledge from classes they know well. The most successful zero-shot learning approaches currently require a particular type of auxiliary information -- namely attribute annotations performed by humans -- that is not readily available for most classes. Our goal is to circumvent this bottleneck by substituting such annotations by extracting multiple pieces of information from multiple unstructured text sources readily available on the web. To compensate for the weaker form of auxiliary information, we incorporate stronger supervision in the form of semantic part annotations on the classes from which we transfer knowledge. We achieve our goal by a joint embedding framework that maps multiple text parts as well as multiple semantic parts into a common space. Our results consistently and significantly improve on the state-of-the-art in zero-short recognition and retrieval.

@InProceedings{Akata_2016_CVPR,author = {Akata, Zeynep and Malinowski, Mateusz and Fritz, Mario and Schiele, Bernt},title = {Multi-Cue Zero-Shot Learning With Strong Supervision},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, Bernt Schiele

Abstract: We present a novel latent embedding model for learning a compatibility function between image and class embeddings, in the context of zero-shot classification. The proposed method augments the state-of-the-art bilinear compatibility model by incorporating latent variables. Instead of learning a single bilinear map, it learns a collection of maps with the selection, of which map to use, being a latent variable for the current image-class pair. We train the model with a ranking based objective function which penalizes incorrect rankings of the true class for a given image. We empirically demonstrate that our model improves the state-of-the-art for various class embeddings consistently on three challenging publicly available datasets for the zero-shot setting. Moreover, our method leads to visually highly interpretable results with clear clusters of different fine-grained object properties that correspond to different latent variable maps.

@InProceedings{Xian_2016_CVPR,author = {Xian, Yongqin and Akata, Zeynep and Sharma, Gaurav and Nguyen, Quynh and Hein, Matthias and Schiele, Bernt},title = {Latent Embeddings for Zero-Shot Classification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Roland Kwitt, Sebastian Hegenbart, Marc Niethammer

Abstract: The appearance of (outdoor) scenes changes considerably with the strength of certain transient attributes, such as "rainy", "dark" or "sunny". Obviously, this also affects the representation of an image in feature space, e.g., as activations at a certain CNN layer, and consequently impacts scene recognition performance. In this work, we investigate the variability in these transient attributes as a rich source of information for studying how image representations change as a function of attribute strength. In particular, we leverage a recently introduced dataset with fine-grain annotations to estimate feature trajectories for a collection of transient attributes and then show how these trajectories can be transferred to new image representations. This enables us to synthesize new data along the transferred trajectories with respect to the dimensions of the space spanned by the transient attributes. Applicability of this concept is demonstrated on the problem of one-shot recognition of scene locations. We show that data synthesized via feature trajectory transfer considerably boosts recognition performance, (1) with respect to baselines and (2) in combination with state-of-the-art approaches in one-shot learning.

@InProceedings{Kwitt_2016_CVPR,author = {Kwitt, Roland and Hegenbart, Sebastian and Niethammer, Marc},title = {One-Shot Learning of Scene Locations via Feature Trajectory Transfer},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: Attributes possess appealing properties and benefit many computer vision problems, such as object recognition, learning with humans in the loop, and  image retrieval. Whereas the existing work mainly pursues utilizing attributes for various computer vision problems, we contend that the most basic problem---how to accurately and robustly detect attributes from images---has been left under explored. Especially, the existing work rarely explicitly tackles the need that attribute detectors should generalize well across different categories, including those previously unseen. Noting that this is analogous to the objective of multi-source domain generalization, if we treat each category as a domain, we provide a novel perspective to attribute detection and propose to gear the techniques in multi-source domain generalization  for the purpose of learning cross-category generalizable attribute detectors. We validate our understanding and approach with extensive experiments on four challenging datasets and three different problems.

@InProceedings{Gan_2016_CVPR,author = {Gan, Chuang and Yang, Tianbao and Gong, Boqing},title = {Learning Attributes Equals Multi-Source Domain Generalization},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Kwang Moo Yi, Yannick Verdie, Pascal Fua, Vincent Lepetit

Abstract: We show how to train a Convolutional Neural Network to assign a canonical orientation to feature points given an image patch centered on the feature point.  Our method improves feature point matching upon the state-of-the art and can be used in conjunction with any existing rotation sensitive descriptors.  To avoid the tedious and almost impossible task of finding a target orientation to learn, we propose to use Siamese networks which implicitly find the optimal orientations during training. We also propose a new type of activation function for Neural Networks that generalizes the popular ReLU, maxout, and PReLU activation functions.  This novel activation performs better for our task.  We validate the effectiveness of our method extensively with four existing datasets, including two non-planar datasets, as well as our own dataset.  We show that we outperform the state-of-the-art without the need of retraining for each dataset.

@InProceedings{Yi_2016_CVPR,author = {Yi, Kwang Moo and Verdie, Yannick and Fua, Pascal and Lepetit, Vincent},title = {Learning to Assign Orientations to Feature Points},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Carl Vondrick, Hamed Pirsiavash, Antonio Torralba

Abstract: Anticipating actions and objects before they start or appear is a difficult problem in computer vision with several real-world applications. This task is challenging partly because it requires leveraging extensive knowledge of the world that is difficult to write down. We believe that a promising resource for efficiently learning this knowledge is through readily available unlabeled video. We present a framework that capitalizes on temporal structure in unlabeled video to learn to anticipate human actions and objects. The key idea behind our approach is that we can train deep networks to predict the visual representation of images in the future. Visual representations are a promising prediction target because they encode images at a higher semantic level than pixels yet are automatic to compute. We then apply recognition algorithms on our predicted representation to anticipate objects and actions. We experimentally validate this idea on two datasets, anticipating actions one second in the future and objects five seconds in the future.

@InProceedings{Vondrick_2016_CVPR,author = {Vondrick, Carl and Pirsiavash, Hamed and Torralba, Antonio},title = {Anticipating Visual Representations From Unlabeled Video},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, Alexei A. Efros

Abstract: Discriminative deep learning approaches have shown impressive results for problems where human-labeled ground truth is plentiful, but what about tasks where labels are difficult or impossible to obtain? This paper tackles one such problem: establishing dense visual correspondence across different object instances. For this task, although we do not know what the ground-truth is, we know it should be consistent across instances of that category. We exploit this consistency as a supervisory signal to train a convolutional neural network to predict cross-instance correspondences between pairs of images depicting objects of the same category. For each pair of training images we find an appropriate 3D CAD model and render two synthetic views to link in with the pair, establishing a correspondence flow 4-cycle. We use ground-truth synthetic-to-synthetic correspondences, provided by the rendering engine, to train a ConvNet to predict synthetic-to-real, real-to-real and real-to-synthetic correspondences that are cycle-consistent with the ground-truth. At test time, no CAD models are required. We demonstrate that our end-to-end trained ConvNet supervised by cycle-consistency outperforms state-of-the-art pairwise matching methods in correspondence-related tasks.

@InProceedings{Zhou_2016_CVPR,author = {Zhou, Tinghui and Krahenbuhl, Philipp and Aubry, Mathieu and Huang, Qixing and Efros, Alexei A.},title = {Learning Dense Correspondence via 3D-Guided Cycle Consistency},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


The Global Patch Collider

Shenlong Wang, Sean Ryan Fanello, Christoph Rhemann, Shahram Izadi, Pushmeet Kohli

Abstract: This paper proposes a novel extremely efficient, fully-parallelizable, task-specific algorithm for the computation of global point-wise correspondences in images and videos. Our algorithm, the Global Patch Collider, is based on detecting unique collisions between image points using a collection of learned tree structures that act as conditional hash functions. In contrast to conventional approaches that rely on pairwise distance computation, our algorithm isolates distinctive pixel pairs that hit the same leaf during traversal through multiple learned tree structures. The split functions stored at the intermediate nodes of the trees are trained to ensure that only visually similar patches or their geometric or photometric transformed versions fall into the same leaf node. The matching process involves passing all pixel positions in the images under analysis through the tree structures. We then compute matches by isolating points that uniquely collide with each other  ie. fell in the same empty leaf in multiple trees. Our algorithm is linear in the number of pixels but can be made constant time on a parallel computation architecture as the tree traversal for individual image points is decoupled. We demonstrate the efficacy of our method by using it to perform optical flow matching and stereo matching on some challenging benchmarks. Experimental results show that not only is our method extremely computationally efficient, but it is also able to match or outperform state of the art methods that are much more complex.

@InProceedings{Wang_2016_CVPR,author = {Wang, Shenlong and Fanello, Sean Ryan and Rhemann, Christoph and Izadi, Shahram and Kohli, Pushmeet},title = {The Global Patch Collider},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Seyed Hamid Rezatofighi, Anton Milan, Zhen Zhang, Qinfeng Shi, Anthony Dick, Ian Reid

Abstract: Matching between two sets of objects is typically approached by finding the object pairs that collectively maximize the joint matching score. In this paper, we argue that this single solution does not necessarily lead to the optimal matching accuracy and that general one-to-one assignment problems can be improved by considering multiple hypotheses before computing the final similarity measure. To that end, we propose to utilize the marginal distributions for each entity. Previously, this idea has been neglected mainly because exact marginalization is intractable due to a combinatorial number of all possible matching permutations. Here, we propose a generic approach to efficiently approximate the marginal distributions by exploiting the m-best solutions of the original problem. This approach not only improves the matching solution, but also provides more accurate ranking of the results, because of the extra information included in the marginal distribution. We validate our claim on two distinct objectives: (i) person re-identification and temporal matching modelled as an integer linear program, and (ii) feature point matching using a quadratic cost function. Our experiments confirm that marginalization indeed leads to superior performance compared to the single (nearly) optimal solution, yielding state-of-the-art results in both applications on standard benchmarks.

@InProceedings{Rezatofighi_2016_CVPR,author = {Rezatofighi, Seyed Hamid and Milan, Anton and Zhang, Zhen and Shi, Qinfeng and Dick, Anthony and Reid, Ian},title = {Joint Probabilistic Matching Using m-Best Solutions},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Chen Liu, Pushmeet Kohli, Yasutaka Furukawa

Abstract: This paper addresses the challenging problem of perceiving the hidden or occluded geometry of the scene depicted in any given RGBD image. Unlike other image labeling problems such as image segmentation where each pixel needs to be assigned a single label, layered decomposition requires us to assign multiple labels to pixels. We propose a novel "Occlusion-CRF" model that allows for the integration of sophisticated priors to regularize the solution space and enables the automatic inference of the layer decomposition. We use a generalization of the Fusion Move algorithm to perform Maximum a Posterior (MAP) inference on the model that can handle the large label sets needed to represent multiple surface assignments to each pixel. We have evaluated the proposed model and the inference algorithm on many RGBD images of cluttered indoor scenes. Our experiments show that not only is our model able to explain occlusions but it also enables automatic inpainting of occluded/invisible surfaces.

@InProceedings{Liu_2016_CVPR,author = {Liu, Chen and Kohli, Pushmeet and Furukawa, Yasutaka},title = {Layered Scene Decomposition via the Occlusion-CRF},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Michael Maire, Takuya Narihira, Stella X. Yu

Abstract: Spectral embedding provides a framework for solving perceptual organization problems, including image segmentation and figure/ground organization.  From an affinity matrix describing pairwise relationships between pixels, it clusters pixels into regions, and, using a complex-valued extension, orders pixels according to layer.  We train a convolutional neural network (CNN) to directly predict the pairwise relationships that define this affinity matrix.  Spectral embedding then resolves these predictions into a globally-consistent segmentation and figure/ground organization of the scene.  Experiments demonstrate significant benefit to this direct coupling compared to prior works which use explicit intermediate stages, such as edge detection, on the pathway from image to affinities.  Our results suggest spectral embedding as a powerful alternative to the conditional random field (CRF)-based globalization schemes typically coupled to deep neural networks.

@InProceedings{Maire_2016_CVPR,author = {Maire, Michael and Narihira, Takuya and Yu, Stella X.},title = {Affinity CNN: Learning Pixel-Centric Pairwise Relations for Figure/Ground Embedding},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Jie Feng, Brian Price, Scott Cohen, Shih-Fu Chang

Abstract: Interactive image segmentation is an important problem in computer vision with many applications including image editing, object recognition and image retrieval. Most existing interactive segmentation methods only operate on color images. Until recently, very few works have been proposed to leverage depth information from low-cost sensors to improve interactive segmentation. While these methods achieve better results than color-based methods, they are still limited in either using depth as an additional color channel or simply combining depth with color in a linear way. We propose a novel interactive segmentation algorithm which can incorporate multiple feature cues like color, depth, and normals in an unified graph cut framework to leverage these cues more effectively. A key contribution of our method is that it automatically selects a single cue to be used at each pixel, based on the intuition that only one cue is necessary to determine the segmentation label locally.  This is achieved by optimizing over both segmentation labels and cue labels, using terms designed to decide where both the segmentation and label cues should change. Our algorithm thus produces not only the segmentation mask but also a cue label map that indicates where each cue contributes to the final result. Extensive experiments on five large scale RGBD datasets show that our proposed algorithm performs significantly better than both other color-based and RGBD based algorithms in reducing the amount of user inputs as well as increasing segmentation accuracy.

@InProceedings{Feng_2016_CVPR,author = {Feng, Jie and Price, Brian and Cohen, Scott and Chang, Shih-Fu},title = {Interactive Segmentation on RGBD Images via Cue Selection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Hailin Shi, Stan Z. Li

Abstract: Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in CV community. However, most algorithms are designed for faces in small to medium poses (below 45 degree), lacking the ability to align faces in large-pose up to 90 degree. The challenges are three-fold: Firstly, the commonly used landmark-based face model assumes that all the landmarks are visible and is therefore not suitable for profile views. Secondly, the face appearance varies more dramatically in large poses, ranging from frontal view to profile view. Thirdly, labelling landmarks in large poses is an extremely challenging work since the invisible landmarks have to be guessed. In this paper, we propose a solution to the three problems in an new alignment framework, called 3D Dense Face Alignment (3DDFA), in which a dense 3D face model is fitted to the image via convolutional neutral network (CNN). We also propose a method to synthesize large-scale training samples in profile views to solve the third problem of data labelling. Experiments on the challenging AFLW database show that our approach achieves significant improvements over state-of-the-art methods.

@InProceedings{Zhu_2016_CVPR,author = {Zhu, Xiangyu and Lei, Zhen and Liu, Xiaoming and Shi, Hailin and Li, Stan Z.},title = {Face Alignment Across Large Poses: A 3D Solution},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Weakly Supervised Object Boundaries

Anna Khoreva, Rodrigo Benenson, Mohamed Omran, Matthias Hein, Bernt Schiele

Abstract: State-of-the-art learning based boundary detection methods require extensive training data. Since labelling object boundaries is one of the most expensive types of annotations, there is a need to relax the requirement to carefully annotate images to make both the training more affordable and to extend the amount of training data. In this paper we propose a technique to generate weakly supervised annotations and show that bounding box annotations alone suffice to reach high-quality object boundaries without using any object-specific boundary annotations. With the proposed weak supervision techniques we achieve the top performance on the object boundary detection task, outperforming by a large margin the current fully supervised state-of-the-art methods.

@InProceedings{Khoreva_2016_CVPR,author = {Khoreva, Anna and Benenson, Rodrigo and Omran, Mohamed and Hein, Matthias and Schiele, Bernt},title = {Weakly Supervised Object Boundaries},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, Ming-Hsuan Yang

Abstract: We develop a deep learning algorithm for contour detection with a fully convolutional encoder-decoder network. Different from previous low-level edge detection, our algorithm focuses on detecting higher-level object contours. Our network is trained end-to-end on PASCAL VOC with refined ground truth from inaccurate polygon annotations, yielding much higher precision in object contour detection than previous methods. We find that the learned model generalizes well to unseen object classes from the same supercategories on MS COCO and can match state-of-the-art edge detection on BSDS500 with fine-tuning. By combining with the multiscale combinatorial grouping algorithm, our method can generate high-quality segmented object proposals, which significantly advance the state-of-the-art on PASCAL VOC (improving average recall from 0.62 to 0.67) with a relatively small amount of candidates (1660 per image).

@InProceedings{Yang_2016_CVPR,author = {Yang, Jimei and Price, Brian and Cohen, Scott and Lee, Honglak and Yang, Ming-Hsuan},title = {Object Contour Detection With a Fully Convolutional Encoder-Decoder Network},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, Anton van den Hengel

Abstract: Much recent progress in Vision-to-Language (V2L) problems has been achieved through a combination of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). This approach does not explicitly represent high-level semantic concepts, but rather seeks to progress directly from image features to text. In this paper we investigate whether this direct approach succeeds due to, or despite, the fact that it avoids the explicit representation of high-level information. We propose a method of incorporating high-level concepts into the successful CNN-RNN approach, and show that it achieves a significant improvement on the state-of-the-art in both image captioning and visual question answering. We also show that the same mechanism can be used to introduce external semantic information and that doing so further improves performance. We achieve the best reported results on both image captioning and VQA on several benchmark datasets, and provide an analysis of the value of explicit high-level concepts in V2L problems.

@InProceedings{Wu_2016_CVPR,author = {Wu, Qi and Shen, Chunhua and Liu, Lingqiao and Dick, Anthony and van den Hengel, Anton},title = {What Value Do Explicit High Level Concepts Have in Vision to Language Problems?},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: We propose using relaxed deep supervision (RDS) within convolutional neural networks for edge detection. The conventional deep supervision utilizes the general ground-truth to guide intermediate predictions. Instead, we build hierarchical supervisory signals with additional relaxed labels to consider the diversities in deep neural networks. We begin by capturing the relaxed labels from simple detectors (e.g. Canny).Then we merge them with the general ground-truth to generate the RDS. Finally we employ the RDS to supervise the edge network following a coarse-to-fine paradigm. These relaxed labels can be seen as some false positives that are difficult to be classified. We consider these false positives in the supervision, and are able to achieve high performance for better edge detection. We compensate for the lack of training images by capturing coarse edge annotations from a large dataset of image segmentations to pretrain the model. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on the well-known BSDS500 dataset (ODS F-score of .792) and obtains superior cross-dataset generalization results on NYUD dataset.

@InProceedings{Liu_2016_CVPR,author = {Liu, Yu and Lew, Michael S.},title = {Learning Relaxed Deep Supervision for Better Edge Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Nati Ofir, Meirav Galun, Boaz Nadler, Ronen Basri

Abstract: Detecting edges is a fundamental problem in computer vision with many applications, some involving very noisy images. While most edge detection methods are fast, they perform well only on relatively clean images. Unfortunately, sophisticated methods that are robust to high levels of noise are quite slow. In this paper we develop a novel multiscale method to detect curved edges in noisy images. Even though our algorithm searches for edges over an exponentially large set of candidate curves, its runtime is nearly linear in the total number of image pixels. As we demonstrate experimentally, our algorithm is orders of magnitude faster than previous methods designed to deal with high noise levels.  At the same time it obtains comparable and often superior results to existing methods on a variety of challenging noisy images.

@InProceedings{Ofir_2016_CVPR,author = {Ofir, Nati and Galun, Meirav and Nadler, Boaz and Basri, Ronen},title = {Fast Detection of Curved Edges at Low SNR},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Wei Shen, Kai Zhao, Yuan Jiang, Yan Wang, Zhijiang Zhang, Xiang Bai

Abstract: Object skeleton is a useful cue for object detection, complementary to the object contour, as it provides a structural representation to describe the relationship among object parts. While object skeleton extraction in natural images is a very challenging problem, as it requires the extractor to be able to capture both local and global image context to determine the intrinsic scale of each skeleton pixel. Existing methods rely on per-pixel based multi-scale feature computation, which results in difficult modeling and high time consumption. In this paper, we present a fully convolutional network with multiple scale-associated side outputs to address this problem. By observing the relationship between the receptive field sizes of the sequential stages in the network and the skeleton scales they can capture, we introduce a scale-associated side output to each stage. We impose supervision to different stages by guiding the scale-associated side outputs toward groundtruth skeletons of different scales. The responses of the multiple scale-associated side outputs are then fused in a scale-specific way to localize skeleton pixels with multiple scales effectively. Our method achieves promising results on two skeleton extraction datasets, and significantly outperforms other competitors.

@InProceedings{Shen_2016_CVPR,author = {Shen, Wei and Zhao, Kai and Jiang, Yuan and Wang, Yan and Zhang, Zhijiang and Bai, Xiang},title = {Object Skeleton Extraction in Natural Images by Fusing Scale-Associated Deep Side Outputs},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Huan Fu, Chaohui Wang, Dacheng Tao, Michael J. Black

Abstract: Occlusion boundaries contain rich perceptual information about the underlying scene structure. They also provide important cues in many visual perception tasks such as scene understanding, object recognition, and segmentation. In this paper, we improve occlusion boundary detection via enhanced exploration of contextual information (e.g., local structural boundary patterns, observations from surrounding regions, and temporal context), and in doing so develop a novel approach based on convolutional neural networks (CNNs) and conditional random fields (CRFs). Experimental results demonstrate that our detector significantly outperforms the state-of-the-art (e.g., improving the F-measure from 0.62 to 0.71 on the commonly used CMU benchmark). Last but not least, we empirically assess the roles of several important components of the proposed detector, so as to validate the rationale behind this approach.

@InProceedings{Fu_2016_CVPR,author = {Fu, Huan and Wang, Chaohui and Tao, Dacheng and Black, Michael J.},title = {Occlusion Boundary Detection via Deep Exploration of Context},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Zizhao Zhang, Fuyong Xing, Xiaoshuang Shi, Lin Yang

Abstract: Supervised contour detection methods usually require many labeled training images to obtain satisfactory performance. However, a large set of annotated data might be unavailable or extremely labor intensive. In this paper, we investigate the usage of semi-supervised learning (SSL) to obtain competitive detection accuracy with very limited training data (three labeled images). Specifically, we propose a semi-supervised structured ensemble learning approach for contour detection built on structured random forests (SRF). To allow SRF to be applicable to unlabeled data, we present an effective sparse representation approach to capture inherent structure in image patches by finding a compact and discriminative low-dimensional subspace representation in an unsupervised manner, enabling the incorporation of abundant unlabeled patches with their estimated structured labels to help SRF perform better node splitting. We re-examine the role of sparsity and propose a novel and fast sparse coding algorithm to boost the overall learning efficiency. To the best of our knowledge, this is the first attempt to apply SSL for contour detection. Extensive experiments on the BSDS500 segmentation dataset and the NYU Depth dataset demonstrate the superiority of the proposed method.

@InProceedings{Zhang_2016_CVPR,author = {Zhang, Zizhao and Xing, Fuyong and Shi, Xiaoshuang and Yang, Lin},title = {SemiContour: A Semi-Supervised Learning Approach for Contour Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Learning to Localize Little Landmarks

Saurabh Singh, Derek Hoiem, David Forsyth

Abstract: We interact everyday with tiny objects such as the door handle of a car or the light switch in a room. These little landmarks are barely visible and hard to localize in images. We describe a method to find such landmarks by finding a sequence of latent landmarks, each with a prediction model. Each latent landmark predicts the next in sequence, and the last localizes the target landmark. For example, to find the door handle of a car, our method learns to start with a latent landmark near the wheel, as it is globally distinctive; subsequent latent landmarks use the context from the earlier ones to get closer to the target. Our method is supervised solely by the location of the little landmark and displays strong performance on more difficult variants of established tasks and on two new tasks.

@InProceedings{Singh_2016_CVPR,author = {Singh, Saurabh and Hoiem, Derek and Forsyth, David},title = {Learning to Localize Little Landmarks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Lingxi Xie, Liang Zheng, Jingdong Wang, Alan L. Yuille, Qi Tian

Abstract: An increasing number of computer vision tasks can be tackled with deep features, which are the intermediate outputs of a pre-trained Convolutional Neural Network. Despite the astonishing performance, deep features extracted from low-level neurons are still below satisfaction, arguably because they cannot access the spatial context contained in the higher layers. In this paper, we present InterActive, a novel algorithm which computes the activeness of neurons and network connections. Activeness is propagated through a neural network in a top-down manner, carrying high-level context and improving the descriptive power of low-level and mid-level neurons. Visualization indicates that neuron activeness can be interpreted as spatial-weighted neuron responses. We achieve state-of-the-art classification performance on a wide range of image datasets.

@InProceedings{Xie_2016_CVPR,author = {Xie, Lingxi and Zheng, Liang and Wang, Jingdong and Yuille, Alan L. and Tian, Qi},title = {InterActive: Inter-Layer Activeness Propagation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Hao Yang, Joey Tianyi Zhou, Yu Zhang, Bin-Bin Gao, Jianxin Wu, Jianfei Cai

Abstract: Convolutional neural networks (CNNs) have shown great performance as general feature representations for object recognition applications. However, for multi-label images that contain multiple objects from different categories, scales and locations, global CNN features are not optimal. In this paper, we incorporate local information to enhance the feature discriminative power. In particular, we first extract object proposals from each image. With each image treated as a bag and object proposals extracted from it treated as instances, we transform the multi-label recognition problem into a multi-class multi-instance learning problem. Then, in addition to extracting the typical CNN feature representation from each proposal, we propose to make use of ground-truth bounding box annotations (strong labels) to add another level of local information by using nearest-neighbor relationships of local regions to form a multi-view pipeline. The proposed multi-view multi-instance framework utilizes both weak and strong labels effectively, and more importantly it has the generalization ability to even boost the performance of unseen categories by partial strong labels from other categories. Our framework is extensively compared with state-of-the-art hand-crafted feature based methods and CNN based methods on two multi-label benchmark datasets. The experimental results validate the discriminative power and the generalization ability of the proposed framework. With strong labels, our framework is able to achieve state-of-the-art results in both datasets.

@InProceedings{Yang_2016_CVPR,author = {Yang, Hao and Zhou, Joey Tianyi and Zhang, Yu and Gao, Bin-Bin and Wu, Jianxin and Cai, Jianfei},title = {Exploit Bounding Box Annotations for Multi-Label Object Recognition},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Dmitry Laptev, Nikolay Savinov, Joachim M. Buhmann, Marc Pollefeys

Abstract: In this paper we present a deep neural network topology that incorporates a simple to implement transformation-invariant pooling operator (TI-pooling). This operator is able to efficiently handle prior knowledge on nuisance variations in the data, such as rotation or scale changes. Most current methods usually make use of dataset augmentation to address this issue, but this requires larger number of model parameters and more training data, and results in significantly increased training time and larger chance of under- or overfitting. The main reason for these drawbacks is that that the learned model needs to capture adequate features for all the possible transformations of the input. On the other hand, we formulate features in convolutional neural networks to be transformation-invariant. We achieve that using parallel siamese architectures for the considered transformation set and applying the TI-pooling operator on their outputs before the fully-connected layers. We show that this topology internally finds the most optimal "canonical" instance of the input image for training and therefore limits the redundancy in learned features. This more efficient use of training data results in better performance on popular benchmark datasets with smaller number of parameters when comparing to standard convolutional neural networks with dataset augmentation and to other baselines.

@InProceedings{Laptev_2016_CVPR,author = {Laptev, Dmitry and Savinov, Nikolay and Buhmann, Joachim M. and Pollefeys, Marc},title = {TI-Pooling: Transformation-Invariant Pooling for Feature Learning in Convolutional Neural Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: We propose a novel approach for learning features from weakly-supervised data by joint ranking and classification. In order to exploit data with weak labels, we jointly train a feature extraction network with a ranking loss and a classification network with a cross-entropy loss. We obtain high-quality compact discriminative features with few parameters, learned on relatively small datasets without additional annotations. This enables us to tackle tasks with specialized images not very similar to the more generic ones in existing fully-supervised datasets. We show that the resulting features in combination with a linear classifier surpass the state-of-the-art on the Hipster Wars dataset despite using features only 0.3% of the size. Our proposed features significantly outperform those obtained from networks trained on ImageNet, despite being 32 times smaller (128 single-precision floats), trained on noisy and weakly-labeled data, and using only 1.5% of the number of parameters.

@InProceedings{Simo-Serra_2016_CVPR,author = {Simo-Serra, Edgar and Ishikawa, Hiroshi},title = {Fashion Style in 128 Floats: Joint Ranking and Classification Using Weak Data for Feature Extraction},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Compact Bilinear Pooling

Yang Gao, Oscar Beijbom, Ning Zhang, Trevor Darrell

Abstract: Bilinear models has been shown to achieve impressive performance on a wide range of visual tasks, such as semantic segmentation, fine grained recognition and face recognition. However, bilinear features are high dimensional, typically on the order of hundreds of thousands to a few million, which makes them impractical for subsequent analysis. We propose two compact bilinear representations with the same discriminative power as the full bilinear representation but with only a few thousand dimensions. Our compact representations allow back-propagation of classification errors enabling an end-to-end optimization of the visual recognition system. The compact bilinear representations are derived through a novel kernelized analysis of bilinear pooling which provide insights into the discriminative power of bilinear pooling, and a platform for further research in compact pooling methods. Experimentation illustrate the utility of the proposed representations for image classification and few-shot learning across several datasets.

@InProceedings{Gao_2016_CVPR,author = {Gao, Yang and Beijbom, Oscar and Zhang, Ning and Darrell, Trevor},title = {Compact Bilinear Pooling},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Tsun-Yi Yang, Yen-Yu Lin, Yung-Yu Chuang

Abstract: This paper proposes a novel local descriptor through accumulated stability voting (ASV). The stability of feature dimensions is measured by their differences across scales. To be more robust to noise, the stability is further quantized by thresholding. The principle of maximum entropy is utilized for determining the best thresholds for maximizing discriminant power of the resultant descriptor. Accumulating stability renders a real-valued descriptor and it can be converted into a binary descriptor by an additional thresholding process. The real-valued descriptor attains high matching accuracy while the binary descriptor makes a good compromise between storage and accuracy. Our descriptors are simple yet effective, and easy to implement. In addition, our descriptors require no training. Experiments on popular benchmarks demonstrate the effectiveness of our descriptors and their superiority to the state-of-the-art descriptors.

@InProceedings{Yang_2016_CVPR,author = {Yang, Tsun-Yi and Lin, Yen-Yu and Chuang, Yung-Yu},title = {Accumulated Stability Voting: A Robust Descriptor From Descriptors of Multiple Scales},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: Most existing dictionary learning algorithms consider a linear sparse model, which often  cannot effectively characterize the nonlinear properties present in many types of visual data, e.g. dynamic texture (DT). Such nonlinear properties can be exploited by the so-called  kernel sparse coding. This paper proposed an equiangular kernel dictionary learning method with optimal mutual coherence to exploit  the nonlinear sparsity of  high-dimensional visual data. Two main issues are addressed in the proposed method: (1) coding stability for redundant dictionary of infinite-dimensional space; and (2) computational efficiency for computing kernel matrix of training samples of high-dimensional data. The proposed kernel sparse coding method is applied to dynamic texture analysis with both local DT pattern extraction and global DT pattern characterization. The experimental results showed its performance gain over existing methods.

@InProceedings{Quan_2016_CVPR,author = {Quan, Yuhui and Bao, Chenglong and Ji, Hui},title = {Equiangular Kernel Dictionary Learning With Applications to Dynamic Texture Analysis},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Da Chen, Jean-Marie Mirebeau, Laurent D. Cohen

Abstract: In this paper, we propose a new curvature penalized minimal path model for image segmentation via closed contour detection based on the weighted Euler elastica curves, firstly introduced to the field of computer vision in [22]. Our image segmentation method extracts a collection of curvature penalized minimal geodesics, concatenated to form a closed contour, by connecting a set of user-specified points. Globally optimal minimal paths can be computed by solving an Eikonal equation. This first order PDE is traditionally regarded as unable to penalize curvature, which is related to the path acceleration in active contour models. We introduce here a new approach that enables finding a global minimum of the geodesic energy including a curvature term. We achieve this through the use of a novel Finsler metric adding to the image domain the orientation as an extra space dimension. This metric is non-Riemannian and asymmetric, defined on an orientation lifted space, incorporating the curvature penalty in the geodesic energy. Experiments show that the proposed Finsler minimal path model indeed outperforms state-of-the-art minimal path models in both synthetic and real images.

@InProceedings{Chen_2016_CVPR,author = {Chen, Da and Mirebeau, Jean-Marie and Cohen, Laurent D.},title = {A New Finsler Minimal Path Model With Curvature Penalization for Image Segmentation and Closed Contour Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: Traditional Feature Detectors and Trackers use information aggregation in 2D patches to detect and match discriminative patches.  However, this information does not remain the same at object boundaries when there is object motion against a significantly varying background.  In this paper, we propose a new approach for feature detection, tracking and re-detection that gives significantly improved results at the object boundaries.  We utilize level lines or iso-intensity curves that often remain stable and can be reliably detected even at the object boundaries, which they often trace. Stable portions of long level lines are detected and points of high curvature  are detected on such curves for corner detection.  Further, this level line is used to separate the portions belonging to the two objects, which is then used for robust matching of such points.  While such CoMaL (Corners on Maximally-stable Level Line Segments) points were found to be much more reliable at the object boundary regions, they perform comparably at the interior regions as well. This is illustrated in exhaustive experiments on real-world datasets.

@InProceedings{Ravindran_2016_CVPR,author = {Ravindran, Swarna K. and Mittal, Anurag},title = {CoMaL: Good Features to Match on Object Boundaries},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: We address two difficulties in establishing an accurate system for image matching. First, image matching relies on the descriptor for feature extraction, but the optimal descriptor often varies from image to image, or even patch to patch. Second, conventional matching approaches carry out geometric checking on a small set of correspondence candidates due to the concern of efficiency. It may result in restricted performance in recall. We aim at tackling the two issues by integrating adaptive descriptor selection and progressive candidate enrichment into image matching. We consider that the two integrated components are complementary: The high-quality matching yielded by adaptively selected descriptors helps in exploring more plausible candidates, while the enriched candidate set serves as a better reference for descriptor selection. It motivates us to formulate image matching as a joint optimization problem, in which adaptive descriptor selection and progressive correspondence enrichment are alternately conducted. Our approach is comprehensively evaluated and compared with the state-of-the-art approaches on two benchmarks. The promising results manifest its effectiveness.

@InProceedings{Hu_2016_CVPR,author = {Hu, Yuan-Ting and Lin, Yen-Yu},title = {Progressive Feature Matching With Alternate Descriptor Selection and Correspondence Enrichment},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Yuhua Chen, Dengxin Dai, Jordi Pont-Tuset, Luc Van Gool

Abstract: Image segmentation is a key component in many computer vision systems, and it is recovering a prominent spot in the literature as methods improve and overcome their limitations. The outputs of most recent algorithms are in the form of a hierarchical segmentation, which provides segmentation at different scales in a single tree-like structure. Commonly, these hierarchical methods start from some low-level features, and are not aware of the scale information of the different regions in them. As such, one might need to work on many different levels of the hierarchy to find the objects in the scene. This work tries to modify the existing hierarchical algorithm by improving their alignment, that is, by trying to modify the depth of the regions in the tree to better couple depth and scale. To do so, we first train a regressor to predict the scale of regions using mid-level features. We then define the anchor slice as the set of regions that better balance between over-segmentation and under-segmentation. The output of our method is an improved hierarchy, re-aligned by the anchor slice. To demonstrate the power of our method, we perform comprehensive experiments, which show that our method, as a post-processing step, can significantly improve the quality of the hierarchical segmentation representations, and ease the usage of hierarchical image segmentation to high-level vision tasks such as object segmentation. We also prove that the improvement generalizes well across different algorithms and datasets, with a low computational cost.

@InProceedings{Chen_2016_CVPR,author = {Chen, Yuhua and Dai, Dengxin and Pont-Tuset, Jordi and Van Gool, Luc},title = {Scale-Aware Alignment of Hierarchical Image Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Deep Interactive Object Selection

Ning Xu, Brian Price, Scott Cohen, Jimei Yang, Thomas S. Huang

Abstract: Interactive object selection is a very important research problem and has many applications. Previous algorithms require substantial user interactions to estimate the foreground and background distributions. In this paper, we present a novel deep-learning-based algorithm which has much better understanding of objectness and can reduce user interactions to just a few clicks. Our algorithm transforms user-provided positive and negative clicks into two Euclidean distance maps which are then concatenated with the RBG channels of images to compose (image, user interactions) pairs. We generate many of such pairs by combining several random sampling strategies to model users' click patterns and use them to finetune deep Fully Convolutional Networks (FCNs). Finally the output probability maps of our FCN-8s model is integrated with graph cut optimization to refine the boundary segments. Our model is trained on the PASCAL segmentation dataset and evaluated on other datasets with different object classes. Experimental results on both seen and unseen objects clearly demonstrate that our algorithm has a good generalization ability and is superior to all existing interactive object selection approaches.

@InProceedings{Xu_2016_CVPR,author = {Xu, Ning and Price, Brian and Cohen, Scott and Yang, Jimei and Huang, Thomas S.},title = {Deep Interactive Object Selection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Yuka Kihara, Matvey Soloviev, Tsuhan Chen

Abstract: We present a new algorithm for multi-region segmentation of 2D images with objects that may partially occlude each other. Our algorithm is based on the observation that human performance on this task is based both on prior knowledge about plausible shapes and taking into account the presence of occluding objects whose shape is already known - once an occluded region is identified, the shape prior can be used to guess the shape of the missing part. We capture the former aspect using a deep learning model of shape; for the latter, we simultaneously minimize the energy of all regions and consider only unoccluded pixels for data agreement.   Existing algorithms incorporating object shape priors consider every object separately in turn and can't distinguish genuine deviation from the expected shape from parts missing due to occlusion. We show that our method significantly improves on the performance of a representative algorithm, as evaluated on both preprocessed natural and synthetic images. Furthermore, on the synthetic images, we recover the ground truth segmentation with good accuracy.

@InProceedings{Kihara_2016_CVPR,author = {Kihara, Yuka and Soloviev, Matvey and Chen, Tsuhan},title = {In the Shadows, Shape Priors Shine: Using Occlusion to Improve Multi-Region Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Loic A. Royer, David L. Richmond, Carsten Rother, Bjoern Andres, Dagmar Kainmueller

Abstract: Segmenting an image into multiple components is a central task in computer vision. In many practical scenarios, prior knowledge about plausible components is available. Incorporating such prior knowledge into models and algorithms for image segmentation is highly desirable, yet can be non-trivial. In this work, we introduce a new approach that allows, for the first time, to constrain some or all components of a segmentation to have convex shapes. Specifically, we extend the Minimum Cost Multicut Problem by a class of constraints that enforce convexity. To solve instances of this NP-hard integer linear program to optimality, we separate the proposed constraints in the branch-and-cut loop of a state-of-the-art ILP solver. Results on photographs and micrographs demonstrate the effectiveness of the approach as well as its advantages over the state-of-the-art heuristic.

@InProceedings{Royer_2016_CVPR,author = {Royer, Loic A. and Richmond, David L. and Rother, Carsten and Andres, Bjoern and Kainmueller, Dagmar},title = {Convexity Shape Constraints for Image Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Ertunc Erdil, Sinan Yildirim, Mujdat Cetin, Tolga Tasdizen

Abstract: Segmenting images of low quality or with missing data is a challenging problem. Integrating statistical prior information about the shapes to be segmented can improve the segmentation results significantly. Most shape-based segmentation algorithms optimize an energy functional and find a point estimate for the object to be segmented. This does not provide a measure of the degree of confidence in that result, neither does it provide a picture of other probable solutions based on the data and the priors. With a statistical view, addressing these issues would involve the problem of characterizing the posterior densities of the shapes of the objects to be segmented. For such characterization, we propose a Markov chain Monte Carlo (MCMC) sampling-based image segmentation algorithm that uses statistical shape priors. In addition to better characterization of the statistical structure of the problem, such an approach would also have the potential to address issues with getting stuck at local optima, suffered by existing shape-based segmentation methods. Our approach is able to characterize the posterior probability density in the space of shapes through its samples, and to return multiple solutions, potentially from different modes of a multimodal probability density, which would be encountered, e.g., in segmenting objects from multiple shape classes. We present promising results on a variety of data sets. We also provide an extension for segmenting shapes of objects with parts that can go through independent shape variations. This extension involves the use of local shape priors on object parts and provides robustness to limitations in shape training data size.

@InProceedings{Erdil_2016_CVPR,author = {Erdil, Ertunc and Yildirim, Sinan and Cetin, Mujdat and Tasdizen, Tolga},title = {MCMC Shape Sampling for Image Segmentation With Nonparametric Shape Priors},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Danna Gurari, Suyog Jain, Margrit Betke, Kristen Grauman

Abstract: Foreground object segmentation is a critical step for many image analysis tasks.  While automated methods can produce high-quality results, their failures disappoint users in need of practical solutions.  We propose a resource allocation framework for predicting how best to allocate a fixed budget of human annotation effort in order to collect higher quality segmentations for a given batch of images and automated methods.  The framework is based on a proposed prediction module that estimates the quality of given algorithm-drawn segmentations.  We demonstrate the value of the framework for two novel tasks related to "pulling the plug" on computer and human annotators.  Specifically, we implement two systems that automatically decide, for a batch of images, when to replace 1) humans with computers to create coarse segmentations required to initialize segmentation tools and 2) computers with humans to create final, fine-grained segmentations.  Experiments demonstrate the advantage of relying on a mix of human and computer efforts over relying on either resource alone for segmenting objects in three diverse datasets representing visible, phase contrast microscopy, and fluorescence microscopy images.

@InProceedings{Gurari_2016_CVPR,author = {Gurari, Danna and Jain, Suyog and Betke, Margrit and Grauman, Kristen},title = {Pull the Plug? Predicting If Computers or Humans Should Segment Images},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Fengyuan Zhu, Guangyong Chen, Pheng-Ann Heng

Abstract: Traditional image denoising algorithms always assume the noise to be homogeneous white Gaussian distributed. However, the noise on real images can be much more complex empirically. This paper addresses this problem and proposes a novel blind image denoising algorithm which can cope with real-world noisy images even when the noise model is not provided. It is realized by modeling image noise with mixture of Gaussian distribution (MoG) which can approximate large varieties of continuous distributions. As the number of components for MoG is unknown practically, this work adopts Bayesian nonparametric technique and proposes a novel Low-rank MoG filter (LR-MoG) to recover clean signals (patches) from noisy ones contaminated by MoG noise. Based on LR-MoG, a novel blind image denoising approach is developed. To test the proposed method, this study conducts extensive experiments on synthesis and real images. Our method achieves the state-of-the-art performance consistently.

@InProceedings{Zhu_2016_CVPR,author = {Zhu, Fengyuan and Chen, Guangyong and Heng, Pheng-Ann},title = {From Noise Modeling to Blind Image Denoising},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, Amit Ashok

Abstract: The goal of this paper is to present a non-iterative and more importantly an extremely fast algorithm to reconstruct images from compressively sensed (CS) random measurements.  To this end, we propose a novel convolutional neural network (CNN) architecture which takes in CS measurements of an image as input and outputs  an intermediate  reconstruction.  We call this network, ReconNet. The intermediate reconstruction is fed into an off-the-shelf denoiser to obtain the final reconstructed image. On a standard dataset of  images we show significant improvements in reconstruction results (both in terms of PSNR and time complexity) over state-of-the-art iterative CS reconstruction algorithms at various measurement rates. Further, through qualitative experiments on real data collected using our block SPC (single pixel camera), we show that our network is highly robust to sensor noise and can recover visually better quality images than competitive algorithms at extremely low sensing rates of 0.1 and 0.04. To demonstrate that our algorithm can recover semantically informative images even at a low measurement rate of 0.01, we present a very robust proof of concept real-time visual tracking application.

@InProceedings{Kulkarni_2016_CVPR,author = {Kulkarni, Kuldeep and Lohit, Suhas and Turaga, Pavan and Kerviche, Ronan and Ashok, Amit},title = {ReconNet: Non-Iterative Reconstruction of Images From Compressively Sensed Measurements},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: Reliable patch-matching forms the basis for many algorithms (super-resolution, denoising, inpainting, etc.) However, when the image quality deteriorates (by noise, blur or  geometric distortions), the reliability of patch-matching deteriorates as well. Matched patches in the degraded image, do not necessarily imply similarity of the underlying patches in the (unknown) high-quality image. This restricts the applicability of patch-based methods. In this paper we present a patch representation called "Needle", which consists of small multi-scale versions of the patch and its immediate surrounding region. While the patch at the finest image scale is severely degraded, the degradation decreases dramatically in coarser needle scales, revealing reliable information for matching. We show that the Needle is  robust to many types of image degradations, leads to matches faithful to the underlying high-quality patches, and to improvement in existing patch-based methods.

@InProceedings{Lotan_2016_CVPR,author = {Lotan, Or and Irani, Michal},title = {Needle-Match: Reliable Patch Matching Under High Uncertainty},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Dongliang Cheng, Abdelrahman Abdelhamed, Brian Price, Scott Cohen, Michael S. Brown

Abstract: This paper examines the problem of white-balance correction when a scene contains two illuminations.  This is a two step process: 1) estimate the two illuminants; and 2) correct the image.   Existing methods attempt to estimate a spatially varying illumination map, however, results are error prone and the resulting illumination maps are too low-resolution to be used for proper spatially varying white-balance correction.  In addition, the spatially varying nature of these methods make them computationally intensive.   We show that this problem can be effectively addressed by not attempting to obtain a spatially varying illumination map, but instead by performing illumination estimation on large sub-regions of the image.   Our approach is able to detect when distinct illuminations are present in the image and accurately measure these illuminants.   Since our proposed strategy is not suitable for spatially varying image correction, a user study is performed to see if there is a preference for how the image should be corrected when two illuminants are present, but only a global correction can be applied.   The user study shows that when the illuminations are distinct, there is a preference for the outdoor illumination to be corrected resulting in warmer final result.  We use these collective findings to demonstrate an effective two illuminant estimation scheme that produces corrected images that users prefer.

@InProceedings{Cheng_2016_CVPR,author = {Cheng, Dongliang and Abdelhamed, Abdelrahman and Price, Brian and Cohen, Scott and Brown, Michael S.},title = {Two Illuminant Estimation and User Correction Preference},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Jinshan Pan, Zhe Hu, Zhixun Su, Hsin-Ying Lee, Ming-Hsuan Yang

Abstract: Object motion blur is a challenging problem as the foreground and the background in the scenes undergo different types of image degradation due to movements in various directions and speed. Most object motion deblurring methods address this problem by segmenting blurred images into regions where different kernels are estimated and applied for restoration. Segmentation on blurred images is difficult due to ambiguous pixels between regions, but it plays an important role for object motion deblurring. To address these problems, we propose a novel model for object motion deblurring. The proposed model is developed based on a maximum a posterior formulation in which soft-segmentation is incorporated for object layer estimation. We propose an efficient algorithm to jointly estimate object segmentation and camera motion where each layer can be deblurred well under the guidance of the soft-segmentation. Experimental results demonstrate that the proposed algorithm performs favorably against the state-of-the-art object motion deblurring methods on challenging scenarios.

@InProceedings{Pan_2016_CVPR,author = {Pan, Jinshan and Hu, Zhe and Su, Zhixun and Lee, Hsin-Ying and Yang, Ming-Hsuan},title = {Soft-Segmentation Guided Object Motion Deblurring},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Jaesik Park, Yu-Wing Tai, Sudipta N. Sinha, In So Kweon

Abstract: We present an efficient technique to optimize color consistency of a collection of images depicting a common scene. Our method first recovers sparse pixel correspondences in the input images and stacks them into a matrix with many missing entries. We show that this matrix satisfies a rank two constraint under a simple color correction model. These parameters can be viewed as pseudo white balance and gamma correction parameters for each input image. We present a robust low-rank matrix factorization method to estimate the unknown parameters of this model. Using them, we improve color consistency of the input images or perform color transfer with any input image as the source. Our approach is insensitive to outliers in the pixel correspondences thereby precluding the need for complex pre-processing steps. We demonstrate high-quality color consistency results on large photo collections of popular tourist landmarks and personal photo collections containing images of people.

@InProceedings{Park_2016_CVPR,author = {Park, Jaesik and Tai, Yu-Wing and Sinha, Sudipta N. and Kweon, In So},title = {Efficient and Robust Color Consistency for Community Photo Collections},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: Photo aesthetics assessment is challenging. Deep convolutional neural network (ConvNet) methods have recently shown promising results for aesthetics assessment. The performance of these deep ConvNet methods, however, is often compromised by the constraint that the neural network only takes the fixed-size input. To accommodate this requirement, input images need to be transformed via cropping, scaling, or padding, which often damages image composition, reduces image resolution, or causes image distortion, thus compromising the aesthetics of the original images. In this paper, we present a composition-preserving deep ConvNet method that directly learns aesthetics features from the original input images without any image transformations. Specifically, our method adds an adaptive spatial pooling layer upon the regular convolution and pooling layers to directly handle input images with original sizes and aspect ratios. To allow for multi-scale feature extraction, we develop the Multi-Net Adaptive Spatial Pooling ConvNet architecture which consists of multiple sub-networks with different adaptive spatial pooling sizes and leverage a scene-based aggregation layer to effectively combine the predictions from multiple sub-networks. Our experiments on the large-scale aesthetics assessment benchmark (AVA) demonstrate that our method can significantly improve the state-of-the-art results in photo aesthetics assessment.

@InProceedings{Mai_2016_CVPR,author = {Mai, Long and Jin, Hailin and Liu, Feng},title = {Composition-Preserving Deep Photo Aesthetics Assessment},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: Salient object detection has recently witnessed substantial progress due to powerful features extracted using deep convolutional neural networks (CNNs). However, existing CNN-based methods operate at the patch level instead of the pixel level. Resulting saliency maps are typically blurry, especially near the boundary of salient objects. Furthermore, image patches are treated as independent samples even when they are overlapping, giving rise to significant redundancy in computation and storage. In this paper, we propose an end-to-end deep contrast network to overcome the aforementioned limitations. Our deep network consists of two complementary components, a pixel-level fully convolutional stream and a segment-wise spatial pooling stream. The first stream directly produces a saliency map with pixel-level accuracy from an input image. The second stream extracts segment-wise features very efficiently, and better models saliency discontinuities along object boundaries. Finally, a fully connected CRF model can be optionally incorporated to improve spatial coherence and contour localization in the fused result from these two streams. Experimental results demonstrate that our deep model significantly improves the state of the art.

@InProceedings{Li_2016_CVPR,author = {Li, Guanbin and Yu, Yizhou},title = {Deep Contrast Learning for Salient Object Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Jiansheng Chen, Gaocheng Bai, Shaoheng Liang, Zhengqin Li

Abstract: Attention based automatic image cropping aims at preserving the most visually important region in an image. A common task in this kind of method is to search for the smallest rectangle inside which the summed attention is maximized. We demonstrate that under appropriate formulations, this task can be achieved using efficient algorithms with low computational complexity. In a  practically useful scenario where the aspect ratio of the cropping rectangle is given, the problem can be solved with a computational complexity linear to the number of image pixels. We also study the possibility of multiple rectangle cropping and a new model facilitating fully automated image cropping.

@InProceedings{Chen_2016_CVPR,author = {Chen, Jiansheng and Bai, Gaocheng and Liang, Shaoheng and Li, Zhengqin},title = {Automatic Image Cropping : A Computational Complexity Study},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Neil D. B. Bruce, Christopher Catton, Sasa Janjic

Abstract: In this paper we consider the problem of visual saliency modeling, including both human gaze prediction and salient object segmentation. The overarching goal of the paper is to identify high level considerations relevant to deriving more sophisticated visual saliency models. A deep learning model based on fully convolutional networks (FCNs) is presented, which shows very favorable performance across a wide variety of benchmarks relative to existing proposals. We also demonstrate that the manner in which training data is selected, and ground truth treated is critical to resulting model behaviour. Recent efforts have explored the relationship between human gaze and salient objects, and we also examine this point further in the context of FCNs. Close examination of the proposed and alternative models serves as a vehicle for identifying problems important to developing more comprehensive models going forward.

@InProceedings{Bruce_2016_CVPR,author = {Bruce, Neil D. B. and Catton, Christopher and Janjic, Sasa},title = {A Deeper Look at Saliency: Feature Contrast, Semantics, and Beyond},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Seung-Hwan Baek, Inchang Choi, Min H. Kim

Abstract: We present a multiview image completion method that provides geometric consistency among different views by propagating space structures. Since a user specifies the region to be completed in one of multiview photographs casually taken in a scene, the proposed method enables us to complete the set of photographs with geometric consistency by creating or removing structures on the specified region. The proposed method incorporates photographs to estimate dense depth maps. We initially complete color as well as depth from a view, and then facilitate two stages of structure propagation and structure-guided completion. Structure propagation optimizes space topology in the scene across photographs, while structure-guide completion enhances, and completes local image structure of both depth and color in multiple photographs with structural coherence by searching nearest neighbor fields in relevant views. We demonstrate the effectiveness of the proposed method in completing multiview images.

@InProceedings{Baek_2016_CVPR,author = {Baek, Seung-Hwan and Choi, Inchang and Kim, Min H.},title = {Multiview Image Completion With Space Structure Propagation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Oriel Frigo, Neus Sabater, Julie Delon, Pierre Hellier

Abstract: This paper presents a novel unsupervised method to transfer the style of an example image to a source image. The complex notion of image style is here considered as a local texture transfer, eventually coupled with a global color transfer. For the local texture transfer, we propose a new method based on an adaptive patch partition that captures the style of the example image and preserves the structure of the source image. More precisely, this example-based partition predicts how well a source patch matches an example patch. Results on various images show that our method outperforms the most recent techniques.

@InProceedings{Frigo_2016_CVPR,author = {Frigo, Oriel and Sabater, Neus and Delon, Julie and Hellier, Pierre},title = {Split and Match: Example-Based Adaptive Patch Sampling for Unsupervised Style Transfer},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Anna Volokitin, Michael Gygli, Xavier Boix

Abstract: Many computational models of visual attention use image features and machine learning techniques to predict eye fixation locations as saliency maps. Recently, the success of Deep Convolutional Neural Networks (DCNNs) for object recognition has opened a new avenue for computational models of visual attention due to the tight link between visual attention and object recognition. In this paper, we show that using features from DCNNs for object recognition we can make predictions that enrich the information provided by saliency models. Namely, we can estimate the reliability of a saliency model from the raw image, which serves as a meta-saliency measure that may be used to select the best saliency algorithm for an image. Analogously, the consistency of the eye fixations among  subjects, i.e. the agreement between the eye fixation locations of different subjects, can also be predicted and used by a designer to assess whether subjects reach a consensus about salient image locations.

@InProceedings{Volokitin_2016_CVPR,author = {Volokitin, Anna and Gygli, Michael and Boix, Xavier},title = {Predicting When Saliency Maps Are Accurate and Eye Fixations Consistent},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: A recent trend in saliency algorithm development is large-scale benchmarking and algorithm ranking with ground truth provided by datasets of human fixations. In order to accommodate the strong bias humans have toward central fixations, it is common to replace traditional ROC metrics with a shuffled ROC metric which uses randomly sampled fixations from other images in the database as the negative set. However, the shuffled ROC introduces a number of problematic elements, including a fundamental assumption that it is possible to separate visual salience and image spatial arrangement.  We argue that it is more informative to directly measure the effect of spatial bias on algorithm performance rather than try to correct for it. To capture and quantify these known sources of bias, we propose a novel metric for measuring saliency algorithm performance: the spatially binned ROC (spROC). This metric provides direct insight into the spatial biases of a saliency algorithm without sacrificing the intuitive raw performance evaluation of traditional ROC measurements. By quantitatively measuring the bias in saliency algorithms, researchers will be better equipped to select and optimize the most appropriate algorithm for a given task. We use a baseline measure of inherent algorithm bias to show that Adaptive Whitening Saliency (AWS) [14], Attention by Information Maximization (AIM) [8], and Dynamic Visual Attention (DVA) [20] provide the least spatially biased results, suiting them for tasks in which there is no information about the underlying spatial bias of the stimuli, whereas algorithms such as Graph Based Visual Saliency (GBVS) [18] and Context-Aware Saliency (CAS) [15] have a significant inherent central bias.

@InProceedings{Wloka_2016_CVPR,author = {Wloka, Calden and Tsotsos, John},title = {Spatially Binned ROC: A Comprehensive Saliency Metric},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Qiaosong Wang, Wen Zheng, Robinson Piramuthu

Abstract: We propose an unsupervised bottom-up saliency detection approach by exploiting novel graph structure and background priors. The input image is represented as an undirected graph with superpixels as nodes. Feature vectors are extracted from each node to cover regional color, contrast and texture information. A novel graph model is proposed to effectively capture local and global saliency cues. To obtain more accurate saliency estimations, we optimize the saliency map by using a robust background measure. Comprehensive evaluations on benchmark datasets indicate that our algorithm universally surpasses state-of-the-art unsupervised solutions and performs favorably against supervised approaches.

@InProceedings{Wang_2016_CVPR,author = {Wang, Qiaosong and Zheng, Wen and Piramuthu, Robinson},title = {GraB: Visual Saliency via Novel Graph Model and Background Priors},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Lilian Calvet, Pierre Gurdjos, Carsten Griwodz, Simone Gasparini

Abstract: Using fiducial markers ensures reliable detection and identification of planar features in images. Fiducials are used in a wide range of applications, especially when a reliable visual reference is needed, e.g., to track the camera in cluttered or textureless environments. A marker designed for such applications must be robust to partial occlusions, varying distances and angles of view, and fast camera motions. In this paper, we present a robust, highly accurate fiducial system, whose markers consist of concentric rings, along with its theoretical foundations. Relying on projective properties, it allows to robustly localize the imaged marker and to accurately detect the position of the image of the (common) circle center. We demonstrate that our system can detect and accurately localize these circular fiducials under very challenging conditions and the experimental results reveal that it outperforms other recent fiducial systems.

@InProceedings{Calvet_2016_CVPR,author = {Calvet, Lilian and Gurdjos, Pierre and Griwodz, Carsten and Gasparini, Simone},title = {Detection and Accurate Localization of Circular Fiducials Under Highly Challenging Conditions},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Junting Pan, Elisa Sayrol, Xavier Giro-i-Nieto, Kevin McGuinness, Noel E. O'Connor

Abstract: The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. Two designs are proposed: a shallow convnet trained from scratch, and a another deeper solution whose first three layers are adapted from another network trained for classification. To the authors knowledge, these are the first end-to-end CNNs trained and tested for the purpose of saliency prediction.

@InProceedings{Pan_2016_CVPR,author = {Pan, Junting and Sayrol, Elisa and Giro-i-Nieto, Xavier and McGuinness, Kevin and O'Connor, Noel E.},title = {Shallow and Deep Convolutional Networks for Saliency Prediction},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, Yi Ma

Abstract: This paper aims to develop a method that can accurately estimate the crowd count from an individual image with arbitrary crowd density and arbitrary perspective. To this end,we have proposed a simple but effective Multi-column Convolutional Neural Network (MCNN) architecture to map the image to its crowd density map. The proposed MCNN allows the input image to be of arbitrary size or resolution. By utilizing filters with receptive fields of different sizes, the features learned by each column CNN are adaptive to variations in people/head size due to perspective effect or image resolution. Furthermore, the true density map is computed accurately based on geometry-adaptive kernels which do not need knowing the perspective map of the input image. Since exiting crowd counting datasets do not adequately cover all the challenging situations considered in our work, we have collected and labelled a large new dataset that includes 1198 images with about 330,000 heads annotated. On this challenging new dataset, as well as all existing datasets, we conduct extensive experiments to verify the effectiveness of the proposed model and method. In particular, with the proposed simple MCNN model, our method outperforms all existing methods. In addition, experiments show that our model, once trained on one dataset, can be readily transferred to a new dataset.

@InProceedings{Zhang_2016_CVPR,author = {Zhang, Yingying and Zhou, Desen and Chen, Siqin and Gao, Shenghua and Ma, Yi},title = {Single-Image Crowd Counting via Multi-Column Convolutional Neural Network},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Mohammad Najafi, Sarah Taghavi Namin, Mathieu Salzmann, Lars Petersson

Abstract: Scene parsing has attracted a lot of attention in computer vision. While parametric models have proven effective for this task, they cannot easily incorporate new training data. By contrast, nonparametric approaches, which bypass any learning phase and directly transfer the labels from the training data to the query images, can readily exploit new labeled samples as they become available. Unfortunately, because of the computational cost of their label transfer procedures, state-of-the-art nonparametric methods typically filter out most training images to only keep a few  relevant ones to label the query. As such, these methods throw away many images that still contain valuable information and generally obtain an unbalanced set of labeled samples. In this paper, we introduce a nonparametric approach to scene parsing that follows a sample-and-filter strategy. More specifically, we propose to sample labeled superpixels according to an image similarity score, which allows us to obtain a balanced set of samples. We then formulate label transfer as an efficient filtering procedure, which lets us exploit more labeled samples than existing techniques. Our experiments evidence the benefits of our approach over state-of-the-art nonparametric methods on two benchmark datasets.

@InProceedings{Najafi_2016_CVPR,author = {Najafi, Mohammad and Namin, Sarah Taghavi and Salzmann, Mathieu and Petersson, Lars},title = {Sample and Filter: Nonparametric Scene Parsing via Efficient Filtering},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: When people observe and interact with physical spaces, they are able to associate functionality to regions in the environment. Our goal is to automate functional understanding of large spaces by leveraging activity demonstrations recorded from an ego-centric viewpoint. The method we describe enables functionality estimation in both large scenes where people have behaved, as well as novel scenes where no behaviors are available. Our method learns and predicts "Action Maps", which encode the ability for a user to perform activities at various locations. With the usage of an egocentric camera to observe demonstrations, our method scales with the size of the scene without the need for mounting multiple static surveillance cameras, and is well-suited to the task of observing activities up-close. We demonstrate that by capturing appearance-based attributes of the environment and associating these attributes with activity demonstrations, our mathematical framework allows for the prediction of Action Maps in new environments. Additionally, we take a preliminary look at the breadth of applicability of Action Maps by demonstrating a proof-of-concept application in which they are used in concert with activity detections to perform localization.

@InProceedings{Rhinehart_2016_CVPR,author = {Rhinehart, Nicholas and Kitani, Kris M.},title = {Learning Action Maps of Large Environments via First-Person Vision},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Luis Herranz, Shuqiang Jiang, Xiangyang Li

Abstract: Since scenes are composed in part of objects, accurate recognition of scenes requires knowledge about both scenes and objects. In this paper we address two related problems: 1) scale induced dataset bias in multi-scale convolutional neural network (CNN) architectures, and 2) how to combine effectively scene-centric and object-centric knowledge (i.e. Places and ImageNet) in CNNs. An earlier attempt, Hybrid-CNN, showed that incorporating ImageNet did not help much. Here we propose an alternative method taking the scale into account, resulting in significant recognition gains. By analyzing the response of ImageNet-CNNs and Places-CNNs at different scales we find that both operate in different scale ranges, so using the same network for all the scales induces dataset bias resulting in limited performance. Thus, adapting the feature extractor to each particular scale (i.e. scale-specific CNNs) is crucial to improve recognition, since the objects in the scenes have their specific range of scales. Experimental results show that the recognition accuracy highly depends on the scale, and that simple yet carefully chosen multi-scale combinations of ImageNet-CNNs and Places-CNNs, can push the state-of-the-art recognition accuracy in SUN397 up to 66.26% (and even 70.17% with deeper architectures, comparable to human performance).

@InProceedings{Herranz_2016_CVPR,author = {Herranz, Luis and Jiang, Shuqiang and Li, Xiangyang},title = {Scene Recognition With CNNs: Objects, Scales and Dataset Bias},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}


Abstract: We propose a system that finds text in natural scenes using a variety of cues. Our novel data-driven method incorporates coarse-to-fine detection of character pixels using convolutional features (Text-Conv), followed by extracting connected components (CCs) from characters using edge and color features, and finally performing a graph-based segmentation of CCs into words (Word-Graph). For Text-Conv, the initial detection is based on convolutional feature maps similar to those used in Convolutional Neural Networks (CNNs), but learned using Convolutional k-means. Convolution masks defined by local and neighboring patch features are used to improve detection accuracy. The Word-Graph algorithm uses contextual information to both improve word segmentation and prune false character/word detections. Different definitions for foreground (text) regions are used to train the detection stages, some based on bounding box intersection, and others on bounding box and pixel intersection. Our system obtains pixel, character, and word detection f-measures of 93.14%, 90.26%, and 86.77% respectively for the ICDAR 2015 Robust Reading Focused Scene Text dataset, out-performing state-of-the-art systems. This approach may work for other detection targets with homogenous color in natural scenes.

@InProceedings{Zhu_2016_CVPR,author = {Zhu, Siyu and Zanibbi, Richard},title = {A Text Detection System for Natural Scenes With Convolutional Feature Learning and Cascaded Classification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2016}}