Abstract: Supervised object detection and semantic segmentation require object or even pixel level annotations. When there exist image level labels only, it is challenging for weakly supervised algorithms to achieve accurate predictions. The accuracy achieved by top weakly supervised algorithms is still significantly lower than their fully supervised counterparts. In this paper, we propose a novel weakly supervised curriculum learning pipeline for multi-label object recognition, detection and semantic segmentation. In this pipeline, we first obtain intermediate object localization and pixel labeling results for the training images, and then use such results to train task-specific deep networks in a fully supervised manner. The entire process consists of four stages, including object localization in the training images, filtering and fusing object instances, pixel labeling for the training images, and task-specific network training. To obtain clean object instances in the training images, we propose a novel algorithm for filtering, fusing and classifying object instances collected from multiple solution mechanisms. In this algorithm, we incorporate both metric learning and density-based clustering to filter detected object instances. Experiments show that our weakly supervised pipeline achieves state-of-the-art results in multi-label image classification as well as weakly supervised object detection and very competitive results in weakly supervised semantic segmentation on MS-COCO, PASCAL VOC 2007 and PASCAL VOC 2012.

@InProceedings{Ge_2018_CVPR,author = {Ge, Weifeng and Yang, Sibei and Yu, Yizhou},title = {Multi-Evidence Filtering and Fusion for Multi-Label Classification, Object Detection and Semantic Segmentation Based on Weakly Supervised Learning},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Yue Cao, Bin Liu, Mingsheng Long, Jianmin Wang

Abstract: Deep learning to hash improves image retrieval performance by end-to-end representation learning and hash coding from training data with pairwise similarity information. Subject to the scarcity of similarity information that is often expensive to collect for many application domains, existing deep learning to hash methods may overfit the training data and result in substantial loss of retrieval quality. This paper presents HashGAN, a novel architecture for deep learning to hash, which learns compact binary hash codes from both real images and diverse images synthesized by generative models. The main idea is to augment the training data with nearly real images synthesized from a new Pair Conditional Wasserstein GAN (PC-WGAN) conditioned on the pairwise similarity information. Extensive experiments demonstrate that HashGAN can generate high-quality binary hash codes and yield state-of-the-art image retrieval performance on three benchmarks, NUS-WIDE, CIFAR-10, and MS-COCO.

@InProceedings{Cao_2018_CVPR,author = {Cao, Yue and Liu, Bin and Long, Mingsheng and Wang, Jianmin},title = {HashGAN: Deep Learning to Hash With Pair Conditional Wasserstein GAN},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Fang Wan, Pengxu Wei, Jianbin Jiao, Zhenjun Han, Qixiang Ye

Abstract: Weakly supervised object detection is a challenging task when provided with image category supervision but required to learn, at the same time, object locations and object detectors. The inconsistency between the weak supervision and learning objectives introduces randomness to object locations and ambiguity to detectors. In this paper, a min-entropy latent model (MELM) is proposed for weakly supervised object detection. Min-entropy is used as a metric to measure the randomness of object localization during learning, as well as serving as a model to learn object locations. It aims to principally reduce the variance of positive instances and alleviate the ambiguity of detectors. MELM is deployed as two sub-models, which respectively discovers and localizes objects by minimizing the global and local entropy. MELM is unified with feature learning and optimized with a recurrent learning algorithm, which progressively transfers the weak supervision to object locations. Experiments demonstrate that MELM significantly improves the performance of weakly supervised detection, weakly supervised localization, and image classification, against the state-of-the-art approaches.

@InProceedings{Wan_2018_CVPR,author = {Wan, Fang and Wei, Pengxu and Jiao, Jianbin and Han, Zhenjun and Ye, Qixiang},title = {Min-Entropy Latent Model for Weakly Supervised Object Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L. Berg

Abstract: In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit,  we propose to decompose them into three modular components related to subject appearance, location, and relationship to other objects. This allows us to flexibly adapt to expressions containing different types of information in an end-to-end framework. In our model, which we call the Modular Attention Network (MAttNet), two types of attention are utilized: language-based attention that learns the module weights as well as the word/phrase attention that each module should focus on; and visual attention that allows the subject and relationship modules to focus on relevant image components.  Module weights combine scores from all three modules dynamically to output an overall score. Experiments show that MAttNet outperforms previous state-of-the-art methods by a large margin on both bounding-box-level and pixel-level comprehension tasks.

@InProceedings{Yu_2018_CVPR,author = {Yu, Licheng and Lin, Zhe and Shen, Xiaohui and Yang, Jimei and Lu, Xin and Bansal, Mohit and Berg, Tamara L.},title = {MAttNet: Modular Attention Network for Referring Expression Comprehension},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, Xiaodong He

Abstract: In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation.  With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different sub-regions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

@InProceedings{Xu_2018_CVPR,author = {Xu, Tao and Zhang, Pengchuan and Huang, Qiuyuan and Zhang, Han and Gan, Zhe and Huang, Xiaolei and He, Xiaodong},title = {AttnGAN: Fine-Grained Text to Image Generation With Attentional Generative Adversarial Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, Thomas S. Huang

Abstract: In this work, we propose Adversarial Complementary Learning (ACoL) to automatically localize integral objects of semantic interest with weak supervision. We first mathematically prove that class localization maps can be obtained by directly selecting the class-specific feature maps of the last convolutional layer, which paves a simple way to identify object regions. We then present a simple network architecture including two parallel-classifiers for object localization. Specifically, we leverage one classification branch to dynamically localize some discriminative object regions during the forward pass. Although it is usually responsive to sparse parts of the target objects, this classifier can drive the counterpart classifier to discover new and complementary object regions by erasing its discovered regions from the feature maps. With such an adversarial learning, the two parallel-classifiers are forced to leverage complementary object regions for classification and can finally generate integral object localization together. The merits of ACoL are mainly two-fold: 1) it can be trained in an end-to-end manner; 2) dynamically erasing enables the counterpart classifier to discover complementary object regions more effectively. We demonstrate the superiority of our ACoL approach in a variety of experiments. In particular, the Top-1 localization error rate on the ILSVRC dataset is 45.14%, which is the new state-of-the-art.

@InProceedings{Zhang_2018_CVPR,author = {Zhang, Xiaolin and Wei, Yunchao and Feng, Jiashi and Yang, Yi and Huang, Thomas S.},title = {Adversarial Complementary Learning for Weakly Supervised Object Localization},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Weixiang Hong, Zhenzhen Wang, Ming Yang, Junsong Yuan

Abstract: In recent years, deep neural nets have triumphed over many computer vision problems, including semantic segmentation, which is a critical task in emerging autonomous driving and medical image diagnostics applications. In general, training deep neural nets requires a humongous amount of labeled data, which is laborious and costly to collect and annotate. Recent advances in computer graphics shed light on utilizing photo-realistic synthetic data with computer generated annotations to train neural nets. Nevertheless, the domain mismatch between real images and synthetic ones is the major challenge against harnessing the generated data and labels. In this paper, we propose a principled way to conduct structured domain adaption for semantic segmentation, i.e., integrating GAN into the FCN framework to mitigate the gap between source and target domains. Specifically, we learn a conditional generator to transform features of synthetic images to real-image like features, and a discriminator to distinguish them. For each training batch, the conditional generator and the discriminator compete against each other so that the generator learns to produce real-image like features to fool the discriminator; afterwards, the FCN parameters are updated to accommodate the changes of GAN. In experiments, without using labels of real image data, our method significantly outperforms the baselines as well as state-of-the-art methods by 12% ∼ 20% mean IoU on the Cityscapes dataset.

@InProceedings{Hong_2018_CVPR,author = {Hong, Weixiang and Wang, Zhenzhen and Yang, Ming and Yuan, Junsong},title = {Conditional Generative Adversarial Network for Structured Domain Adaptation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, Jinsong Su

Abstract: Most image captioning models focus on one-line (single image) captioning, where the correlations like relevance and diversity among group images (e.g., within the same album or event) are simply neglected, resulting in less accurate and diverse captions. Recent works mainly consider imposing the diversity during the online inference only, which neglect the correlation among visual structures in offline training. In this paper, we propose a novel group-based image captioning scheme (termed GroupCap), which jointly models the structured relevance and diversity among visual contents of group images towards an optimal collaborative captioning. In particular, we first propose a visual tree parser (VP-Tree) to construct the structured semantic correlations within individual images. Then, the relevance and diversity among images are well modeled by exploiting the correlations among their tree structures. Finally, such correlations are modeled as constraints and sent into the LSTM-based captioning generator. In offline optimization, we adopt an end-to-end formulation, which jointly trains the visual tree parser, the structured relevance and diversity constraints, as well as the LSTM based captioning model. To facilitate quantitative evaluation, we further release two group captioning datasets derived from the MS-COCO benchmark, serving as the first of their kind. Quantitative results show that the proposed GroupCap model outperforms the state-of-the-art and alternative approaches, which can generate much more accurate and discriminative captions under various evaluation metrics.

@InProceedings{Chen_2018_CVPR,author = {Chen, Fuhai and Ji, Rongrong and Sun, Xiaoshuai and Wu, Yongjian and Su, Jinsong},title = {GroupCap: Group-Based Image Captioning With Structured Relevance and Diversity Constraints},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Xiang Wang, Shaodi You, Xi Li, Huimin Ma

Abstract: Weakly-supervised semantic segmentation under image tags supervision is a challenging task as it directly associates high-level semantic to low-level appearance. To bridge this gap, in this paper, we propose an iterative bottom-up and top-down framework which alternatively expands object regions and optimizes segmentation network. We start from initial localization produced by classification networks. While classification networks are only responsive to small and coarse discriminative object regions, we argue that, these regions contain significant common features about objects. So in the bottom-up step, we mine common object features from the initial localization and expand object regions with the mined features. To supplement non-discriminative regions, saliency maps are then considered under Bayesian framework to refine the object regions. Then in the top-down step, the refined object regions are used as supervision to train the segmentation network and to predict object masks. These object masks provide more accurate localization and contain more regions of object. Further, we take these object masks as initial localization and mine common object features from them. These processes are conducted iteratively to progressively produce fine object masks and optimize segmentation networks. Experimental results on Pascal VOC 2012 dataset demonstrate that the proposed method outperforms previous state-of-the-art methods by a large margin.

@InProceedings{Wang_2018_CVPR,author = {Wang, Xiang and You, Shaodi and Li, Xi and Ma, Huimin},title = {Weakly-Supervised Semantic Segmentation by Iteratively Mining Common Object Features},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Tong Shen, Guosheng Lin, Chunhua Shen, Ian Reid

Abstract: Fully supervised methods for semantic segmentation require pixel-level class masks to train, the creation of which are expensive in terms of manual labour and time. In this work, we focus on weak supervision, developing a method for training a high-quality pixel-level classifier for semantic segmentation, using only image-level class labels as the provided ground-truth. Our method is formulated as a two-stage approach in which we first aim to create accurate pixel-level masks for the training images via a bootstrapping process, and then use these now-accurately segmented images as a proxy ground-truth in a more standard supervised setting. The key driver for our work is that in the target dataset we typically have reliable ground-truth image-level labels, while data crawled from the web may have unreliable labels, but can be filtered to comprise only easy images to segment, therefore having reliable boundaries. These two forms of information are complementary and we use this observation to build a novel bi-directional transfer learning. This framework transfers knowledge between two domains, target domain and web domain, bootstrapping the performance of weakly supervised semantic segmentation. Conducting experiments on the popular benchmark dataset PASCAL VOC 2012 based on both a VGG16 network and on ResNet50, we reach state-of-the-art performance with scores of 60.2% IoU and 63.9% IoU respectively.

@InProceedings{Shen_2018_CVPR,author = {Shen, Tong and Lin, Guosheng and Shen, Chunhua and Reid, Ian},title = {Bootstrapping the Performance of Webly Supervised Semantic Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Zhishuai Zhang, Cihang Xie, Jianyu Wang, Lingxi Xie, Alan L. Yuille

Abstract: In this paper, we study the task of detecting semantic parts of an object, e.g., a wheel of a car, under partial occlusion. We propose that all models should be trained without seeing occlusions while being able to transfer the learned knowledge to deal with occlusions. This setting alleviates the difficulty in collecting an exponentially large dataset to cover occlusion patterns and is more essential. In this scenario, the proposal-based deep networks, like RCNN-series, often produce unsatisfactory results, because both the proposal extraction and classification stages may be confused by the irrelevant occluders. To address this, [25] proposed a voting mechanism that combines multiple local visual cues to detect semantic parts. The semantic parts can still be detected even though some visual cues are missing due to occlusions. However, this method is manually-designed, thus is hard to be optimized in an end-to-end manner.  In this paper, we present DeepVoting, which incorporates the robustness shown by [25] into a deep network, so that the whole pipeline can be jointly optimized. Specifically, it adds two layers after the intermediate features of a deep network,  e.g., the pool-4 layer of VGGNet. The first layer extracts the evidence of local visual cues, and the second layer performs a voting mechanism by utilizing the spatial relationship between visual cues and semantic parts. We also propose an improved version DeepVoting+ by learning visual cues from context outside objects. In experiments, DeepVoting achieves significantly better performance than several baseline methods, including Faster-RCNN, for semantic part detection under occlusion. In addition, DeepVoting enjoys explainability as the detection results can be diagnosed via looking up the voting cues.

@InProceedings{Zhang_2018_CVPR,author = {Zhang, Zhishuai and Xie, Cihang and Wang, Jianyu and Xie, Lingxi and Yuille, Alan L.},title = {DeepVoting: A Robust and Explainable Deep Network for Semantic Part Detection Under Partial Occlusion},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Fangfang Wang, Liming Zhao, Xi Li, Xinchao Wang, Dacheng Tao

Abstract: Localizing text in the wild is challenging in the situations of complicated geometric layout of the targets like random orientation and large aspect ratio. In this paper, we propose a geometry-aware modeling approach tailored for scene text representation with an end-to-end learning scheme. In our approach, a novel Instance Transformation Network (ITN) is presented to learn the geometry-aware representation encoding the unique geometric configurations of scene text instances with in-network transformation embedding, resulting in a robust and elegant framework to detect words or text lines at one pass. An end-to-end multi-task learning strategy with transformation regression, text/non-text classification and coordinate regression is adopted in the ITN. Experiments on the benchmark datasets demonstrate the effectiveness of the proposed approach in detecting scene text in various geometric configurations.

@InProceedings{Wang_2018_CVPR,author = {Wang, Fangfang and Zhao, Liming and Li, Xi and Wang, Xinchao and Tao, Dacheng},title = {Geometry-Aware Scene Text Detection With Instance Transformation Network},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Shuyang Sun, Zhanghui Kuang, Lu Sheng, Wanli Ouyang, Wei Zhang

Abstract: Motion representation plays a vital role in human action recognition in videos. In this study, we introduce a novel compact motion representation for video action recognition, named Optical Flow guided Feature (OFF), which enables the network to distill temporal information through a fast and robust approach. The OFF is derived from the definition of optical flow and is orthogonal to the optical flow. The derivation also provides theoretical support for using the difference between two frames. By directly calculating pixel-wise spatio-temporal gradients of the deep feature maps, the OFF could be embedded in any existing CNN based video action recognition framework with only a slight additional cost. It enables the CNN to extract spatio-temporal information, especially the temporal information between frames simultaneously. This simple but powerful idea is validated by experimental results. The network with OFF fed only by RGB inputs achieves a competitive accuracy of 93.3% on UCF-101, which is comparable with the result obtained by two streams (RGB and optical flow), but is 15 times faster in speed. Experimental results also show that OFF is complementary to other motion modalities such as optical flow. When the proposed method is plugged into the state-of-the-art video action recognition framework, it has 96.0% and 74.2% accuracy on UCF-101 and HMDB-51 respectively. The code for this project is available at: https://github.com/kevin-ssy/Optical-Flow-Guided-Feature

@InProceedings{Sun_2018_CVPR,author = {Sun, Shuyang and Kuang, Zhanghui and Sheng, Lu and Ouyang, Wanli and Zhang, Wei},title = {Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Ping Hu, Gang Wang, Xiangfei Kong, Jason Kuen, Yap-Peng Tan

Abstract: Deep CNNs have achieved superior performance in many tasks of computer vision and image understanding. However, it is still difficult to effectively apply deep CNNs to video object segmentation(VOS) since treating video frames as separate and static will lose the information hidden in motion. To tackle this problem, we propose a Motion-guided Cascaded Refinement Network for VOS. By assuming the object motion is normally different from the background motion, for a video frame we first apply an active contour model on optical flow to coarsely segment objects of interest. Then, the proposed Cascaded Refinement Network(CRN) takes the coarse segmentation as guidance to generate an accurate segmentation of full resolution. In this way, the motion information and the deep CNNs can well complement each other to accurately segment objects from video frames. Furthermore, in CRN we introduce a Single-channel Residual Attention Module to incorporate the coarse segmentation map as attention, making our network effective and efficient in both training and testing. We perform experiments on the popular benchmarks and the results show that our method achieves state-of-the-art performance at a much faster speed.

@InProceedings{Hu_2018_CVPR,author = {Hu, Ping and Wang, Gang and Kong, Xiangfei and Kuen, Jason and Tan, Yap-Peng},title = {Motion-Guided Cascaded Refinement Network for Video Object Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Sangho Lee, Jinyoung Sung, Youngjae Yu, Gunhee Kim

Abstract: We address the problem of story-based temporal summarization of long 360° videos. We propose a novel memory network model named Past-Future Memory Network (PFMN), in which we first compute the scores of 81 normal field of view (NFOV) region proposals cropped from the input 360° video, and then recover a latent, collective summary using the network with two external memories that store the embeddings of previously selected subshots and future candidate subshots. Our major contributions are two-fold. First, our work is the first to address story-based temporal summarization of 360° videos. Second, our model is the first attempt to leverage memory networks for video summarization tasks. For evaluation, we perform three sets of experiments. First, we investigate the view selection capability of our model on the Pano2Vid dataset. Second, we evaluate the temporal summarization with a newly collected 360° video dataset. Finally, we experiment our model's performance in another domain, with image-based storytelling VIST dataset. We verify that our model achieves state-of-the-art performance on all the tasks.

@InProceedings{Lee_2018_CVPR,author = {Lee, Sangho and Sung, Jinyoung and Yu, Youngjae and Kim, Gunhee},title = {A Memory Network Approach for Story-Based Temporal Summarization of 360° Videos},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, Min Sun

Abstract: Automatic saliency prediction in 360° videos is critical for viewpoint guidance applications (e.g., Facebook 360 Guide). We propose a spatial-temporal network which is (1) unsupervisedly trained and (2) tailor-made for 360° viewing sphere. Note that most existing methods are less scalable since they rely on annotated saliency map for training. Most importantly, they convert 360° sphere to 2D images (e.g., a single equirectangular image or multiple separate Normal Field-of-View (NFoV) images) which introduces distortion and image boundaries. In contrast, we propose a simple and effective Cube Padding (CP) technique as follows. Firstly, we render the 360° view on six faces of a cube using perspective projection. Thus, it introduces very little distortion. Then, we concatenate all six faces while utilizing the connectivity between faces on the cube for image padding (i.e., Cube Padding) in convolution, pooling, convolutional LSTM layers. In this way, PC introduces no image boundary while being applicable to almost all Convolutional Neural Network (CNN) structures. To evaluate our method, we propose Wild-360, a new 360° video saliency dataset, containing challenging videos with saliency heatmap annotations. In experiments, our method outperforms all baseline methods in both speed and quality.

@InProceedings{Cheng_2018_CVPR,author = {Cheng, Hsien-Tzu and Chao, Chun-Hung and Dong, Jin-Dong and Wen, Hao-Kai and Liu, Tyng-Luh and Sun, Min},title = {Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Limin Wang, Wei Li, Wen Li, Luc Van Gool

Abstract: Spatiotemporal feature learning in videos is a fundamental problem in computer vision. This paper presents a new architecture, termed as Appearance-and-Relation Network (ARTNet), to learn video representation in an end-to-end manner. ARTNets are constructed by stacking multiple generic building blocks, called as SMART, whose goal is to simultaneously model appearance and relation from RGB input in a separate and explicit manner. Specifically, SMART blocks decouple the spatiotemporal learning module into an appearance branch for spatial modeling and a relation branch for temporal modeling. The appearance branch is implemented based on the linear combination of pixels or filter responses in each frame, while the relation branch is designed based on the multiplicative interactions between pixels or filter responses across multiple frames. We perform experiments on three action recognition benchmarks: Kinetics, UCF101, and HMDB51, demonstrating that SMART blocks obtain an evident improvement over 3D convolutions for spatiotemporal feature learning. Under the same training setting, ARTNets achieve superior performance on these three datasets to the existing state-of-the-art methods.

@InProceedings{Wang_2018_CVPR,author = {Wang, Limin and Li, Wei and Li, Wen and Van Gool, Luc},title = {Appearance-and-Relation Networks for Video Classification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Excitation Backprop for RNNs

Sarah Adel Bargal, Andrea Zunino, Donghyun Kim, Jianming Zhang, Vittorio Murino, Stan Sclaroff

Abstract: Deep models are state-of-the-art or many vision tasks including video action recognition and video captioning. Models are trained to caption or classify activity in videos, but little is known about the evidence used to make such decisions. Grounding decisions made by deep networks has been studied in spatial visual content, giving more insight into model predictions for images. However, such studies are relatively lacking for models of spatiotemporal visual content - videos. In this work, we devise a formulation that simultaneously grounds evidence in space and time, in a single pass, using top-down saliency. We visualize the spatiotemporal cues that contribute to a deep model's classification/captioning output using the model's internal representation. Based on these spatiotemporal cues, we are able to localize segments within a video that correspond with a specific action, or phrase from a caption, without explicitly optimizing/training for these tasks.

@InProceedings{Bargal_2018_CVPR,author = {Bargal, Sarah Adel and Zunino, Andrea and Kim, Donghyun and Zhang, Jianming and Murino, Vittorio and Sclaroff, Stan},title = {Excitation Backprop for RNNs},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: Learning based temporal action localization methods require vast amounts of training data. However, such large-scale video datasets, which are expected to capture the dynamics of every action category, are not only very expensive to acquire but are also not practical simply because there exists an uncountable number of action classes. This poses a critical restriction to the current methods when the training samples are few and rare (e.g. when the target action classes are not present in the current publicly available datasets). To address this challenge, we conceptualize a new example-based action detection problem where only a few examples are provided, and the goal is to find the occurrences of these examples in an untrimmed video sequence. Towards this objective, we introduce a novel one-shot action localization method that alleviates the need for large amounts of training samples. Our solution adopts the one-shot learning technique of Matching Network and utilizes correlations to mine and localize actions of previously unseen classes. We evaluate our one-shot action localization method on the THUMOS14 and ActivityNet datasets, of which we modified the configuration to fit our one-shot problem setup.

@InProceedings{Yang_2018_CVPR,author = {Yang, Hongtao and He, Xuming and Porikli, Fatih},title = {One-Shot Action Localization by Learning Sequence Matching Network},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Structure Preserving Video Prediction

Jingwei Xu, Bingbing Ni, Zefan Li, Shuo Cheng, Xiaokang Yang

Abstract: Despite recent emergence of adversarial based methods for video prediction, existing algorithms often produce unsatisfied results in image regions with rich structural information (i.e., object boundary) and detailed motion (i.e., articulated body movement). To this end, we present a structure preserving video prediction framework to explicitly address above issues and enhance video prediction quality. On one hand, our framework contains a two-stream generation architecture which deals with high frequency video content (i.e., detailed object or articulated motion structure) and low frequency video content (i.e., location or moving directions) in two separate streams. On the other hand, we propose a RNN structure for video prediction, which employs temporal-adaptive convolutional kernels to capture time-varying motion patterns as well as the tiny object within a scene. Extensive experiments on diverse scene, ranging from human motion to semantic layout prediction, demonstrate the effectiveness of the proposed video prediction approach.

@InProceedings{Xu_2018_CVPR,author = {Xu, Jingwei and Ni, Bingbing and Li, Zefan and Cheng, Shuo and Yang, Xiaokang},title = {Structure Preserving Video Prediction},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Yicheng Wang, Zhenzhong Chen, Feng Wu, Gang Wang

Abstract: In this paper, a novel deep architecture named BraidNet is proposed for person re-identification. BraidNet has a specially designed WConv layer, and the cascaded WConv structure learns to extract the comparison features of two images, which are robust to misalignments and color differences across cameras. Furthermore, a Channel Scaling layer is designed to optimize the scaling factor of each input channel, which helps mitigate the zero gradient problem in the training phase. To solve the problem of imbalanced volume of negative and positive training samples, a Sample Rate Learning strategy is proposed to adaptively update the ratio between positive and negative samples in each batch. Experiments conducted on CUHK03-Detected, CUHK03-Labeled, CUHK01, Market-1501 and DukeMTMC-reID datasets demonstrate that our method achieves competitive performance when compared to state-of-the-art methods.

@InProceedings{Wang_2018_CVPR,author = {Wang, Yicheng and Chen, Zhenzhong and Wu, Feng and Wang, Gang},title = {Person Re-Identification With Cascaded Pairwise Convolutions},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Aleksandar Zlateski, Ronnachai Jaroensri, Prafull Sharma, Frédo Durand

Abstract: Convolutional networks (ConvNets) have become the dominant approach to semantic image segmentation. Producing accurate, pixel--level labels required for this task is a tedious and time consuming process; however, producing approximate, coarse labels could take only a fraction of the time and effort.  We investigate the relationship between the quality of labels and the performance of ConvNets for semantic segmentation.  We create a very large synthetic dataset with perfectly labeled street view scenes.  From these perfect labels, we synthetically coarsen labels with different qualities and estimate human--hours required for producing them.  We perform a series of experiments by training ConvNets with a varying number of training images and label quality.  We found that the performance of ConvNets mostly depends on the time spent creating the training labels. That is, a larger coarsely--annotated dataset can yield the same performance as a smaller finely--annotated one.  Furthermore, fine--tuning coarsely pre--trained ConvNets with few finely-annotated labels can yield comparable or superior performance to training it with a large amount of finely-annotated labels alone, at a fraction of the labeling cost. We demonstrate that our result is also valid for different network architectures, and various object classes in an urban scene.

@InProceedings{Zlateski_2018_CVPR,author = {Zlateski, Aleksandar and Jaroensri, Ronnachai and Sharma, Prafull and Durand, Frédo},title = {On the Importance of Label Quality for Semantic Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Xiaobin Chang, Tao Xiang, Timothy M. Hospedales

Abstract: Recently the widely used multi-view learning model, Canonical Correlation Analysis (CCA) has been generalised to the non-linear setting via deep neural networks. Existing deep CCA models typically first decorrelate the feature dimensions of each view before the different views are maximally correlated in a common latent space. This feature decorrelation is achieved by enforcing an exact decorrelation constraint; these models are thus computationally expensive due to the matrix inversion or SVD operations required for exact decorrelation at each training iteration. Furthermore, the decorrelation step is often separated from the gradient descent based optimisation, resulting in sub-optimal solutions. We propose a novel deep CCA model Soft CCA to overcome these problems. Specifically, exact decorrelation is replaced by soft decorrelation via a mini-batch based Stochastic Decorrelation Loss (SDL) to be optimised jointly with the other training objectives. Extensive experiments show that the proposed soft CCA is more effective and efficient than existing deep CCA models. In addition, our SDL loss can be applied to other deep models beyond multi-view learning, and obtains superior performance compared to existing decorrelation losses.

@InProceedings{Chang_2018_CVPR,author = {Chang, Xiaobin and Xiang, Tao and Hospedales, Timothy M.},title = {Scalable and Effective Deep CCA via Soft Decorrelation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Lanqing Hu, Meina Kan, Shiguang Shan, Xilin Chen

Abstract: Domain adaptation attempts to transfer the knowledge obtained from the source domain to the target domain, i.e., the domain where the testing data are. The main challenge lies in the distribution discrepancy between source and target domain. Most existing works endeavor to learn domain invariant representation usually by minimizing a distribution distance, e.g., MMD and the discriminator in the recently proposed generative adversarial network (GAN). Following the similar idea of GAN, this work proposes a novel GAN architecture with duplex adversarial discriminators (referred to as DupGAN), which can achieve domain-invariant representation and domain transformation. Specifically, our proposed network consists of three parts, an encoder, a generator and two discriminators. The encoder embeds samples from both domains into the latent representation, and the generator decodes the latent representation to both source and target domains respectively conditioned on a domain code, i.e., achieves domain transformation. The generator is pitted against duplex discriminators, one for source domain and the other for target, to ensure the reality of domain transformation, the latent representation domain invariant and the category information of it preserved as well. Our proposed work achieves the state-of-the-art performance on unsupervised domain adaptation of digit classification and object recognition.

@InProceedings{Hu_2018_CVPR,author = {Hu, Lanqing and Kan, Meina and Shan, Shiguang and Chen, Xilin},title = {Duplex Generative Adversarial Network for Unsupervised Domain Adaptation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Fan Bai, Zhanzhan Cheng, Yi Niu, Shiliang Pu, Shuigeng Zhou

Abstract: We consider the scene text recognition problem under the attention-based encoder-decoder framework, which is the state of the art. The existing methods usually employ a frame-wise maximal likelihood loss to optimize the models. When we train the model, the misalignment between the ground truth strings and the attention's output sequences of probability distribution, which is caused by missing or superfluous characters, will confuse and mislead the training process, and consequently make the training costly and degrade the recognition accuracy. To handle this problem, we propose a novel method called edit probability (EP) for scene text recognition. EP tries to effectively estimate the probability of generating a string from the output sequence of probability distribution conditioned on the input image, while considering the possible occurrences of missing/superfluous characters. The advantage lies in that the training process can focus on the missing, superfluous and unrecognized characters, and thus the impact of the misalignment problem can be alleviated or even overcome. We conduct extensive experiments on standard benchmarks, including the IIIT-5K, Street View Text and ICDAR datasets. Experimental results show that the EP can substantially boost scene text recognition performance.

@InProceedings{Bai_2018_CVPR,author = {Bai, Fan and Cheng, Zhanzhan and Niu, Yi and Pu, Shiliang and Zhou, Shuigeng},title = {Edit Probability for Scene Text Recognition},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Guo-Jun Qi, Liheng Zhang, Hao Hu, Marzieh Edraki, Jingdong Wang, Xian-Sheng Hua

Abstract: In this paper, we present a novel localized Generative Adversarial Net (GAN) to learn on the manifold of real data. Compared with the classic GAN that {em globally} parameterizes a manifold, the Localized GAN (LGAN) uses local coordinate charts to parameterize distinct local geometry of how data points can transform at different locations on the manifold. Specifically, around each point there exists a {em local} generator that can produce data following diverse patterns of transformations on the manifold.  The locality nature of LGAN enables local generators to adapt to and directly access the local geometry without need to invert the generator in a global GAN. Furthermore, it can prevent the manifold from being locally collapsed to a dimensionally deficient tangent subspace by imposing an orthonormality prior between tangents. This provides a geometric approach to alleviating mode collapse at least locally on the manifold by imposing independence between data transformations in different tangent directions. We will also demonstrate the LGAN can be applied to train a robust classifier that prefers locally consistent classification decisions on the manifold, and the resultant regularizer is closely related with the Laplace-Beltrami operator. Our experiments show that the proposed LGANs can not only produce diverse image transformations, but also deliver superior classification performances.

@InProceedings{Qi_2018_CVPR,author = {Qi, Guo-Jun and Zhang, Liheng and Hu, Hao and Edraki, Marzieh and Wang, Jingdong and Hua, Xian-Sheng},title = {Global Versus Localized Generative Adversarial Nets},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz

Abstract: Visual signals in a video can be divided into content and motion. While content specifies which objects are in the video, motion describes their dynamics. Based on this prior, we propose the Motion and Content decomposed Generative Adversarial Network (MoCoGAN) framework for video generation. The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is  kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utilizing both image and video discriminators. Extensive experimental results on several challenging datasets with qualitative and quantitative comparison to the state-of-the-art approaches, verify effectiveness of the proposed framework. In addition, we show that MoCoGAN allows one to generate videos with same content but different motion as well as videos with different content and same motion. Our code is available at https://github.com/sergeytulyakov/mocogan.

@InProceedings{Tulyakov_2018_CVPR,author = {Tulyakov, Sergey and Liu, Ming-Yu and Yang, Xiaodong and Kautz, Jan},title = {MoCoGAN: Decomposing Motion and Content for Video Generation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Bowen Pan, Wuwei Lin, Xiaolin Fang, Chaoqin Huang, Bolei Zhou, Cewu Lu

Abstract: Deep convolutional neural networks (CNNs) have made impressive progress in many video recognition tasks such as video pose estimation and video object detection. However, running CNN inference on video requires numerous computation and is usually slow. In this work, we propose a framework called Recurrent Residual Module (RRM) to accelerate the CNN inference for video recognition tasks. This framework has a novel design of using the similarity of the intermediate feature maps of two consecutive frames to largely reduce the redundant computation. One unique property of the proposed method compared to previous work is that feature maps of each frame are precisely computed. The experiments show that, while maintaining the similar recognition performance, our RRM yields averagely 2× acceleration on the commonly used CNNs such as AlexNet, ResNet, deep compression model (thus 8−12× faster than the original dense models on the efﬁcient inference engine), and impressively 9× acceleration on some binary networks such as XNOR-Nets (thus 500× faster than the original model). We further verify the effectiveness of the RRM on speeding CNNs for video pose estimation and video object detection.

@InProceedings{Pan_2018_CVPR,author = {Pan, Bowen and Lin, Wuwei and Fang, Xiaolin and Huang, Chaoqin and Zhou, Bolei and Lu, Cewu},title = {Recurrent Residual Module for Fast Inference in Videos},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher Pal, Jan Kautz

Abstract: We present two techniques to improve landmark localization in images from partially annotated datasets. Our primary goal is to leverage the common situation where precise landmark locations are only provided for a small data subset, but where class labels for classification or regression tasks related to the landmarks are more abundantly available.  First, we propose the framework of sequential multitasking and explore it here through an architecture for landmark localization where training with class labels acts as an auxiliary signal to guide the landmark localization on unlabeled data. A key aspect of our approach is that errors can be backpropagated through a complete landmark localization model. Second, we propose and explore an unsupervised learning technique for landmark localization based on having a model predict equivariant landmarks with respect to transformations applied to the image. We show that these techniques, improve landmark prediction considerably and can learn effective detectors even when only a small fraction of the dataset has landmark labels. We present results on two toy datasets and four real datasets, with hands and faces, and report new state-of-the-art on two datasets in the wild, e.g. with only 5% of labeled images we outperform previous state-of-the-art trained on the AFLW dataset.

@InProceedings{Honari_2018_CVPR,author = {Honari, Sina and Molchanov, Pavlo and Tyree, Stephen and Vincent, Pascal and Pal, Christopher and Kautz, Jan},title = {Improving Landmark Localization With Semi-Supervised Learning},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: Paucity of large curated hand labeled training data forms a major bottleneck in the deployment of machine learning models in computer vision and other fields. Recent work (Data Programming) has shown how distant supervision signals in the form of labeling functions can be used to obtain labels for given data in near-constant time. In this work, we present Adversarial Data Programming (ADP), which presents an adversarial methodology to generate data as well as a curated aggregated label, given a set of weak labeling functions. We validated our method on the MNIST, Fashion MNIST, CIFAR 10 and SVHN datasets, and it outperformed many state-of-the-art models. We conducted extensive experiments to study its usefulness, as well as showed how the proposed ADP framework can be used for transfer learning as well as multitask learning, where data from two domains are generated simultaneously using the framework along with the label information. Our future work will involve understanding the theoretical implications of this new framework from a game-theoretic perspective, as well as explore the performance of the method on more complex datasets.

@InProceedings{Pal_2018_CVPR,author = {Pal, Arghya and Balasubramanian, Vineeth N.},title = {Adversarial Data Programming: Using GANs to Relax the Bottleneck of Curated Labeled Data},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Tobias Plötz, Anne S. Wannenwetsch, Stefan Roth

Abstract: Variational inference has experienced a recent surge in popularity owing to stochastic approaches, which have yielded practical tools for a wide range of model classes. A key benefit is that stochastic variational inference obviates the tedious process of deriving analytical expressions for closed-form variable updates. Instead, one simply needs to derive the gradient of the log-posterior, which is often much easier. Yet for certain model classes, the log-posterior itself is difficult to optimize using standard gradient techniques. One such example are random field models, where optimization based on gradient linearization has proven popular, since it speeds up convergence significantly and can avoid poor local optima. In this paper we propose stochastic variational inference with gradient linearization (SVIGL). It is similarly convenient as standard stochastic variational inference - all that is required is a local linearization of the energy gradient. Its benefit over stochastic variational inference with conventional gradient methods is a clear improvement in convergence speed, while yielding comparable or even better variational approximations in terms of KL divergence. We demonstrate the benefits of SVIGL in three applications: Optical flow estimation, Poisson-Gaussian denoising, and 3D surface reconstruction.

@InProceedings{Plötz_2018_CVPR,author = {Plötz, Tobias and Wannenwetsch, Anne S. and Roth, Stefan},title = {Stochastic Variational Inference With Gradient Linearization},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Chung-Wei Lee, Wei Fang, Chih-Kuan Yeh, Yu-Chiang Frank Wang

Abstract: In this paper, we propose a novel deep learning architecture for multi-label zero-shot learning (ML-ZSL), which is able to predict multiple unseen class labels for each input instance. Inspired by the way humans utilize semantic knowledge between objects of interests, we propose a framework that incorporates knowledge graphs for describing the relationships between multiple labels. Our model learns an information propagation mechanism from the semantic label space, which can be applied to model the interdependencies between seen and unseen class labels. With such investigation of structured knowledge graphs for visual reasoning, we show that our model can be applied for solving multi-label classification and ML-ZSL tasks. Compared to state-of-the-art approaches, comparable or improved performances can be achieved by our method.

@InProceedings{Lee_2018_CVPR,author = {Lee, Chung-Wei and Fang, Wei and Yeh, Chih-Kuan and Wang, Yu-Chiang Frank},title = {Multi-Label Zero-Shot Learning With Structured Knowledge Graphs},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Ariel Gordon, Elad Eban, Ofir Nachum, Bo Chen, Hao Wu, Tien-Ju Yang, Edward Choi

Abstract: We present MorphNet, an approach to automate the design of neural  network structures. MorphNet iteratively shrinks and expands a network, shrinking via a resource-weighted sparsifying regularizer on  activations and expanding via a uniform multiplicative factor on all layers. In contrast to previous  approaches, our method is scalable to large networks, adaptable to specific resource constraints (e.g.   the number of floating-point operations per inference), and capable of increasing the network’s performance. When applied to standard network architectures on a  wide variety of datasets, our  approach discovers novel structures  in each domain, obtaining higher  performance while respecting the resource constraint.

@InProceedings{Gordon_2018_CVPR,author = {Gordon, Ariel and Eban, Elad and Nachum, Ofir and Chen, Bo and Wu, Hao and Yang, Tien-Ju and Choi, Edward},title = {MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Deep Adversarial Subspace Clustering

Abstract: Most existing subspace clustering methods hinge on self-expression of handcrafted representations and are unaware of potential clustering errors. Thus they  perform unsatisfactorily on real data with complex underlying subspaces. To solve this issue, we propose a novel deep adversarial subspace clustering (DASC) model, which learns more favorable sample representations by deep learning for subspace clustering, and more importantly introduces adversarial learning to supervise sample representation learning and subspace clustering. Specifically, DASC consists of a subspace clustering generator and a quality-verifying discriminator, which learn against each other. The generator produces subspace estimation and sample clustering. The discriminator  evaluates  current clustering performance by inspecting whether the re-sampled data from estimated subspaces have consistent subspace properties, and  supervises the generator to progressively improve subspace clustering. Experimental results on the handwritten recognition, face and object clustering tasks demonstrate the advantages of DASC over shallow and few deep subspace clustering models. Moreover, to our best knowledge, this is the first successful application of GAN-alike model for unsupervised subspace clustering, which also paves the way for deep learning to solve other unsupervised learning problems.

@InProceedings{Zhou_2018_CVPR,author = {Zhou, Pan and Hou, Yunqing and Feng, Jiashi},title = {Deep Adversarial Subspace Clustering},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Keze Wang, Xiaopeng Yan, Dongyu Zhang, Lei Zhang, Liang Lin

Abstract: Though quite challenging, leveraging large-scale unlabeled or partially labeled images in a cost-effective way has increasingly attracted interests for its great importance to computer vision. To tackle this problem, many Active Learning (AL) methods have been developed. However, these methods mainly define their sample selection criteria within a single image context, leading to the suboptimal robustness and impractical solution for large-scale object detection. In this paper, aiming to remedy the drawbacks of existing AL methods, we present a principled Self-supervised Sample Mining (SSM) process accounting for the real challenges in object detection. Specifically, our SSM process concentrates on automatically discovering and pseudo-labeling reliable region proposals for enhancing the object detector via the introduced cross image validation, i.e., pasting these proposals into different labeled images to comprehensively measure their values under different image contexts. By resorting to the SSM process, we propose a new AL framework for gradually incorporating unlabeled or partially labeled data into the model learning while minimizing the annotating effort of users. Extensive experiments on two public benchmarks clearly demonstrate our proposed framework can achieve the comparable performance to the state-of-the-art methods with significantly fewer annotations.

@InProceedings{Wang_2018_CVPR,author = {Wang, Keze and Yan, Xiaopeng and Zhang, Dongyu and Zhang, Lei and Lin, Liang},title = {Towards Human-Machine Cooperation: Self-Supervised Sample Mining for Object Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Emanuel Laude, Jan-Hendrik Lange, Jonas Schüpfer, Csaba Domokos, Laura Leal-Taixé, Frank R. Schmidt, Bjoern Andres, Daniel Cremers

Abstract: This paper introduces a novel algorithm for transductive inference in higher-order MRFs, where the unary energies are parameterized by a variable classifier. The considered task is posed as a joint optimization problem in the continuous classifier parameters and the discrete label variables. In contrast to prior approaches such as convex relaxations, we propose an advantageous decoupling of the objective function into discrete and continuous subproblems and a novel, efficient optimization method related to ADMM. This approach preserves integrality of the discrete label variables and guarantees global convergence to a critical point.  We demonstrate the advantages of our approach in several experiments including video object segmentation on the DAVIS data set and interactive image segmentation.

@InProceedings{Laude_2018_CVPR,author = {Laude, Emanuel and Lange, Jan-Hendrik and Schüpfer, Jonas and Domokos, Csaba and Leal-Taixé, Laura and Schmidt, Frank R. and Andres, Bjoern and Cremers, Daniel},title = {Discrete-Continuous ADMM for Transductive Inference in Higher-Order MRFs},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, Dawn Song

Abstract: Recent studies show that the state-of-the-art deep neural networks (DNNs) are vulnerable to adversarial examples, resulting from small-magnitude perturbations added to the input. Given that that emerging physical systems are using DNNs in safety-critical situations, adversarial examples could mislead these systems and cause dangerous situations. Therefore, understanding adversarial examples in the physical world is an important step towards developing resilient learning algorithms. We propose a general attack algorithm, Robust Physical Perturbations (RP 2 ), to generate robust visual adversarial perturbations under different physical conditions. Using the real-world case of road sign classification, we show that adversarial examples generated using RP 2 achieve high targeted misclassification rates against standard-architecture road sign classifiers in the physical world under various environmental conditions, including viewpoints. Due to the current lack of a standardized testing method, we propose a two-stage evaluation methodology for robust physical adversarial examples consisting of lab and field tests. Using this methodology, we evaluate the efficacy of physical adversarial manipulations on real objects. With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8% of the captured video frames obtained on a moving vehicle (field test) for the target classifier.

@InProceedings{Eykholt_2018_CVPR,author = {Eykholt, Kevin and Evtimov, Ivan and Fernandes, Earlence and Li, Bo and Rahmati, Amir and Xiao, Chaowei and Prakash, Atul and Kohno, Tadayoshi and Song, Dawn},title = {Robust Physical-World Attacks on Deep Learning Visual Classification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: Generating a novel image by manipulating two input images is an interesting research problem in the study of generative adversarial networks (GANs). We propose a new GAN-based network that generates a fusion image with the identity of input image x and the shape of input image y. Our network can simultaneously train on more than two image datasets in an unsupervised manner. We define an identity loss LI to catch the identity of image x and a shape loss LS to get the shape of y. In addition, we propose a novel training method called Min-Patch training to focus the generator on crucial parts of an image, rather than its entirety. We show qualitative results on the VGG Youtube Pose dataset , Eye dataset (MPIIGaze and UnityEyes), and the Photo–Sketch–Cartoon dataset.

@InProceedings{Joo_2018_CVPR,author = {Joo, DongGyu and Kim, Doyeon and Kim, Junmo},title = {Generating a Fusion Image: One's Identity and Another's Shape},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Learning to Promote Saliency Detectors

Yu Zeng, Huchuan Lu, Lihe Zhang, Mengyang Feng, Ali Borji

Abstract: The categories and appearance of salient objects vary from image to image, therefore, saliency detection is an image-specific task. Due to lack of large-scale saliency training data, using deep neural networks (DNNs) with pre-training is difficult to precisely capture the image-specific saliency cues. To solve this issue, we formulate a zero-shot learning problem to promote existing saliency detectors. Concretely, a DNN is trained as an embedding function to map pixels and the attributes of the salient/background regions of an image into the same metric space, in which an image-specific classifier is learned to classify the pixels. Since the image-specific task is performed by the classifier, the DNN embedding effectively plays the role of a general feature extractor. Compared with transferring the learning to a new recognition task using limited data, this formulation makes the DNN learn more effectively from small data. Extensive experiments on five data sets show that our method significantly improves accuracy of existing methods and compares favorably against state-of-the-art approaches.

@InProceedings{Zeng_2018_CVPR,author = {Zeng, Yu and Lu, Huchuan and Zhang, Lihe and Feng, Mengyang and Borji, Ali},title = {Learning to Promote Saliency Detectors},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Wei Han, Shiyu Chang, Ding Liu, Mo Yu, Michael Witbrock, Thomas S. Huang

Abstract: Advances in image super-resolution (SR) have recently benefited significantly from rapid developments in deep neural networks. Inspired by these recent discoveries, we note that many state-of-the-art deep SR architectures can be reformulated as a single-state recurrent neural network (RNN) with finite unfoldings. In this paper, we explore new structures for SR based on this compact RNN view, leading us to a dual-state design, the Dual-State Recurrent Network (DSRN). Compared to its single-state counterparts that op- erate at a fixed spatial resolution, DSRN exploits both low- resolution (LR) and high-resolution (HR) signals jointly. Recurrent signals are exchanged between these states in both directions (both LR to HR and HR to LR) via de- layed feedback. Extensive quantitative and qualitative eval- uations on benchmark datasets and on a recent challenge demonstrate that the proposed DSRN performs favorably against state-of-the-art algorithms in terms of both mem- ory consumption and predictive accuracy.

@InProceedings{Han_2018_CVPR,author = {Han, Wei and Chang, Shiyu and Liu, Ding and Yu, Mo and Witbrock, Michael and Huang, Thomas S.},title = {Image Super-Resolution via Dual-State Recurrent Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Muhammad Haris, Gregory Shakhnarovich, Norimichi Ukita

Abstract: The feed-forward architectures of recently proposed deep super-resolution networks learn representations of low-resolution inputs, and the non-linear mapping from those to high-resolution output. However, this approach does not fully address the mutual dependencies of low- and high-resolution images. We propose Deep Back-Projection Networks (DBPN), that exploit iterative up- and down-sampling layers, providing an error feedback mechanism for projection errors at each stage. We construct mutually-connected up- and down-sampling stages each of which represents different types of image degradation and high-resolution components. We show that extending this idea to allow concatenation of features across up- and down-sampling stages (Dense DBPN) allows us to reconstruct further improve super-resolution, yielding superior results and in particular establishing new state of the art results for large scaling factors such as 8x across multiple data sets.

@InProceedings{Haris_2018_CVPR,author = {Haris, Muhammad and Shakhnarovich, Gregory and Ukita, Norimichi},title = {Deep Back-Projection Networks for Super-Resolution},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: With the rise of misinformation spread via social media channels, enabled by the increasing automation and realism of image manipulation tools, image forensics is an increasingly relevant problem.  Classic image forensic methods leverage low-level cues such as metadata, sensor noise fingerprints, and others that are easily fooled when the image is re-encoded upon upload to facebook, etc.  This necessitates the use of higher-level physical and semantic cues that, once hard to estimate reliably in the wild, have become more effective due to the increasing power of computer vision.  In particular, we detect  manipulations introduced by artificial blurring of the image, which creates inconsistent photometric relationships between image intensity and various cues.  We achieve 98% accuracy on the most challenging cases in a new dataset of blur manipulations, where the blur is geometrically correct and consistent with the scene's physical arrangement.  Such manipulations are now easily generated, for instance, by smartphone cameras having hardware to measure depth, e.g. `Portrait Mode' of the iPhone7Plus. We also demonstrate good performance on a challenge dataset evaluating a wider range of manipulations in imagery representing `in the wild' conditions.

@InProceedings{Chen_2018_CVPR,author = {Chen, Can and McCloskey, Scott and Yu, Jingyi},title = {Focus Manipulation Detection via Photometric Histogram Analysis},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Nathan D. Cahill, Tyler L. Hayes, Renee T. Meinhold, John F. Hamilton

Abstract: The Normalized Cut (NCut) objective function, widely used in data clustering and image segmentation, quantifies the cost of graph partitioning in a way that biases clusters or segments that are balanced towards having lower values than unbalanced partitionings. However, this bias is so strong that it avoids any singleton partitions, even when vertices are very weakly connected to the rest of the graph. Motivated by the Buehler-Hein family of balanced cut costs, we propose the family of Compassionately Conservative Balanced (CCB) Cut costs, which are indexed by a parameter that can be used to strike a compromise between the desire to avoid too many singleton partitions and the notion that all partitions should be balanced. We show that CCB-Cut minimization can be relaxed into an orthogonally constrained $ell_{	au}$-minimization problem that coincides with the problem of computing Piecewise Flat Embeddings (PFE) for one particular index value, and we present an algorithm for solving the relaxed problem by iteratively minimizing a sequence of reweighted Rayleigh quotients (IRRQ). Using images from the BSDS500 database, we show that image segmentation based on CCB-Cut minimization provides better accuracy with respect to ground truth and greater variability in region size than NCut-based image segmentation.

@InProceedings{Cahill_2018_CVPR,author = {Cahill, Nathan D. and Hayes, Tyler L. and Meinhold, Renee T. and Hamilton, John F.},title = {Compassionately Conservative Balanced Cuts for Image Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abdelrahman Abdelhamed, Stephen Lin, Michael S. Brown

Abstract: The last decade has seen an astronomical shift from imaging with DSLR and point-and-shoot cameras to imaging with smartphone cameras.  Due to the small aperture and sensor size, smartphone images have notably more noise than their DSLR counterparts.  While denoising for smartphone images is an active research area, the research community currently lacks a denoising image dataset representative of real noisy images from smartphone cameras with high-quality ground truth.  We address this issue in this paper with the following contributions.    We propose a systematic procedure for estimating ground truth for noisy images that can be used to benchmark denoising performance for smartphone cameras.  Using this procedure, we have captured a dataset, the Smartphone Image Denoising Dataset (SIDD), of ~30,000 noisy images from 10 scenes under different lighting conditions using five representative smartphone cameras and generated their ground truth images.  We used this dataset to benchmark a number of denoising algorithms.  We show that CNN-based methods perform better when trained on our high-quality dataset than when trained using alternative strategies, such as low-ISO images used as a proxy for ground truth data.

@InProceedings{Abdelhamed_2018_CVPR,author = {Abdelhamed, Abdelrahman and Lin, Stephen and Brown, Michael S.},title = {A High-Quality Denoising Dataset for Smartphone Cameras},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: Video frame interpolation algorithms typically estimate optical flow or its variations and then use it to guide the synthesis of an intermediate frame between two consecutive original frames. To handle challenges like occlusion, bidirectional flow between the two input frames is often estimated and used to warp and blend the input frames. However, how to effectively blend the two warped frames still remains a challenging problem. This paper presents a context-aware synthesis approach that warps not only the input frames but also their pixel-wise contextual information and uses them to interpolate a high-quality intermediate frame. Specifically, we first use a pre-trained neural network to extract per-pixel contextual information for input frames. We then employ a state-of-the-art optical flow algorithm to estimate bidirectional flow between them and pre-warp both input frames and their context maps. Finally, unlike common approaches that blend the pre-warped frames, our method feeds them and their context maps to a video frame synthesis neural network to produce the interpolated frame in a context-aware fashion. Our neural network is fully convolutional and is trained end to end. Our experiments show that our method can handle challenging scenarios such as occlusion and large motion and outperforms representative state-of-the-art approaches.

@InProceedings{Niklaus_2018_CVPR,author = {Niklaus, Simon and Liu, Feng},title = {Context-Aware Synthesis for Video Frame Interpolation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Wenguan Wang, Jianbing Shen, Xingping Dong, Ali Borji

Abstract: Research in visual saliency has been focused on two major types of models namely fixation prediction and salient object detection. The relationship between the two, however, has been less explored. In this paper, we propose to employ the former model type to identify and segment salient objects in scenes. We build a novel neural network called Attentive Saliency Network (ASNet) that learns to detect salient objects from fixation maps. The fixation map, derived at the upper network layers, captures a high-level understanding of the scene. Salient object detection is then viewed as fine-grained object-level saliency segmentation and is progressively optimized with the guidance of the fixation map in a top-down manner. ASNet is based on a hierarchy of convolutional LSTMs (convLSTMs) that offers an efficient recurrent mechanism for sequential refinement of the segmentation map. Several loss functions are introduced for boosting the performance of the ASNet. Extensive experimental evaluation shows that our proposed ASNet is capable of generating accurate segmentation maps with the help of the computed fixation map. Our work offers a deeper insight into the mechanisms of attention and narrows the gap between salient object detection and fixation prediction.

@InProceedings{Wang_2018_CVPR,author = {Wang, Wenguan and Shen, Jianbing and Dong, Xingping and Borji, Ali},title = {Salient Object Detection Driven by Fixation Prediction},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Daniel S. Jeon, Seung-Hwan Baek, Inchang Choi, Min H. Kim

Abstract: We present a novel method that can enhance the spatial resolution of stereo images using a parallax prior. While traditional stereo imaging has focused on estimating depth from stereo images, our method utilizes stereo images to enhance spatial resolution instead of estimating disparity. The critical challenge for enhancing spatial resolution from stereo images: how to register corresponding pixels with subpixel accuracy. Since disparity in traditional stereo imaging is calculated per pixel, it is directly inappropriate for enhancing spatial resolution. We, therefore, learn a parallax prior from stereo image datasets by jointly training two-stage networks. The first network learns how to enhance the spatial resolution of stereo images in luminance, and the second network learns how to reconstruct a high-resolution color image from high-resolution luminance and chrominance of the input image. Our two-stage joint network enhances the spatial resolution of stereo images significantly more than single-image super-resolution methods. The proposed method is directly applicable to any stereo depth imaging methods, enabling us to enhance the spatial resolution of stereo images.

@InProceedings{Jeon_2018_CVPR,author = {Jeon, Daniel S. and Baek, Seung-Hwan and Choi, Inchang and Kim, Min H.},title = {Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Amos Sironi, Manuele Brambilla, Nicolas Bourdis, Xavier Lagorce, Ryad Benosman

Abstract: Event-based cameras have recently drawn the attention of the Computer Vision community thanks to their advantages  in terms of high temporal resolution, low power consumption and high dynamic range, compared to traditional frame-based cameras.  These properties make event-based cameras an ideal choice for autonomous vehicles, robot navigation or UAV vision, among others.  However, the accuracy of event-based object classification algorithms,  which is of crucial importance for any reliable system working in real-world conditions,  is still far behind their frame-based counterparts. Two main reasons for this performance gap are:  1. The lack of effective low-level representations and architectures for event-based object classification and  2. The absence of large real-world event-based datasets. In this paper we address both problems.  First, we introduce a novel event-based feature representation together with a new machine learning architecture. Compared to previous approaches, we use local memory units to efficiently leverage past temporal information  and build a robust event-based representation.  Second, we release the first large real-world event-based dataset for object classification. We compare our method to the state-of-the-art with extensive experiments,  showing better classification performance and real-time computation.

@InProceedings{Sironi_2018_CVPR,author = {Sironi, Amos and Brambilla, Manuele and Bourdis, Nicolas and Lagorce, Xavier and Benosman, Ryad},title = {HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Lu Zhang, Ju Dai, Huchuan Lu, You He, Gang Wang

Abstract: Recent progress on salient object detection is beneficial from Fully Convolutional Neural Network (FCN). The saliency cues contained in multi-level convolutional features are complementary for detecting salient objects. How to integrate multi-level features becomes an open problem in saliency detection. In this paper, we propose a novel bi-directional message passing model to integrate multi-level features for salient object detection. At first, we adopt a Multi-scale Context-aware Feature Extraction Module (MCFEM) for multi-level feature maps to capture rich context information. Then a bi-directional structure is designed to pass messages between multi-level features, and a gate function is exploited to control the message passing rate. We use the features after message passing, which simultaneously encode semantic information and spatial details, to predict saliency maps. Finally, the predicted results are efficiently combined to generate the final saliency map. Quantitative and qualitative experiments on five benchmark datasets demonstrate that our proposed model performs favorably against the state-of-the-art methods under different evaluation metrics.

@InProceedings{Zhang_2018_CVPR,author = {Zhang, Lu and Dai, Ju and Lu, Huchuan and He, You and Wang, Gang},title = {A Bi-Directional Message Passing Model for Salient Object Detection},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: We propose a new error measure for matching pixels that is based on co-occurrence statistics. The measure relies on a co-occurrence matrix that counts the number of times pairs of pixel values co-occur within a window. The error incurred by matching a pair of pixels is inverse proportional to the probability that their values co-occur together, and not their color difference. This measure also works with features other than color, e.g. deep features. We show that this improves the state-of-the-art performance of template matching on standard benchmarks.  We then propose an embedding scheme that maps the input image to an embedded image such that the Euclidean distance between pixel values in the embedded space resembles the co-occurrence statistics in the original space. This lets us run existing vision algorithms on the embedded images and enjoy the power of co-occurrence statistics for free. We demonstrate this on two algorithms, the Lucas-Kanade image registration and the Kernelized Correlation Filter (KCF) tracker. Experiments show that performance of each algorithm improves by about 10%.

@InProceedings{Kat_2018_CVPR,author = {Kat, Rotal and Jevnisek, Roy and Avidan, Shai},title = {Matching Pixels Using Co-Occurrence Statistics},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Gwangmo Song, Heesoo Myeong, Kyoung Mu Lee

Abstract: In this paper, we propose an automatic seed generation technique with deep reinforcement learning to solve the interactive segmentation problem. One of the main issues of the interactive segmentation problem is robust and consistent object extraction with less human effort. Most of the existing algorithms highly depend on the distribution of inputs, which differs from one user to another and hence need sequential user interactions to achieve adequate performance. In our system, when a user first specifies a point on the desired object and a point in the background, a sequence of artificial user input is automatically generated for precisely segmenting the desired object. The proposed system allows the user to reduce the number of input significantly. This problem is difficult to cast as a supervised learning problem because it is not possible to define globally optimal user input at some stage of the interactive segmentation task. Hence, we formulate automatic seed generation problem as Markov Decision Process (MDP) and then optimize it by reinforcement learning with Deep Q-Network (DQN). We train our network on the MSRA10K dataset and show that the network achieves notable performance improvement from inaccurate initial segmentation on both seen and unseen datasets.

@InProceedings{Song_2018_CVPR,author = {Song, Gwangmo and Myeong, Heesoo and Lee, Kyoung Mu},title = {SeedNet: Automatic Seed Generation With Deep Reinforcement Learning for Robust Interactive Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Shoichiro Takeda, Kazuki Okami, Dan Mikami, Megumi Isogai, Hideaki Kimata

Abstract: Video magnification reveals subtle changes invisible to the naked eye, but such tiny yet meaningful changes are often hidden under large motions: small deformation of the muscles in doing sports, or tiny vibrations of strings in ukulele playing. For magnifying subtle changes under large motions, video acceleration magnification method has recently been proposed. This method magnifies subtle acceleration changes and ignores slow large motions. However, quick large motions severely distort this method. In this paper, we present a novel use of jerk to make the acceleration method robust to quick large motions. Jerk has been used to assess smoothness of time series data in the neuroscience and mechanical engineering fields. On the basis of our observation that subtle changes are smoother than quick large motions at temporal scale, we used jerk-based smoothness to design a jerk-aware filter that passes subtle changes only under quick large motions. By applying our filter to the acceleration method, we obtain impressive magnification results better than those obtained with state-of-the-art.

@InProceedings{Takeda_2018_CVPR,author = {Takeda, Shoichiro and Okami, Kazuki and Mikami, Dan and Isogai, Megumi and Kimata, Hideaki},title = {Jerk-Aware Video Acceleration Magnification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, Jun Zhu

Abstract: Neural networks are vulnerable to adversarial examples, which poses a threat to their application in security sensitive systems. We propose high-level representation guided denoiser (HGD) as a defense for image classification. Standard denoiser suffers from the error amplification effect, in which small residual adversarial noise is progressively amplified and leads to wrong classifications. HGD overcomes this problem by using a loss function defined as the difference between the target model's outputs activated by the clean image and denoised image. Compared with ensemble adversarial training which is the state-of-the-art defending method on large images, HGD has three advantages. First, with HGD as a defense, the target model is more robust to either white-box or black-box adversarial attacks. Second, HGD can be trained on a small subset of the images and generalizes well to other images and unseen classes. Third, HGD can be transferred to defend models other than the one guiding it. In NIPS competition on defense against adversarial attacks, our HGD solution won the first place and outperformed other models by a large margin. footnote{Code: url{https://github.com/lfz/Guided-Denoise}.}

@InProceedings{Liao_2018_CVPR,author = {Liao, Fangzhou and Liang, Ming and Dong, Yinpeng and Pang, Tianyu and Hu, Xiaolin and Zhu, Jun},title = {Defense Against Adversarial Attacks Using High-Level Representation Guided Denoiser},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: Understanding shadows from a single image consists of two types of task in previous studies, containing shadow detection and shadow removal. In this paper, we present a multi-task perspective, which is not embraced by any existing work, to jointly learn both detection and removal in an end-to-end fashion that aims at enjoying the mutually improved benefits from each other. Our framework is based on a novel STacked Conditional Generative Adversarial Network (ST-CGAN), which is composed of two stacked CGANs, each with a generator and a discriminator. Specifically, a shadow image is fed into the first generator which produces a shadow detection mask. That shadow image, concatenated with its predicted mask, goes through the second generator in order to recover its shadow-free image consequently. In addition, the two corresponding discriminators are very likely to model higher level relationships and global scene characteristics for the detected shadow region and reconstruction via removing shadows, respectively. More importantly, for multi-task learning, our design of stacked paradigm provides a novel view which is notably different from the commonly used one as the multi-branch version. To fully evaluate the performance of our proposed framework, we construct the first large-scale benchmark with 1870 image triplets (shadow image, shadow mask image, and shadow-free image) under 135 scenes. Extensive experimental results consistently show the advantages of ST-CGAN over several representative state-of-the-art methods on two large-scale publicly available datasets and our newly released one.

@InProceedings{Wang_2018_CVPR,author = {Wang, Jifeng and Li, Xiang and Yang, Jian},title = {Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Xin Yang, Ke Xu, Yibing Song, Qiang Zhang, Xiaopeng Wei, Rynson W.H. Lau

Abstract: Image correction aims to adjust an input image into a visually pleasing one with the detail in the under/over exposed regions recovered. However, existing image correction methods are mainly based on image pixel operations, and attempting to recover the lost detail from these under/over exposed regions is challenging. We, therefore, revisit the image formation procedure and notice that detail is contained in the high dynamic range (HDR) light intensities(which can be perceived by human eyes) but is lost during the nonlinear imaging process by of the camera in the low dynamic range (LDR) domain. Inspired by this observation, we formulate the image correction problem as the Deep Reciprocating HDR Transformation (DRHT) process and propose a novel approach to first reconstruct the lost detail in the HDR domain and then transfer them back to the LDR image as the output image with the recovered detail preserved. To this end, we propose an end-to-end DRHT model, which contains two CNNs, one for HDR detail reconstruction and the other for LDR detail correction. Experiments on the standard benchmarks demonstrate the effectiveness of the proposed method, compared with state-of-the-art image correction methods.

@InProceedings{Yang_2018_CVPR,author = {Yang, Xin and Xu, Ke and Song, Yibing and Zhang, Qiang and Wei, Xiaopeng and Lau, Rynson W.H.},title = {Image Correction via Deep Reciprocating HDR Transformation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Ekta Prashnani, Hong Cai, Yasamin Mostofi, Pradeep Sen

Abstract: The ability to estimate the perceptual error between images is an important problem in computer vision with many applications. Although it has been studied extensively, however, no method currently exists that can robustly predict visual differences like humans. Some previous approaches used hand-coded models, but they fail to model the complexity of the human visual system. Others used machine learning to train models on human-labeled datasets, but creating large, high-quality datasets is difficult because people are unable to assign consistent error labels to distorted images. In this paper, we present a new learning-based method that is the first to predict perceptual image error like human observers. Since it is much easier for people to compare two given images and identify the one more similar to a reference than to assign quality scores to each, we propose a new, large-scale dataset labeled with the probability that humans will prefer one image over another. We then train a deep-learning model using a novel, pairwise-learning framework to predict the preference of one distorted image over the other. Our key observation is that our trained network can then be used separately with only one distorted image and a reference to predict its perceptual error, without ever being trained on explicit human perceptual-error labels. The perceptual error estimated by our new metric, PieAPP, is well-correlated with human opinion. Furthermore, it significantly outperforms existing algorithms, beating the state-of-the-art by almost 3x on our test set in terms of binary error rate, while also generalizing to new kinds of distortions, unlike previous learning-based methods.

@InProceedings{Prashnani_2018_CVPR,author = {Prashnani, Ekta and Cai, Hong and Mostofi, Yasamin and Sen, Pradeep},title = {PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Meng Tang, Abdelaziz Djelouah, Federico Perazzi, Yuri Boykov, Christopher Schroers

Abstract: Most recent semantic segmentation methods train deep convolutional neural networks with fully annotated masks requiring pixel-accuracy for good quality training. Common weakly-supervised approaches generate full masks from partial input (e.g. scribbles or seeds) using standard interactive segmentation methods as preprocessing. But, errors in such masks result in poorer training since standard loss functions (e.g. cross-entropy) do not distinguish seeds from potentially mislabeled other pixels. Inspired by the general ideas in semi-supervised learning, we address these problems via a new principled loss function evaluating network output with criteria standard in ``shallow'' segmentation, e.g. normalized cut. Unlike prior work, the cross entropy part of our loss evaluates only seeds where labels are known while normalized cut softly evaluates consistency of all pixels.  We focus on normalized cut loss where dense Gaussian kernel is efficiently implemented in linear time by fast Bilateral filtering. Our normalized cut loss approach to segmentation brings the quality of weakly-supervised training significantly closer to fully supervised methods.

@InProceedings{Tang_2018_CVPR,author = {Tang, Meng and Djelouah, Abdelaziz and Perazzi, Federico and Boykov, Yuri and Schroers, Christopher},title = {Normalized Cut Loss for Weakly-Supervised CNN Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: With the aim of developing a fast yet accurate algorithm for compressive sensing (CS) reconstruction of natural images, we combine in this paper the merits of two existing categories of CS methods: the structure insights of traditional optimization-based methods and the performance/speed of recent network-based ones. Specifically, we propose a novel structured deep network, dubbed ISTA-Net,  which is inspired by the Iterative Shrinkage-Thresholding Algorithm (ISTA) for optimizing a general L1 norm CS reconstruction model. To cast ISTA into deep network form, we develop an effective strategy to solve the proximal mapping associated with the sparsity-inducing regularizer using nonlinear transforms. All the parameters in ISTA-Net (e.g. nonlinear transforms, shrinkage thresholds, step sizes, etc.) are learned end-to-end, rather than being hand-crafted. Moreover, considering that the residuals of natural images are more compressible, an enhanced version of ISTA-Net in the residual domain, dubbed ISTA-Net+, is derived to further improve CS reconstruction. Extensive CS experiments demonstrate that the proposed ISTA-Nets outperform existing state-of-the-art optimization-based and network-based CS methods by large margins, while maintaining fast computational speed.

@InProceedings{Zhang_2018_CVPR,author = {Zhang, Jian and Ghanem, Bernard},title = {ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Fast End-to-End Trainable Guided Filter

Huikai Wu, Shuai Zheng, Junge Zhang, Kaiqi Huang

Abstract: Image processing and pixel-wise dense prediction have been advanced by harnessing the capabilities of deep learning. One central issue of deep learning is the limited capacity to handle joint upsampling. We present a deep learning building block for joint upsampling, namely guided filtering layer. This layer aims at efficiently generating the high-resolution output given the corresponding low-resolution one and a high-resolution guidance map. The proposed layer is composed of a guided filter, which is reformulated as a fully differentiable block. To this end, we show that a guided filter can be expressed as a group of spatial varying linear transformation matrices. This layer could be integrated with the convolutional neural networks (CNNs) and jointly optimized through end-to-end training. To further take advantage of end-to-end training, we plug in a trainable transformation function that generates task-specific guidance maps. By integrating the CNNs and the proposed layer, we form deep guided filtering networks. The proposed networks are evaluated on five advanced image processing tasks. Experiments on MIT-Adobe FiveK Dataset demonstrate that the proposed approach runs 10-100 times faster and achieves the state-of-the-art performance. We also show that the proposed guided filtering layer helps to improve the performance of multiple pixel-wise dense prediction tasks.

@InProceedings{Wu_2018_CVPR,author = {Wu, Huikai and Zheng, Shuai and Zhang, Junge and Huang, Kaiqi},title = {Fast End-to-End Trainable Guided Filter},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Andrew Gilbert, John Collomosse, Hailin Jin, Brian Price

Abstract: Content-aware image completion or in-painting is a fundamental tool for the correction of defects or removal of objects in images.  We propose a non-parametric in-painting algorithm that enforces both structural and aesthetic (style) consistency within the resulting image.  Our contributions are two-fold: 1) we explicitly disentangle image structure and style during patch search and selection to ensure a visually consistent look and feel within the target image; 2) we perform adaptive stylization of patches to conform the aesthetics of selected patches to the target image, so harmonising the integration of selected patches into the final composition.  We show that explicit consideration of visual style during in-painting delivers excellent qualitative and quantitative results across the varied image styles and content, over the Places2 photographic dataset and a challenging new in-painting dataset of artwork derived from BAM!

@InProceedings{Gilbert_2018_CVPR,author = {Gilbert, Andrew and Collomosse, John and Jin, Hailin and Price, Brian},title = {Disentangling Structure and Aesthetics for Style-Aware Image Completion},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, Nong Sang

Abstract: Most existing methods of semantic segmentation still suffer from two aspects of challenges: intra-class inconsistency and inter-class indistinction. To tackle these two problems, we propose a Discriminative Feature Network (DFN), which contains two sub-networks: Smooth Network and Border Network. Specifically, to handle the intra-class inconsistency problem, we specially design a Smooth Network with Channel Attention Block and global average pooling to select the more discriminative features. Furthermore, we propose a Border Network to make the bilateral features of boundary distinguishable with deep semantic boundary supervision. Based on our proposed DFN, we achieve state-of-the-art performance 86.2% mean IOU on PASCAL VOC 2012 and 80.3% mean IOU on Cityscapes dataset.

@InProceedings{Yu_2018_CVPR,author = {Yu, Changqian and Wang, Jingbo and Peng, Chao and Gao, Changxin and Yu, Gang and Sang, Nong},title = {Learning a Discriminative Feature Network for Semantic Segmentation},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}