T. Nathan Mundhenk, Daniel Ho, Barry Y. Chen

Abstract: We develop a set of methods to improve on the results of self-supervised learning using context. We start with a baseline of patch based arrangement context learning and go from there. Our methods address some overt problems such as chromatic aberration as well as other potential problems such as spatial skew and mid-level feature neglect. We prevent problems with testing generalization on common self-supervised benchmark tests by using different datasets during our development. The results of our methods combined yield top scores on all standard self-supervised benchmarks, including classification and detection on PASCAL VOC 2007, segmentation on PASCAL VOC 2012, and "linear tests" on the ImageNet and CSAIL Places datasets. We obtain an improvement over our baseline method of between 4.0 to 7.1 percentage points on transfer learning classification tests. We also show results on different standard network architectures to demonstrate generalization as well as portability. All data, models and programs are available at: https://gdo-datasci. llnl.gov/selfsupervised/.

@InProceedings{Mundhenk_2018_CVPR,author = {Mundhenk, T. Nathan and Ho, Daniel and Chen, Barry Y.},title = {Improvements to Context Based Self-Supervised Learning},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Rohit Keshari, Mayank Vatsa, Richa Singh, Afzel Noore

Abstract: Convolutional Neural Networks have provided state-of-the-art results in several computer vision problems. However, due to a large number of parameters in CNNs, they require a large number of training samples which is a limiting factor for small sample size problems. To address this limitation, in this paper, we propose SSF-CNN which focuses on learning the “structure" and “strength" of filters. The structure of the filter is initialized using a dictionary based filter learning algorithm and the strength of the filter is learned using the small sample training data. The architecture provides the flexibility of training with both small and large training databases, and yields good accuracies even with small size training data. The effectiveness of the algorithm is demonstrated on MNIST, CIFAR10, NORB, Omniglot, and Newborn Face Image databases, with varying number of training samples. The results show that SSF-CNN significantly reduces the number of parameters required for training while providing high accuracies on the test database. On small problems such as newborn face recognition, the results demonstrate improvement in rank-1 identification accuracy by at least 10%.

@InProceedings{Keshari_2018_CVPR,author = {Keshari, Rohit and Vatsa, Mayank and Singh, Richa and Noore, Afzel},title = {Learning Structure and Strength of CNN Filters for Small Sample Size Training},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, Hamed Pirsiavash

Abstract: In self-supervised learning one trains a model to solve a so-called pretext task on a dataset without the need for human annotation. The main objective, however, is to transfer this model to a target domain and task. Currently, the most effective transfer strategy is fine-tuning, which restricts one to use the same model or parts thereof for both pretext and target tasks. In this paper, we present a novel framework for self-supervised learning that overcomes limitations in designing and comparing different tasks, models, and data domains. In particular, our framework decouples the structure of the self-supervised model from the final task-specific fine-tuned model. This allows us to: 1) quantitatively assess previously incompatible models including handcrafted features; 2) show that deeper neural network models can learn better representations from the same pretext task; 3) transfer knowledge learned with a deep model to a shallower one and thus boost its learning.  We use this framework to design a novel self-supervised task, which achieves state-of-the-art performance on the common benchmarks in PASCAL VOC 2007, ILSVRC12 and Places by a significant margin. A surprising result is that our learned features shrink the mAP gap between models trained via self-supervised learning and supervised learning from $5.9$ to $2.6$ in object detection on PASCAL VOC 2007.

@InProceedings{Noroozi_2018_CVPR,author = {Noroozi, Mehdi and Vinjimoor, Ananth and Favaro, Paolo and Pirsiavash, Hamed},title = {Boosting Self-Supervised Learning via Knowledge Transfer},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


William H. Beluch, Tim Genewein, Andreas Nürnberger, Jan M. Köhler

Abstract: Deep learning methods have become the de-facto standard for challenging image processing tasks such as image classification. One major hurdle of deep learning approaches is that large sets of labeled data are necessary, which can be prohibitively costly to obtain, particularly in medical image diagnosis applications. Active learning techniques can alleviate this labeling effort. In this paper we investigate some recently proposed methods for active learning with high-dimensional data and convolutional neural network classifiers. We compare ensemble-based methods against Monte-Carlo Dropout and geometric approaches. We find that ensembles perform better and lead to more calibrated predictive uncertainties, which are the basis for many active learning algorithms. To investigate why Monte-Carlo Dropout uncertainties perform worse, we explore potential differences in isolation in a series of experiments. We show results for MNIST and CIFAR-10, on which we achieve a test set accuracy of $90 %$ with roughly 12,200 labeled images, and initial results on ImageNet. Additionally, we show results on a large, highly class-imbalanced diabetic retinopathy dataset. We observe that the ensemble-based active learning effectively counteracts this imbalance during acquisition.

@InProceedings{Beluch_2018_CVPR,author = {Beluch, William H. and Genewein, Tim and Nürnberger, Andreas and Köhler, Jan M.},title = {The Power of Ensembles for Active Learning in Image Classification},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, Zenglin Xu

Abstract: Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in Videos and Image Captioning. To overcome this problem, we propose a compact and flexible structure, namely Block-Term tensor decomposition, which greatly reduces the parameters of RNNs and improves their training efficiency. Compared with alternative low-rank approximations, such as tensor-train RNN (TT-RNN), our method, Block-Term RNN (BT-RNN), is not only more concise (when using the same rank), but also able to attain a better approximation to the original RNNs with much fewer parameters.  On three challenging tasks, including Action Recognition in Videos, Image Captioning and Image Generation, BT-RNN outperforms TT-RNN and the standard RNN in terms of both prediction accuracy and convergence rate. Specifically, BT-LSTM utilizes 17,388 times fewer parameters than the standard LSTM to achieve an accuracy improvement over 15.6% in the Action Recognition task on the UCF11 dataset.

@InProceedings{Ye_2018_CVPR,author = {Ye, Jinmian and Wang, Linnan and Li, Guangxi and Chen, Di and Zhe, Shandian and Chu, Xinqi and Xu, Zenglin},title = {Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Domen Tabernik, Matej Kristan, Aleš Leonardis

Abstract: Classical deep convolutional networks increase receptive field size by either gradual resolution reduction or application of hand-crafted dilated convolutions to prevent increase in the number of parameters. In this paper we propose a novel displaced aggregation unit (DAU) that does not require hand-crafting. In contrast to classical filters with units (pixels) placed on a fixed regular grid, the displacement of the DAUs are learned, which enables filters to spatially-adapt their receptive field to a given problem. We extensively demonstrate the strength of DAUs on a classification and semantic segmentation tasks. Compared to ConvNets with regular filter, ConvNets with DAUs achieve comparable performance at faster convergence and up to 3-times reduction in parameters. Furthermore, DAUs allow us to study deep networks from novel perspectives. We study spatial distributions of DAU filters and analyze the number of parameters allocated for spatial coverage in a filter.

@InProceedings{Tabernik_2018_CVPR,author = {Tabernik, Domen and Kristan, Matej and Leonardis, Aleš},title = {Spatially-Adaptive Filter Units for Deep Neural Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: This paper presents SO-Net, a permutation invariant architecture for deep learning with orderless point clouds. The SO-Net models the spatial distribution of point cloud by building a Self-Organizing Map (SOM). Based on the SOM, SO-Net performs hierarchical feature extraction on individual points and SOM nodes, and ultimately represents the input point cloud by a single feature vector. The receptive field of the network can be systematically adjusted by conducting point-to-node k nearest neighbor search. In recognition tasks such as point cloud reconstruction, classification, object part segmentation and shape retrieval, our proposed network demonstrates performance that is similar with or better than state-of-the-art approaches. In addition, the training speed is significantly faster than existing point cloud recognition networks because of the parallelizability and simplicity of the proposed architecture. Our code is available at the project website.

@InProceedings{Li_2018_CVPR,author = {Li, Jiaxin and Chen, Ben M. and Lee, Gim Hee},title = {SO-Net: Self-Organizing Network for Point Cloud Analysis},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: The Generative Adversarial Networks (GANs) have demonstrated impressive performance for data synthesis, and are now used in a wide range of computer vision tasks. In spite of this success, they gained a reputation for being difficult to train, what results in a time-consuming and human-involved development process to use them.  We consider an alternative training process, named SGAN, in which several adversarial "local" pairs of networks are trained independently so that a "global" supervising pair of networks can be trained against them. The goal is to train the global pair with the corresponding ensemble opponent for improved performances in terms of mode coverage. This approach aims at increasing the chances that learning will not stop for the global pair, preventing both to be trapped in an unsatisfactory local minimum, or to face oscillations often observed in practice. To guarantee the latter, the global pair never affects the local ones.  The rules of SGAN training are thus as follows: the global generator and discriminator are trained using the local discriminators and generators, respectively, whereas the local networks are trained with their fixed local opponent.  Experimental results on both toy and real-world problems demonstrate that this approach outperforms standard training in terms of better mitigating mode collapse, stability while converging and that it surprisingly, increases the convergence speed as well.

@InProceedings{Chavdarova_2018_CVPR,author = {Chavdarova, Tatjana and Fleuret, François},title = {SGAN: An Alternative Training of Generative Adversarial Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: Synthesizing realistic images from human drawn sketches is a challenging problem in computer graphics and vision. Existing approaches either need exact edge maps, or rely on retrieval of existing photographs. In this work, we propose a novel Generative Adversarial Network (GAN) approach that synthesizes plausible images from 50 categories including motorcycles, horses and couches. We demonstrate a data augmentation technique for sketches which is fully automatic, and we show that the augmented data is helpful to our task. We introduce a new network building block suitable for both the generator and discriminator which improves the information flow by injecting the input image at multiple scales. Compared to state-of-the-art image translation methods, our approach generates more realistic images and achieves significantly higher Inception Scores.

@InProceedings{Chen_2018_CVPR,author = {Chen, Wengling and Hays, James},title = {SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Aojun Zhou, Anbang Yao, Kuan Wang, Yurong Chen

Abstract: Benefiting from tens of millions of hierarchically stacked learnable parameters, Deep Neural Networks (DNNs) have demonstrated overwhelming accuracy on a variety of artificial intelligence tasks. However reversely, the large size of DNN models lays a heavy burden on storage, computation and power consumption, which prohibits their deployments on the embedded and mobile systems. In this paper, we propose Explicit Loss-error-aware Quantization (ELQ), a new method that can train DNN models with very low-bit parameter values such as ternary and binary ones to approximate 32-bit floating-point counterparts without noticeable loss of predication accuracy. Unlike existing methods that usually pose the problem as a straightforward approximation of the layer-wise weights or outputs of the original full-precision model (specifically, minimizing the error of the layer-wise weights or inner products of the weights and the inputs between the original and respective quantized models), our ELQ elaborately bridges the loss perturbation from the weight quantization and an incremental quantization strategy to address DNN quantization. Through explicitly regularizing the loss perturbation and the weight approximation error in an incremental way, we show that such a new optimization method is theoretically reasonable and practically effective. As validated with two mainstream convolutional neural network families (i.e., fully convolutional and non-fully convolutional), our ELQ shows better results than the state-of-the-art quantization methods on the large scale ImageNet classification dataset. Code will be made publicly available.

@InProceedings{Zhou_2018_CVPR,author = {Zhou, Aojun and Yao, Anbang and Wang, Kuan and Chen, Yurong},title = {Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Yi Zhu, Yang Long, Yu Guan, Shawn Newsam, Ling Shao

Abstract: Unseen Action Recognition (UAR) aims to recognise novel action categories without training examples. While previous methods focus on inner-dataset seen/unseen splits, this paper proposes a pipeline using a large-scale training source to achieve a Universal Representation (UR) that can generalise to a more realistic Cross-Dataset UAR (CD-UAR) scenario. We first address UAR as a Generalised Multiple-Instance Learning (GMIL) problem and discover "building-blocks" from the large-scale ActivityNet dataset using distribution kernels. Essential visual and semantic components are preserved in a shared space to achieve the UR that can efficiently generalise to new datasets. Predicted UR exemplars can be improved by a simple semantic adaptation, and then an unseen action can be directly recognised using UR during the test. Without further training, extensive experiments manifest significant improvements over the UCF101 and HMDB51 benchmarks.

@InProceedings{Zhu_2018_CVPR,author = {Zhu, Yi and Long, Yang and Guan, Yu and Newsam, Shawn and Shao, Ling},title = {Towards Universal Representation for Unseen Action Recognition},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Deep Image Prior

Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky

Abstract: Deep convolutional networks have become a popular tool for image generation and restoration. Generally, their excellent performance is imputed to their ability to learn realistic image priors from a large number of example images. In this paper, we show that, on the contrary, the structure of a generator network is sufficient to capture a great deal of low-level image statistics prior to any learning. In order to do so, we show that a randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting. Furthermore, the same prior can be used to invert deep neural representations to diagnose them, and to restore images based on flash-no flash input pairs.  Apart from its diverse applications, our approach highlights the inductive bias captured by standard generator network architectures. It also bridges the gap between two very popular families of image restoration methods: learning-based methods using deep convolutional networks and learning-free methods based on handcrafted image priors such as self-similarity.

@InProceedings{Ulyanov_2018_CVPR,author = {Ulyanov, Dmitry and Vedaldi, Andrea and Lempitsky, Victor},title = {Deep Image Prior},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, Simon Lucey

Abstract: We address the problem of finding realistic geometric corrections to a foreground object such that it appears natural when composited into a background image. To achieve this, we propose a novel Generative Adversarial Network (GAN) architecture that utilizes Spatial Transformer Networks (STNs) as the generator, which we call Spatial Transformer GANs (ST-GANs). ST-GANs seek image realism by operating in the geometric warp parameter space. In particular, we exploit an iterative STN warping scheme and propose a sequential training strategy that achieves better results compared to naive training of a single generator. One of the key advantages of ST-GAN is its applicability to high-resolution images indirectly since the predicted warp parameters are transferable between reference frames. We demonstrate our approach in two applications: (1) visualizing how indoor furniture (e.g. from product images) might be perceived in a room, (2) hallucinating how accessories like glasses would look when matched with real portraits.

@InProceedings{Lin_2018_CVPR,author = {Lin, Chen-Hsuan and Yumer, Ersin and Wang, Oliver and Shechtman, Eli and Lucey, Simon},title = {ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}


Abstract: In this paper, we propose a solution to transforming photos of real-world scenes into cartoon style images, which is valuable and challenging in computer vision and computer graphics. Our solution belongs to learning based methods, which have recently become popular to stylize images in artistic forms such as painting. However, existing methods do not produce satisfactory results for cartoonization, due to the fact that (1) cartoon styles have unique characteristics with high level simplification and abstraction, and (2) cartoon images tend to have clear edges, smooth color shading and relatively simple textures, which exhibit significant challenges for texture-descriptor-based loss functions used in existing methods. In this paper, we propose CartoonGAN, a generative adversarial network (GAN) framework for cartoon stylization. Our method takes unpaired photos and cartoon images for training, which is easy to use. Two novel losses suitable for cartoonization are proposed: (1) a semantic content loss, which is formulated as a sparse regularization in the high-level feature maps of the VGG network to cope with substantial style variation between photos and cartoons, and (2) an edge-promoting adversarial loss for preserving clear edges. We further introduce an initialization phase, to improve the convergence of the network to the target manifold. Our method is also much more efficient to train than existing methods. Experimental results show that our method is able to generate high-quality cartoon images from real-world photos (i.e., following specific artists' styles and with clear edges and smooth shading) and outperforms state-of-the-art methods.

@InProceedings{Chen_2018_CVPR,author = {Chen, Yang and Lai, Yu-Kun and Liu, Yong-Jin},title = {CartoonGAN: Generative Adversarial Networks for Photo Cartoonization},booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},month = {June},year = {2018}}