Camera Orientation Estimation in Natural Scenes Using Semantic Cues

camera orientation estimation, camera calibration, semantic segmentation, digital elevation model of a terrain, OpenStreetMap, geo-localization, computer vision, computer graphics

Camera orientation estimation in natural scenes has recently been approached by several methods, which rely mainly on matching a single modality - edges or horizon lines with 3D digital elevation models. In contrast to previous works, our new image to model matching scheme is based on a fusion of multiple modalities and is designed to be naturally extensible with different cues. In this paper, we use semantic segments and edges. To our knowledge, we are the first to consider using semantic segments jointly with edges for alignment with digital elevation model. We show that high-level features, such as semantic segments, complement the low-level edge information and together help to estimate the camera orientation more robustly compared to methods relying solely on edges or horizon lines. In a series of experiments, we show that segment boundaries tend to be imprecise and important information for matching is encoded in the segment area and a coarse shape. Intuitively, semantic segments encode low frequency information as opposed to edges, which encode high frequencies. Our experiments exhibit that semantic segments and edges are complementary, improving camera orientation estimation reliability when used together. We demonstrate that our method combining semantic and edge features is able to reach state-of-the-art performance on three datasets.

We introduce a novel method for camera orientation estimation with the assumption that camera position and field-of-view are known. Our method includes following steps: (I) from an input image edges and semantic segments describing semantic areas, such as forests, bodies of water, sky or glaciers are detected; (II) with the knowledge of camera position a synthetic 360-degree panorama containing semantic segments from OpenStreetMap database and mountain silhouettes from digital elevation model is rendered; (III) detected and rendered semantic segments and edges are matched with each other using our novel Confidence Fusion (CF) framework to estimate the camera orientation. The proposed CF framework is based on a spherical cross-correlation, which is computed for each semantic class and edge layer separately to form per layer confidences. The confidences are fused into a final result using a weighted geometric average with custom weights. 

Our method is suitable for camera orientation estimation in outdoor environments and can be used in various applications. The applications include virtual and augmented reality, navigation, and visualizations including historical re-photography. We provide extensive experiments and ablation studies to show that our method is able to reach state-of-the-art performance on three publicly available datasets.

This publication has been presented on International Conference on 3D Vision 2018 in Verona, Italy, and is available online: https://ieeexplore.ieee.org/document/8490971. Supplementary materials and a video summarizing our paper are available on the project webpage: http://cphoto.fit.vutbr.cz/semantic-orientation/.