Researchers at FIT BUT in cooperation with Adobe Research developed a new augmented reality method and are waiting for their pat

Imagine yourself pulling out your phone to take a picture of a beautiful view. You point the camera towards the landscape and your device tells you, in augmented reality, the names of surrounding hills, their elevation, tourist paths in the area and can even show you the terrain contours. The new software tool created by researchers of the CPhoto@FIT BUT group is able to recognise the photographer's location and the photographed object and knows what the surrounding area looks like under different weather conditions. This allows you to edit the picture at home, e.g. make it sharper or adjust the shadows. On top of that, it can take you back to the place where the picture was taken using virtual reality.

The new software developed by a team of members of the Computational Photography research group at the Faculty of Information Technology of Brno University of Technology in cooperation with Adobe Research can do all of that. It was presented at the prestigious ECCV conference and its authors are currently waiting to have it patented.

"Our software is designed to give more accuracy to the position and direction of a camera in the outdoor environment. The mobile application uses virtual reality to provide information on the surrounding area, such as names of mountains and rivers. It can also show isolines or distances to a mountain lodge or, simply put, any piece of topographic information about the actual terrain," says Martin Čadík, head of the CPhoto@FIT research group.

The phone uses GPS location to generate a synthetic view of the landscape, similarly to how Google Earth works. It then detects significant points in the picture or the screen, such as skyline of hills and shape of rivers or forests, and compares them to terrain models. It can thus determine the location and direction of the camera with an accuracy within metres. The comparison of points in the photograph and the 3D model is automatic thanks to a neural network , which was trained on thousands of landscape pictures from the researchers' own sources, as well as photos available on the Internet.

VIDEO: LandscapeAR

"At first, we used these pictures to compare individual points. That is, we were comparing photographs to photographs. There was a number of disadvantages. For one, we were unable to perform the localisation in places from where we had no pictures. Now we compare the photographs to 3D terrain models. These models cover the whole planet, even remote areas where people cannot go, and also include data with textures specific for different seasons. That helps with the localisation if the landscape changes over the year," explains Martin Čadík. This is a great success in the field of computational photography, which was made possible especially thanks to the development of neural networks and improved access to accurate terrain models with textures.

The algorithms will then help users especially at home when using a computer. Thanks to the software, it already knows perfectly the location from which the picture was taken and the orientation of the camera, i.e. the place the photographer wanted to capture. This enables for the photographs to be edited in all sorts of ways that would otherwise be very complicated - the users can, for example, move the point of focus to a different peak, add shadows or adjust the lighting of the photograph. The software can also fit the picture directly into the landscape and bring the photographer back to the same spot from where the picture was taken through virtual reality. Using a special pair of glasses, the photographer's friends and family have the chance to take a look at the same spot where the photographer stood when taking the photo and to see its surroundings beyond the frame of the original picture.

The tool was created as a part of the Deep-Learning Approach to Topographical Image Analysis, a project of the Czech Ministry of Education. The FIT's scientists intend to follow up with further research in order to teach the software to determine the location and direction of the camera on a larger scale using neural networks and terrain models without the previous rough location estimate from GPS.