BUT technology powered drone to reveal dangers at large gatherings

A traffic accident, an athlete's collapse or a smoke bomb - all of these need to be addressed as soon as possible to prevent injury or further damage. However, it is sometimes not possible for human beings to identify exactly where the problem originated in the ensuing confusion and to direct rescuers to the right place. The new "anomaly detection system" was created by computer science students and offered to police officers, who could use it to quickly and automatically detect a problem and put it on a map.

"A typical situation is the end of a sporting event. A huge number of people leaving for home. It would be very problematic to keep an eye on the whole area and try to find out if something is wrong. Using a drone, we can determine from above that a problem has occurred and immediately project it on a map. Another situation may be traffic at an intersection or on the street where an accident occurs and cars start to go around it. Our detector can identify the anomaly and alert the operator at the computer," says David Bažout, a recent graduate of the BUT Faculty of Information Technology, who is going to continue his doctorate and at the same time improve his smart greenhouse project. He has been working on the anomaly detection system for the last two years and has managed to develop his own method that allows the drone to be launched over a completely new territory and find its way around such terrain immediately.

The goals of the developed system are clear: helping rescuers and police officers, and even saving lives. | Author: Jan Prokopius

The system is based on a neural network that mimics the human brain. "The video from the drone is divided into a grid composed of smaller cells. The neural network looks at each of these cells and pulls out a simplified description of the situation that is going on there. Based on these labels, the method then creates a model of the usual behaviour in the given grid and determines any unusual behaviour by comparing the current label with the model. Based on this, the system can tell by itself whether this is normal behaviour or something unexpected is happening in the area. The biggest challenge was that the neural network needed a lot of learning data to function, but such data was not available at all. When the police start using a drone and flies over fans at a concert, for example, this is the first time the neural network sees such a situation," says David Bažout about the advantages of the method he developed. The method does not need to teach the neural network in advance what it sees; rather, it can make sense of the situation in real time without any prior information: "The neural network looks at the new scene and immediately learns what is typical of the scene and what is not."

Thanks to the system, the image captured by the drone turns red where something unusual is happening, such as where an athlete suddenly falls to the ground. | Author: David Bažout, Adam Ferenecz

The monitored normal behaviour may be the already mentioned crowd of football fans moving slowly to the bus after the match, but also, for example, a sports competition. Such competition was, among other things, used by Bažout to the system together with his younger classmate Adam Ferenecz. "We had a group of runners who we told where to run and simulated a crowd moving in one direction. In reality, a person, such as a potential attacker, may suddenly separate from the crowd. But it can also happen that the whole crowd suddenly changes direction, spreads, or everyone suddenly falls to the ground. In our test, there is a scene where people run and, at my command, they suddenly drop to the ground. At that moment, the whole area turns red. However, the system is constantly learning, so after a while it is normal for the runners to lie down and the area gradually lightens. The moment they get up, the image turns red again," Ferenecz explains what the operator using their application will see on the computer screen. In order for the transmitted video not to be constantly full of red spots, it is possible to adjust the sensitivity of perceived changes. "When runners run into a corner, for example, the system does not evaluate it as an anomaly, because the persons move along the expected path and it is not a sudden change of direction," adds Ferenecz.

While David Bažout wants to continue his studies at FIT to get a doctorate, Adam Ferenecz is already working part-time in education - he would like to teach young people how to programme. | Author: Jan Prokopius

When used in practice, for example, a police officer present on the spot could pilot a drone over a group of demonstrators. The video, which needs to be processed by artificial intelligence, would be sent to a server located, for example, at a police station, and the resulting image of the advancing crowd would already be drawn on a real map projected either to the dispatcher on a computer or to another nearby police officer with a tablet. According to the authors, it is also possible to add other functions to the application, so that in addition to warnings of danger, it would also be able to count participants or create real-time statistics in the future.

The developed system is part of a project in which the faculty cooperates with police officers. They will now decide if it can be useful to them and start testing it in real situations.