Project Details

HAAWAII - Highly Automated Air Traffic Controller Workstations with Artificial Intelligence Integration

Project Period: 1. 6. 2020 - 30. 11. 2022

Project Type: grant

Code: H2020-SESAR-2019-2, 884287

Agency: European Comission EU

Program: Horizon 2020

Czech title
HAAWAII - Highly Automated Air Traffic Controller Workstations with Artificial Intelligence Integration
Type
grant
Keywords

Artificial Intelligence , Machine Learning, Air-Traffic Control, Natural Language Processing, Automatic Speech Recognition, 

Abstract


Advanced automation support developed in Wave 1 of SESAR IR includes using of automatic speech recognition (ASR) to
reduce the amount of manual data inputs by air-traffic controllers. Evaluation of controllers feedback has been subdued due
to the limited recognition performance of the commercial of the shell ASR engines that were used, even in laboratory
conditions. The reasons for the unsatisfactory conclusions include e.g. inability to distinguish controllers accents, deviations
from standard phraseology and limited real-time recognition performance. Past exploratory research funded project
MALORCA, however, has shown (on restricted use-cases) that satisfactory performance can be reached with novel datadriven
machine learning approaches.
Based on the results of MALORCA HAAWAII project aims to research and develop a reliable, error resilient and adaptable
solution to automatically transcribe voice commands issued by both air-traffic controllers and pilots. The project will build on
very large collection of data, organized with a minimum expert effort to develop a new set of models for complex
environments of Icelandic en-route and London TMA. HAAWAII aims to perform proof-of-concept trials in challenging
environments, i.e. to be directly connected with real-life data from ops room. As pilot read-back error detection is the main
application, HAAWAII aims to significantly enhance the validity of the speech recognition models. The proposed work goes
far beyond the work planned for the Wave 2 IR programme and will improve both safety and reduce controllers workload.
The digitization of controller and pilot voice utterances can be used for a wide variety of safety and performance related
benefits including, but not limiting to pre-fill entries into electronic flight strips and CPDLC messages. Another application
demonstrated during proof-of-concept will be to objectively estimate controllers workload utilising digitized voice recordings
of the complex London TMA.

Team members
Smrž Pavel, doc. RNDr., Ph.D. (UPGM FIT VUT) , research leader
Doležal Jan, Ing. (UPGM FIT VUT)
Dytrych Jaroslav, Ing., Ph.D. (UPGM FIT VUT)
Hradiš Michal, Ing., Ph.D. (UPGM FIT VUT)
Jírovec Martin, Ing. (Děkanát FIT VUT)
Musil Martin, Ing., Ph.D. (UPGM FIT VUT)
Otrusina Lubomír, Ing. (UPGM FIT VUT)
Veselý Karel, Ing., Ph.D. (UPGM FIT VUT)
Publications

2023

  • ZULUAGA-GOMEZ Juan, PRASAD Amrutha, NIGMATULINA Iuliia, MOTLÍČEK Petr and KLEINERT Matthias. A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers. Aerospace, vol. 10, no. 5, 2023, pp. 1-25. ISSN 2226-4310. Detail
  • MOTLÍČEK Petr, PRASAD Amrutha, NIGMATULINA Iuliia, HELMKE Hartmut, OHNEISER Oliver and KLEINERT Matthias. Automatic Speech Analysis Framework for ATC Communication in HAAWAII. In: Proceedings of the 13th SESAR Innovation Days. Seville: SESAR Joint Undertaking, 2023, pp. 1-9. Detail
  • ZULUAGA-GOMEZ Juan, SARFJOO Seyyed Saeed, PRASAD Amrutha, NIGMATULINA Iuliia, MOTLÍČEK Petr, ONDŘEJ Karel, OHNEISER Oliver and HELMKE Hartmut. BERTraffic: BERT-based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications. In: IEEE Spoken Language Technology Workshop, SLT 2022 - Proceedings. Doha: IEEE Signal Processing Society, 2023, pp. 633-640. ISBN 978-1-6654-7189-3. Detail

2022

  • NIGMATULINA Iuliia, ZULUAGA-GOMEZ Juan, PRASAD Amrutha, SARFJOO Saeed and MOTLÍČEK Petr. A Two-Step Approach to Leverage Contextual Data: Speech Recognition in Air-Traffic Communications. In: ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings. Singapore: IEEE Signal Processing Society, 2022, pp. 6282-6286. ISBN 978-1-6654-0540-9. Detail
  • PRASAD Amrutha, ZULUAGA-GOMEZ Juan, MOTLÍČEK Petr, SARFJOO Seyyed Saeed, NIGMATULINA Iuliia, OHNEISER Oliver and HELMKE Hartmut. Grammar Based Speaker Role Identification for Air Traffic Control Speech Recognition. In: Proceedings of the 12th SESAR Innovation Days. Budapest, 2022, pp. 1-9. Detail
  • KOCOUR Martin, ŽMOLÍKOVÁ Kateřina, ONDEL Yang Lucas Antoine Francois, ŠVEC Ján, DELCROIX Marc, OCHIAI Tsubasa, BURGET Lukáš and ČERNOCKÝ Jan. Revisiting joint decoding based multi-talker speech recognition with DNN acoustic model. In: Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH. Incheon: International Speech Communication Association, 2022, pp. 4955-4959. ISSN 1990-9772. Detail
  • PRASAD Amrutha, ZULUAGA-GOMEZ Juan, MOTLÍČEK Petr, SARFJOO Seyyed Saeed, NIGMATULINA Iuliia and VESELÝ Karel. Speech and Natural Language Processing Technologies for Pseudo-Pilot Simulator. In: Proceedings of the 12th SESAR Innovation Days. Budapest, 2022, pp. 1-9. Detail

2021

  • KLEINERT Matthias, HELMKE Hartmut, SHETTY Shruthi, OHNEISER Oliver, EHR Heiko, PRASAD Amrutha, MOTLÍČEK Petr and HARFMANN Julia. Automated Interpretation of Air Traffic Control Communication: The Journey from Spoken Words to a Deeper Understanding of the Meaning. In: Proceedings of DASC 2021. San Antonio, Texas: Institute of Electrical and Electronics Engineers, 2021, pp. 1-9. ISBN 978-1-6654-3420-1. Detail
  • KOCOUR Martin, VESELÝ Karel, BLATT Alexander, ZULUAGA-GOMEZ Juan, SZŐKE Igor, ČERNOCKÝ Jan, KLAKOW Dietrich and MOTLÍČEK Petr. Boosting of Contextual Information in ASR for Air-Traffic Call-Sign Recognition. In: Proceedings Interspeech 2021. Brno: International Speech Communication Association, 2021, pp. 3301-3305. ISSN 1990-9772. Detail
  • ZULUAGA-GOMEZ Juan, NIGMATULINA Iuliia, PRASAD Amrutha, MOTLÍČEK Petr, VESELÝ Karel, KOCOUR Martin and SZŐKE Igor. Contextual Semi-Supervised Learning: An Approach to Leverage Air-Surveillance and Untranscribed ATC Data in ASR Systems. In: Proceedings Interspeech 2021. Brno: International Speech Communication Association, 2021, pp. 3296-3300. ISSN 1990-9772. Detail
  • SZŐKE Igor, KESIRAJU Santosh, NOVOTNÝ Ondřej, KOCOUR Martin, VESELÝ Karel and ČERNOCKÝ Jan. Detecting English Speech in the Air Traffic Control Voice Communication. In: Proceedings Interspeech 2021. Brno: International Speech Communication Association, 2021, pp. 3286-3290. ISSN 1990-9772. Detail
  • HELMKE Hartmut, SHETTY Shruthi, KLEINERT Matthias, OHNEISER Oliver, EHR Heiko, MOTLÍČEK Petr, PRASAD Amrutha and WINDISCH Christian et al. Measuring Speech Recognition And Understanding Performance in Air Traffic Control Domain Beyond Word Error Rates. In: Proceedings of 11th SESAR Innovation Days 2021. Belgie, 2021, pp. 1-8. Detail
  • HELMKE Hartmut, KLEINERT Matthias, SHETTY Shruthi, OHNEISER Oliver, EHR Heiko, PRASAD Amrutha, MOTLÍČEK Petr, VESELÝ Karel, ONDŘEJ Karel, SMRŽ Pavel, HARFMANN Julia and WINDISCH Christian et al. Readback Error Detection by Automatic Speech Recognition to Increase ATM Safety. In: Proceedings of ATM Seminar. on-line: EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION, 2021, pp. 1-10. Detail
Back to top