Increasing Throughput of Intrusion Detection Systems by Hash-Based Short String Pre-Filter

regular expressions, network traffic filtration, hash-based short string prefilter, IDS, network security systems, field programmable gate arrays

With an increasing speed of network links, it is also necessary to increase the throughput of network security systems. An intrusion detection system (IDS) is one of the key components in the protection of network infrastructure. Unfortunately, the IDS has to match a large set of regular expressions (REs) in network streams, which has a negative impact on its throughput. Currently, multiple parallel machines have to be used to support 100 Gbps throughput of Suricata or Bro IDS.
A fast pre-filtration of network traffic can allow the IDS to achieve a higher overall throughput. Therefore, we have designed a new algorithm, which is able to select a set of short strings that represents an RE set utilized in the IDS. Such a set of strings can facilitate fast and efficient pre-filtration. Compared to previous methods, strings selected by the proposed algorithm can reduce network traffic up to 3.3 times better. Moreover, the algorithm is able to select strings representing a single RE in less than a second, thus  allowing fast updates of an IDS ruleset. As all selected strings have the same length, they can be used in a hash-based pre-filter, which is able to process more 100 Gbps of network traffic.