A hybrid method for detecting anomalous traffic in computer networks

Abstract

This study addresses the increasing difficulty of detecting anomalies in network traffic caused by growing threats to information and communication systems. Traditional intrusion detection systems often fail to adapt to new threats, particularly when analyzing outbound traffic, which may signal internal compromise. To overcome these limitations, the study proposes a hybrid detection method aimed at improving anomaly identification accuracy. The method integrates three components. First, traffic is classified using a signature-based approach with predefined sets of allowed and prohibited signatures. Second, self-similarity analysis with the Hurst coefficient detects long-term traffic patterns. Third, fuzzy logic is applied to interpret uncertain traffic characteristics, such as port numbers, protocols, intensity, and packet sizes, using linguistic variables and fuzzy rules. The research presents formalized models of both legitimate and malicious user behavior and a composite packet signature model for comprehensive traffic analysis. This approach enhances adaptability and reduces the proportion of unclassified traffic. Experimental validation using real and synthetic data confirms improved detection accuracy and a lower false positive rate compared to conventional methods. The scientific novelty lies in combining deterministic classification with fuzzy logic within a single detection pipeline, with a special emphasis on outbound traffic monitoring. The practical value of the proposed system is its suitability for integration into existing cybersecurity frameworks, contributing to more effective threat detection and reduced operational risks in evolving network environments.