AI in Cyber-Attack Detection

Research Project: AI in cyber-attack detection.

The TeleMARS team comprises software engineers, data scientists, and data engineers. We have been exploring the power of AI through various research projects, including one particularly interesting project in the area of AI for cybersecurity.

Project Background

Cybersecurity is a vital area of research due to the growing reliance on digital systems across government, military, commercial, financial, and civilian domains. In recent years, artificial intelligence (AI) has emerged as a transformative technology for improving Network Intrusion Detection Systems (NIDS) capabilities. Research has demonstrated the potential of AI algorithms, including machine learning models, to address the complexity and scale of modern cyber-attacks. However, gaps remain in integrating these research advancements into practical solutions for real-world applications.

TeleMARS, with over 15 years of experience in applying AI to industry challenges, is committed to advancing the development of AI-based solutions. The project aims to bridge the gap between research and practical implementation, contributing to the cybersecurity domain by developing effective, scalable, and inclusive approaches to tackle emerging network threats.

Challenges

  1. Comprehensive and Up-to-Date Datasets
    Producing datasets that accurately capture both benign and malicious network behaviors is challenging. Designing realistic environments that encompass a wide diversity of attack and normal scenarios is critical for building robust models.
  2. Emerging Attacks
    AI and machine learning models often perform poorly when exposed to previously unseen or emerging attack patterns. Models trained on outdated datasets struggle to adapt to changes in the feature space, leading to detection failures.
  3. Real-Time Detection
    Achieving real-time detection is hindered by challenges such as noisy or irrelevant features in network traffic data and the need for lightweight detection methods. These methods must ensure a balance between processing efficiency and resource utilization.

Objectives

  1. Advance AI-Based Anomaly Detection
    Develop and refine AI-based anomaly detection methods and models to improve the detection of sophisticated and emerging network attacks.
  2. Enhance Research Contributions
    Contribute to the global research community by addressing the challenges of anomaly detection.
  3. Deploy Practical Solutions
    Focus on translating research findings into scalable, real-world solutions that strengthen cybersecurity defences for governments, businesses, and communities.

Expected Research Outcomes

The anticipated research outcomes of this project include the following:

  • Gaining a deeper understanding of the capabilities of reinforcement learning methods in detecting cyber-attacks.
  • Performing a comparative analysis of the performance of several popular machine learning (ML) models.
  • Developing an evaluation method and framework deployable in real-world network environments.
  • Comparing the robustness of various ML models in response to changes in the feature space.

In-Scope Research Work

To achieve these objectives, the project encompasses the following research components:

  1. Literature Review and Data Analysis
    • Conducting a comprehensive literature review to refine project designs, including the selection of research datasets.
    • Performing data analysis to inform the selection and design of learning algorithms, ensuring alignment with project objectives.
  2. Dataset and Model Selection
    • Identifying datasets and ML models that demonstrate strong anomaly detection capabilities.
  3. Design and Development of Machine Learning Models
    • Designing and implementing various machine learning models using traditional experimental approaches, including:
      • Classic shallow learning models.
      • Neural network-based deep learning models.
      • A novel reinforcement learning model.
  4. Evaluation and Experimentation
    • Developing and experimenting with a novel evaluation method for trained models.
    • Testing the evaluation of trained models on separate datasets with feature spaces different from the training data.

Findings

  1. Performance of Shallow Learning Models
    The shallow learning Random Forest (RF) model demonstrated the best overall performance in detecting emerging attacks using a traditional machine learning experimental approach. It was particularly effective in classifying anomaly categories with a small number of records. The K-Nearest Neighbor (KNN) model showed similar strong capabilities.
  2. Reinforcement Learning Model Performance
    The reinforcement learning model (ARL) did not outperform other models. However, there is potential for improving prediction accuracy and detection sensitivity in deep learning and reinforcement learning models. Neural network architectures could be enhanced by adjusting layers, the number of neurons, hyperparameters, and inter-layer dependencies. Such architectural refinements could significantly improve anomaly detection performance.
  3. Sensitivity to Data Structure
    The performance differences observed between experiments using the NSL-KDD dataset and the CIC-IDS2017 dataset highlight that machine learning models are sensitive to the structure of the data.
  4. Efficiency of Models
    The RF, MPL-NN, and ARL models were efficient in execution time and resource usage for small batch files. However, the KNN model consumed significantly more resources and had a longer processing time, making it potentially expensive to implement in real-world network environments.
  5. Impact of Feature Space Variability
    When testing datasets with feature spaces different from the training datasets, machine learning models generally exhibited poor performance. However, within the same feature space, trained models performed consistently on continuous small batch files, even with variations in data volume, anomaly density, and category distribution. This indicates that traditional machine learning methods are significantly constrained by the structure of the training data.
  6. Resilience to Feature Space Changes
    When the CES-CICIDS2018 dataset introduced changes in feature and anomaly value distribution, all trained models exhibited reduced performance. However, the MPL-NN model demonstrated relatively better robustness, indicating that regression-based neural network methods may be more resilient in adapting to feature space changes. Such methods could improve the robustness of anomaly detection models in real network environments.

Lessons Learned and Recommendations

This research demonstrated the effectiveness of shallow learning classification methods for anomaly detection using traditional machine learning experiments. However, the minor variations among performance metrics in controlled experiments do not reflect real-world effectiveness in dynamic network environments.

The experiments revealed that machine learning models are highly sensitive to data. Their effectiveness is often limited to the specific network environments where the training data was generated.

Challenges were identified as below:

  1. Training Data Construction
    Constructing comprehensive training data requires significant effort from cybersecurity and network engineers to analyze and identify network behaviors. Collaboration with data scientists and analysts is essential to structure raw data into suitable features for model development.
  2. Real-Time Data Extraction
    Developing a method to extract and structure real-time network data without compromising operational performance is a critical challenge.
  3. Noise in Real-Time Data
    Real-time network data may contain noise, requiring models to maintain robustness for effective anomaly detection.
  4. Dynamic Network Topologies
    Changes in network topology and architecture over time may alter the data profile, necessitating model retraining.
  5. Evolving Network Behaviours
    Regular evaluations are needed to account for evolving network behaviours and events.
  6. Emerging Cyber-Attacks
    Cyber-attacks continue to evolve. Continuous model update strategy is required to handle new forms of attacks.
  1. Business Changes

In addition to the technical challenges, there are challenges in applying machine learning solutions as part of business operation, for example, the dependencies and impacts on the existing systems, the team readiness, the costs of deployment, implementation and maintenance.

Recommendations

Future research should pivot from improving traditional machine learning models to addressing real-time detection challenges, including:

  • Constructing training data tailored for specific network environments.
  • Utilising Generative AI to train robust anomaly detection models.
  • Enhancing robustness by combining classification-based methods with neural networks.
  • Developing dynamic evaluation mechanisms and adaptive model training frameworks.
  • Designing lightweight model architectures for efficient real-time processing.
  • Implementing operation performance management strategies.
  • Discovering the potential of creating general anomaly detection models for a wide range of network environments.

A more practical approach involves developing anomaly detection models tailored to specific network environments with consistent architecture, development, and management. When training and real-time data share the same feature space structure, machine learning models can achieve high detection accuracy and reliability.

The business processes and technical frameworks need to be redesigned or improved to enable effective benefit realization.

Conclusion

The findings underscore the need for adaptive, environment-specific machine learning models for anomaly detection. By addressing the outlined challenges and prioritizing real-time detection, future research can significantly enhance the robustness and effectiveness of machine learning models in combating evolving cyber threats.