Christopher McCarthy 69c3f97cd9 Merged PR 178: Database Client/Server Simulation
## Summary
This pull request focuses on implementing key functionalities needed for network frame processing and database interactions. The primary changes are:

1. **Internal Frame Processing:** The logic has been implemented in various components like `NIC`, `Node`, `SessionManager`, and `SoftwareManager`. These changes enable the system to process incoming and outgoing network frames in a structured manner.
2. **Database Service and Client:** The `DatabaseService` simulates a SQL database server, while the `DatabaseClient` provides a client interface for connecting to this service. These functionalities have been built and integrated into the existing architecture.
3. **Networking and Communication:** Tests have been added to confirm that database queries can be sent over the network, demonstrating end-to-end functionality.

## Commits

- #1816 Simplified a bunch of stuff in the file system in prep for services and applications. Started adding the database logic. Waiting for the software manager/session manager work from another tick. Merge branch 'dev' into feature/1816_Database-Service-(Network-and-User-Interaction)
- #1816 Added the final pieces of the puzzle to get data up from NIC → session manager → software manager → service.
- #1816 DatabaseService now uses the send function when responding.
- #1816 Added database client. Installed the database client on the Web Server node in the UC2 network. Updated the integration test to query the DB server using the DB client.
- #1816 Added full documentation on the database client/server, and the internal frame processing process
- #1816 Fixed tests. Used node and link added number (id) in observation space.

## Test process
For testing these functionalities, the following steps were taken:

1. **Unit Tests:** Tests have been written to confirm that database queries can be sent over the network successfully.
2. **Integration Tests:** Manually tested the frame processing flow from NIC to Service/Application, ensuring the functionality behaves as expected.
3. **Database Queries** Executed sample SQL queries using the `DatabaseClient` to make sure it interacts correctly with the `DatabaseService`.

## Checklist
- [ ] This PR is linked to a **work item**
- [ ] I have performed **self-review** of the code
- [ ] I have written **tests** for any new functionality added with this PR
- [ ] I have updated the **documentation** if this PR changes or adds functionality
- [ ] I have written/updated **design docs** if this PR implements new functionality
- [ ] I have update the **change log**
- [ ] I have run **pre-commit** checks for code style

Related work items: #1816
2023-09-12 13:31:08 +00:00
2023-08-01 16:18:49 +01:00
2023-08-15 13:28:02 +01:00
2023-07-20 10:54:42 +01:00
2023-08-23 14:41:59 +01:00
2023-09-06 11:19:30 +01:00
2023-06-02 12:59:01 +01:00

PrimAITE

image

The ARCD Primary-level AI Training Environment (PrimAITE) provides an effective simulation capability for the purposes of training and evaluating AI in a cyber-defensive role. It incorporates the functionality required of a primary-level ARCD environment, which includes:

  • The ability to model a relevant platform / system context;

  • The ability to model key characteristics of a platform / system by representing connections, IP addresses, ports, traffic loading, operating systems and services;

  • Operates at machine-speed to enable fast training cycles.

PrimAITE presents the following features:

  • Highly configurable (via YAML files) to provide the means to model a variety of platform / system laydowns and adversarial attack scenarios;

  • A Reinforcement Learning (RL) reward function based on (a) the ability to counter the specific modelled adversarial cyber-attack, and (b) the ability to ensure success;

  • Provision of logging to support AI evaluation and metrics gathering;

  • Uses the concept of Information Exchange Requirements (IERs) to model background pattern of life and adversarial behaviour;

  • An Access Control List (ACL) function, mimicking the behaviour of a network firewall, is applied across the model, following standard ACL rule format (e.g. DENY/ALLOW, source IP address, destination IP address, protocol and port);

  • Application of IERs to the platform / system laydown adheres to the ACL ruleset;

  • Presents an OpenAI gym or RLLib interface to the environment, allowing integration with any compliant defensive agents;

  • Full capture of discrete logs relating to agent training (full system state, agent actions taken, instantaneous and average reward for every step of every episode);

  • NetworkX provides laydown visualisation capability.

Getting Started with PrimAITE

💫 Install & Run

PrimAITE is designed to be OS-agnostic, and thus should work on most variations/distros of Linux, Windows, and MacOS. Currently, the PrimAITE wheel can only be installed from GitHub. This may change in the future with release to PyPi.

Windows (PowerShell)

Prerequisites:

  • Manual install of Python >= 3.8 < 3.11

Install:

mkdir ~\primaite
cd ~\primaite
python3 -m venv .venv
attrib +h .venv /s /d # Hides the .venv directory
.\.venv\Scripts\activate
pip install https://github.com/Autonomous-Resilient-Cyber-Defence/PrimAITE/releases/download/v2.0.0/primaite-2.0.0-py3-none-any.whl
primaite setup

Run:

primaite session

Unix

Prerequisites:

  • Manual install of Python >= 3.8 < 3.11
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.10
sudo apt-get install python3-pip
sudo apt-get install python3-venv

Install:

mkdir ~/primaite
cd ~/primaite
python3 -m venv .venv
source .venv/bin/activate
pip install https://github.com/Autonomous-Resilient-Cyber-Defence/PrimAITE/releases/download/v2.0.0/primaite-2.0.0-py3-none-any.whl
primaite setup

Run:

primaite session

Developer Install from Source

To make your own changes to PrimAITE, perform the install from source (developer install)

1. Clone the PrimAITE repository

git clone git@github.com:Autonomous-Resilient-Cyber-Defence/PrimAITE.git

2. CD into the repo directory

cd PrimAITE

3. Create a new python virtual environment (venv)

python3 -m venv venv

4. Activate the venv

Unix
source venv/bin/activate
Windows (Powershell)
.\venv\Scripts\activate

5. Install primaite with the dev extra into the venv along with all of it's dependencies

python3 -m pip install -e .[dev]

6. Perform the PrimAITE setup:

primaite setup

📚 Building documentation

The PrimAITE documentation can be built with the following commands:

Unix
cd docs
make html
Windows (Powershell)
cd docs
.\make.bat html
Description
ARCD Primary-Level AI Training Environment (PrimAITE)
Readme 21 MiB
Languages
Python 80.2%
Jupyter Notebook 19.8%