Commit Graph

1550 Commits

Author SHA1 Message Date
Czar Echavez
c94fa653f7 Merge remote-tracking branch 'origin/dev' into feature/2533-improve-dev-tools-in-primaite-cli 2024-04-29 16:24:19 +01:00
Czar Echavez
a7492ba39f #2447: cli commands for dev mode 2024-04-29 13:45:10 +01:00
Marek Wolan
32a0542b73 Merged PR 353: Episode Schedule
## Summary
* Made It Possible™ to vary the layout and agents between episodes.
* Standardised the environments to all use `env_config` as the __init__ parameter name, previously PrimaiteGymEnv was using `game_config`
* Added a notebook that demonstrates how to use the variable episodes.

## Test process
Checked that existing pytests run. Added more tests. Checked that notebooks run.

## Checklist
- [x] PR is linked to a **work item**
- [x] **acceptance criteria** of linked ticket are met
- [x] performed **self-review** of the code
- [x] written **tests** for any new functionality added with this PR
- [x] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [x] updated the **change log**
- [x] ran **pre-commit** checks for code style
- [x] attended to any **TO-DOs** left in the code

Related work items: #2269, #2334, #2336, #2475, #2476
2024-04-29 09:08:37 +00:00
Marek Wolan
394e76048f Merge remote-tracking branch 'origin/dev' into feature/2476-training-schedules-mockup 2024-04-26 16:23:41 +01:00
Marek Wolan
30cc04411d #2476 apply pr comments 2024-04-26 16:23:02 +01:00
Charlie Crane
5ee23dcb17 Merged PR 348: #2462 - Refactor of DatabaseClient and DatabaseServer
## Summary
Refactor of `DatabaseClient` and `DatabaseService` to update how connection IDs are generated. These are now provided by DatabaseService when establishing a connection.
Creation of `DatabaseClientConnection` class. This is used by `DatabaseClient` to hold a dictionary of active db connections.

## Test process
Tests have been updated to reflect the changes and all pass

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [X] performed **self-review** of the code
- [X] written **tests** for any new functionality added with this PR
- [X] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [X] updated the **change log**
- [X] ran **pre-commit** checks for code style
- [X] attended to any **TO-DOs** left in the code

Related work items: #2462
2024-04-26 14:52:21 +00:00
Marek Wolan
be5cf0b877 Merge remote-tracking branch 'origin/dev' into feature/2476-training-schedules-mockup 2024-04-25 15:17:08 +01:00
Marek Wolan
37945c0065 #2476 Remove temporary testing file 2024-04-25 15:16:20 +01:00
Czar Echavez
e1ac6255ad Merged PR 352: #2447: added mode to primaite cli + printing session to current working direc...
## Summary
- Added the mode command in PrimAITE CLI
- command allows users to switch between developer mode and production mode
- developer mode changes where the simulation_output is written
  - dev mode writes simulation output where the current working directory is

## Test process
CLI:
![primaite mode.gif](https://dev.azure.com/ma-dev-uk/b50a61ee-86c4-48bc-9a0b-a67645ba12ee/_apis/git/repositories/2825053e-bd3b-45b2-8680-1281809eefa2/pullRequests/352/attachments/primaite%20mode.gif)

Running prod mode:
![primaite mode prod.gif](https://dev.azure.com/ma-dev-uk/b50a61ee-86c4-48bc-9a0b-a67645ba12ee/_apis/git/repositories/2825053e-bd3b-45b2-8680-1281809eefa2/pullRequests/352/attachments/primaite%20mode%20prod.gif)

Running dev mode:
![primaite mode dev.gif](https://dev.azure.com/ma-dev-uk/b50a61ee-86c4-48bc-9a0b-a67645ba12ee/_apis/git/repositories/2825053e-bd3b-45b2-8680-1281809eefa2/pullRequests/352/attachments/primaite%20mode%20dev.gif)

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [X] performed **self-review** of the code
- [ ] written **tests** for any new functionality added with this PR
- [X] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [X] updated the **change log**
- [X] ran **pre-commit** checks for code style
- [ ] attended to any **TO-DOs** left in the code

#2447: added mode to primaite cli + printing session to current working directory if in dev mode

Related work items: #2447
2024-04-25 14:13:55 +00:00
Marek Wolan
736408f8b4 Reference the episode schedule notebook in docs 2024-04-25 15:12:46 +01:00
Marek Wolan
66f31e8ed1 #2476 Add test for episode scheduler 2024-04-25 15:09:46 +01:00
Marek Wolan
42ce264e73 #2476 Fix string formatting 2024-04-25 13:54:05 +01:00
Czar Echavez
0fff29ef27 #2447: changelog + documentation 2024-04-25 13:50:41 +01:00
Marek Wolan
e3f9162a77 Merge remote-tracking branch 'origin/dev' into feature/2476-training-schedules-mockup 2024-04-25 13:50:11 +01:00
Czar Echavez
96b6fb81da #2447: added mode to primaite cli + printing session to current working directory if in dev mode 2024-04-25 13:32:57 +01:00
Marek Wolan
a92898d001 #2476 Finalise explanation notebook for episode schedule 2024-04-25 13:25:26 +01:00
Czar Echavez
f01c30864d Merged PR 347: Implementing log levels
## Summary
- Implementation of log levels
- log levels can be set via io_settings.log_level in the yaml config file

## Test process
https://dev.azure.com/ma-dev-uk/PrimAITE/_git/PrimAITE/pullrequest/347?_a=files&path=/tests/unit_tests/_primaite/_simulator/_system/core/test_sys_log.py

set up a test where the test checks the contents of the terminal so make sure that specific log level messages appear (or not) depending on the log level

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [X] performed **self-review** of the code
- [X] written **tests** for any new functionality added with this PR
- [X] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [X] updated the **change log**
- [X] ran **pre-commit** checks for code style
- [X] attended to any **TO-DOs** left in the code

Related work items: #2470
2024-04-24 11:09:48 +00:00
Czar Echavez
d4aaeda4b6 #2470: update doc to reflect the default log level 2024-04-24 10:11:23 +01:00
Czar Echavez
23231e432e Merge remote-tracking branch 'origin/dev' into feature/2470-refactor-syslog-to-be-able-to-leverage-a-configured-log-level 2024-04-24 09:17:28 +01:00
Czar Echavez
16fc6f4fd0 Merge remote-tracking branch 'origin/dev' into feature/2470-refactor-syslog-to-be-able-to-leverage-a-configured-log-level 2024-04-23 16:54:03 +01:00
Czar Echavez
32cd7dba04 Merged PR 350: #2510: cleaning up logs and add logging handler clearing to prevent duplicate...
## Summary
- Cleaning up multiple logs that say the same thing
- Added Nick's suggestion to clear log handlers before adding a handler on SysLog initialisation

## Test process
*How have you tested this (if applicable)?*

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [ ] performed **self-review** of the code
- [ ] written **tests** for any new functionality added with this PR
- [ ] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [ ] updated the **change log**
- [ ] ran **pre-commit** checks for code style
- [ ] attended to any **TO-DOs** left in the code

#2510: cleaning up logs and add logging handler clearing to prevent duplicate logs

Related work items: #2510
2024-04-23 15:53:58 +00:00
Czar Echavez
90343bd5ec #2470: apply PR suggestions 2024-04-23 16:52:53 +01:00
Christopher McCarthy
a60e8b2203 Merged PR 349: #2511 - Upgraded pydantic to version 2.7.0
## Summary
- Upgraded pydantic to version 2.7.0.
- Added ipywidgets to the dependencies (for #2300)

## Test process
N/A

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [X] performed **self-review** of the code
- [ ] written **tests** for any new functionality added with this PR
- [ ] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [X] updated the **change log**
- [X] ran **pre-commit** checks for code style
- [ ] attended to any **TO-DOs** left in the code

#2511 - Upgraded pydantic to version 2.7.0. Added ipywidgets to the dependencies (for #2300)

Related work items: #2300, #2511
2024-04-23 13:43:44 +00:00
Marek Wolan
8e008b6d24 #2511 Appease isort pre commit hook 2024-04-23 13:54:56 +01:00
Czar Echavez
c13b7d7c81 #2510: cleaning up logs and add logging handler clearing to prevent duplicate logs 2024-04-23 11:54:34 +01:00
Marek Wolan
28c8b7c9d9 #2476 Get episode schedule working 2024-04-23 11:51:50 +01:00
Marek Wolan
1eca20157b Updated CHANGELOG.md 2024-04-23 09:04:49 +00:00
Marek Wolan
2b19c8c91d #2551 upgrade ray to >2.9 and resolve logging error in db client 2024-04-23 09:51:56 +01:00
Marek Wolan
2b3664ce36 #2476 Add proof of concept yaml combining notebook 2024-04-23 09:29:59 +01:00
Chris McCarthy
6060cbbc5b #2511 - Put the ray rlllib dep back to "ray[rllib] == 2.8.0, < 3" 2024-04-23 09:04:17 +01:00
Nick Todd
779a9587bc Merged PR 338: #2455: Fix typo in config file and notebook
## Summary
Fixed small typo in data_manipulation_marl.yaml. Correct database key in Data-Manipulation-Customising-Red-Agent notebook.

## Test process
Successfully re-ran all notebooks in src/primaite/notebooks on Windows and Linux (WSL).
Successfully re-ran all tests in primaite/tests.
Also checked that Ransomware-Kill-Chain-E2E and UC6-E2E-Demo ran correctly in PrimAITE Sensitive repo.

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [X] performed **self-review** of the code
- [ ] written **tests** for any new functionality added with this PR
- [ ] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [ ] updated the **change log**
- [X] ran **pre-commit** checks for code style
- [ ] attended to any **TO-DOs** left in the code

#2455: Fix typo in config file and notebook

Related work items: #2455
2024-04-22 15:35:51 +00:00
Nick Todd
6726a2da37 #2245: Add comment 2024-04-22 14:09:12 +01:00
Chris McCarthy
a7dae6e373 #2511 - Upgraded pydantic to version 2.7.0. Added ipywidgets to the dependencies (for #2300) 2024-04-22 08:49:08 +01:00
Czar Echavez
57e6f8bca7 #2470: update documentation 2024-04-19 11:55:38 +01:00
Nick Todd
873b70c2c9 Merge branch 'dev' into bugfix/2455-notebook_updates 2024-04-19 11:52:09 +01:00
Czar Echavez
b8d4a8cc8d #2470: changed default log level to warning + changed sys logging in code to be more aligned with severity of messages 2024-04-19 11:37:52 +01:00
Nick Todd
9334b1e79b #2455: Add NICObservation.ConfigSchema to NIC list. 2024-04-19 11:07:32 +01:00
Czar Echavez
833fd18936 #2470: add write_sys_log_to_terminal to documentation 2024-04-18 16:44:18 +01:00
Czar Echavez
cbea0fd595 #2470: fix documentation 2024-04-18 16:40:05 +01:00
Czar Echavez
34773ed225 #2470: implement PR suggestions 2024-04-18 16:38:42 +01:00
Czar Echavez
2a1203675d #2470: changelog + documentation 2024-04-18 15:57:02 +01:00
Czar Echavez
94ca13c0f9 #2470: add log level via config + test 2024-04-18 15:14:40 +01:00
Charlie Crane
56d03aaf7b Merged PR 339: #2453 - Example Notebooks Refactor
## Summary
Updates to `create-simulation_demo.ipynb` and `network_simulator_demo.ipynb` to reflect changes within repo.
 - Corrected import paths
 - Corrected typo in create-simulation_demo: my_swtich -> my_switch and switch.sys_log.show -> computer.sys_log.show()
 - Changes to ARP implementation within nodes. Now defined as a property within HostNode and NetworkNode, allowing router.arp.show() to work as intended within the notebook.

## Test process
 - Both notebooks now run without issue when ran.
 - All tests pass

## Checklist
- [X] PR is linked to a **work item**
- [X] **acceptance criteria** of linked ticket are met
- [X] performed **self-review** of the code
- [ ] written **tests** for any new functionality added with this PR
- [ ] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [ ] updated the **change log**
- [X] ran **pre-commit** checks for code style
- [X] attended to any **TO-DOs** left in the code

Related work items: #2453, #2477, #2478
2024-04-18 13:36:23 +00:00
Charlie Crane
abf94fc4bb #2453 - Committing additional explanations to notebook 2024-04-18 13:52:43 +01:00
Charlie Crane
bb88d43b90 #2453 - Updating with some explanation to improve the readability of the notebook 2024-04-18 13:50:47 +01:00
Charlie Crane
49be3b3639 #2453 - Updates to Training-an-SB3-agent.ipynb following review 2024-04-18 10:23:26 +01:00
Charlie Crane
d05ded0dc2 Merge branch 'dev' into bugfix/2453-Example_Notebooks_require_refactor 2024-04-18 09:30:12 +01:00
Czar Echavez
b57deaf9e1 #2470: implementing log levels into sys log 2024-04-17 18:13:00 +01:00
Charlie Crane
999044a444 2453 - removal of notebook metadata 2024-04-17 10:44:44 +01:00
Marek Wolan
137a203acc Merged PR 346: Deprecate PrimaiteSession
## Summary
A while ago we have added several notebooks to get users started training their own agents. These notebooks overlap in functionality with PrimAITE session but they are more suitable.

* removed primaite session
* removed any tests relying on primaite session (except some tests were converted into environment tests.
* removed any CLI commands that rely on `PrimaiteSession`
* Added deprecation warnings to any benchmarking code that will break now because they rely on the old Primaite 2 session. (I left the code in because when we update benchmarking to v3, this will be a useful starting point)
* added similar deprecation warnings to Session metadata writer, reader and transaction stuff, for the same reason.
* removed parts of the documentation that are no longer relevant (session, training config, policy)
* went through all existing yaml config files and removed training options (as they were for session only)

## Test process
Checked that existing pytests all pass, and updated some pytests to use environment instead of session.

## Checklist
- [x] PR is linked to a **work item**
- [x] **acceptance criteria** of linked ticket are met
- [x] performed **self-review** of the code
- [x] written **tests** for any new functionality added with this PR
- [x] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [ ] updated the **change log**
- [x] ran **pre-commit** checks for code style
- [x] attended to any **TO-DOs** left in the code

Related work items: #2374
2024-04-17 08:25:43 +00:00