Merged PR 452: Automate the benchmark process

## Summary
This pull request integrates full automation of the PrimAITE benchmarking process into our Azure DevOps pipeline, enhancing consistency and minimizing manual effort.

#### Key Changes:

1.  **Pipeline Automation**: Configured to trigger on commits to `dev` and `release` branches, with scheduled runs during off-peak hours.

2.  **Error Handling Improvements**: The pipeline is set to fail on any Python execution errors, ensuring immediate identification of issues.

3.  **Artifact Publishing**: Benchmark results are now automatically compressed and published as artifacts, facilitating easy access and analysis.

## Test process
Tested the pipeline runs from end-to-end and produces a working benchmark report: https://dev.azure.com/ma-dev-uk/PrimAITE/_build/results?buildId=5280&view=results

## Checklist
- [ ] PR is linked to a **work item**
- [ ] **acceptance criteria** of linked ticket are met
- [ ] performed **self-review** of the code
- [ ] written **tests** for any new functionality added with this PR
- [ ] updated the **documentation** if this PR changes or adds functionality
- [ ] written/updated **design docs** if this PR implements new functionality
- [ ] updated the **change log**
- [ ] ran **pre-commit** checks for code style
- [ ] attended to any **TO-DOs** left in the code

Related work items: #2739
This commit is contained in:
Christopher McCarthy
2024-07-09 19:39:27 +00:00
2 changed files with 63 additions and 59 deletions

View File

@@ -11,14 +11,17 @@ schedules:
branches: branches:
include: include:
- 'refs/heads/dev' - 'refs/heads/dev'
pool:
vmImage: ubuntu-latest
variables: variables:
VERSION: '' VERSION: ''
MAJOR_VERSION: '' MAJOR_VERSION: ''
jobs:
- job: PrimAITE_Benchmark
timeoutInMinutes: 360 # 6-hour maximum
pool:
vmImage: ubuntu-latest
workspace:
clean: all
steps: steps:
- checkout: self - checkout: self
persistCredentials: true persistCredentials: true
@@ -50,6 +53,7 @@ steps:
displayName: 'Install Dependencies' displayName: 'Install Dependencies'
- script: | - script: |
set -e
cd benchmark cd benchmark
python3 primaite_benchmark.py python3 primaite_benchmark.py
cd .. cd ..

View File

@@ -26,7 +26,7 @@
"av_s_per_session": 3205.6340542, "av_s_per_session": 3205.6340542,
"av_s_per_step": 0.10017606419375, "av_s_per_step": 0.10017606419375,
"av_s_per_100_steps_10_nodes": 10.017606419375, "av_s_per_100_steps_10_nodes": 10.017606419375,
"combined_av_reward_per_episode": { "combined_total_reward_per_episode": {
"1": -53.42999999999999, "1": -53.42999999999999,
"2": -25.18000000000001, "2": -25.18000000000001,
"3": -42.00000000000002, "3": -42.00000000000002,