Merged PR 452: Automate the benchmark process
## Summary This pull request integrates full automation of the PrimAITE benchmarking process into our Azure DevOps pipeline, enhancing consistency and minimizing manual effort. #### Key Changes: 1. **Pipeline Automation**: Configured to trigger on commits to `dev` and `release` branches, with scheduled runs during off-peak hours. 2. **Error Handling Improvements**: The pipeline is set to fail on any Python execution errors, ensuring immediate identification of issues. 3. **Artifact Publishing**: Benchmark results are now automatically compressed and published as artifacts, facilitating easy access and analysis. ## Test process Tested the pipeline runs from end-to-end and produces a working benchmark report: https://dev.azure.com/ma-dev-uk/PrimAITE/_build/results?buildId=5280&view=results ## Checklist - [ ] PR is linked to a **work item** - [ ] **acceptance criteria** of linked ticket are met - [ ] performed **self-review** of the code - [ ] written **tests** for any new functionality added with this PR - [ ] updated the **documentation** if this PR changes or adds functionality - [ ] written/updated **design docs** if this PR implements new functionality - [ ] updated the **change log** - [ ] ran **pre-commit** checks for code style - [ ] attended to any **TO-DOs** left in the code Related work items: #2739
This commit is contained in:
@@ -11,14 +11,17 @@ schedules:
|
|||||||
branches:
|
branches:
|
||||||
include:
|
include:
|
||||||
- 'refs/heads/dev'
|
- 'refs/heads/dev'
|
||||||
|
|
||||||
pool:
|
|
||||||
vmImage: ubuntu-latest
|
|
||||||
|
|
||||||
variables:
|
variables:
|
||||||
VERSION: ''
|
VERSION: ''
|
||||||
MAJOR_VERSION: ''
|
MAJOR_VERSION: ''
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
- job: PrimAITE_Benchmark
|
||||||
|
timeoutInMinutes: 360 # 6-hour maximum
|
||||||
|
pool:
|
||||||
|
vmImage: ubuntu-latest
|
||||||
|
workspace:
|
||||||
|
clean: all
|
||||||
steps:
|
steps:
|
||||||
- checkout: self
|
- checkout: self
|
||||||
persistCredentials: true
|
persistCredentials: true
|
||||||
@@ -50,6 +53,7 @@ steps:
|
|||||||
displayName: 'Install Dependencies'
|
displayName: 'Install Dependencies'
|
||||||
|
|
||||||
- script: |
|
- script: |
|
||||||
|
set -e
|
||||||
cd benchmark
|
cd benchmark
|
||||||
python3 primaite_benchmark.py
|
python3 primaite_benchmark.py
|
||||||
cd ..
|
cd ..
|
||||||
|
|||||||
@@ -26,7 +26,7 @@
|
|||||||
"av_s_per_session": 3205.6340542,
|
"av_s_per_session": 3205.6340542,
|
||||||
"av_s_per_step": 0.10017606419375,
|
"av_s_per_step": 0.10017606419375,
|
||||||
"av_s_per_100_steps_10_nodes": 10.017606419375,
|
"av_s_per_100_steps_10_nodes": 10.017606419375,
|
||||||
"combined_av_reward_per_episode": {
|
"combined_total_reward_per_episode": {
|
||||||
"1": -53.42999999999999,
|
"1": -53.42999999999999,
|
||||||
"2": -25.18000000000001,
|
"2": -25.18000000000001,
|
||||||
"3": -42.00000000000002,
|
"3": -42.00000000000002,
|
||||||
|
|||||||
Reference in New Issue
Block a user