Azure DevOps Self-Hosted Agent Pool | Number of agents to keep on standby


Azure Pipelines announced the general availability of scale-set agents in september 2020.

Azure virtual machine scale set agents are a form of self-hosted agents that can be autoscaled to meet your demands. This elasticity reduces your need to run dedicated agents all the time. Unlike Microsoft-hosted agents, you have flexibility over the size and the image of machines on which agents run.

Read more here:

The top benefits of using Scale-set Agents are:

  • Out-of-the-box auto scaling
  • Automatically tear down virtual machines after every use

You don’t longer need to worry about a build that leaves credentials or other sensitive information behind in an Agent.

Or a build that messes the config of the Agent. Well, you get a new fresh Agent every time.

The Problem

One of the settings, that you can set is how many “Number of agents to keep on standby”

Azure Pipelines will automatically scale-down the number of agents, but will ensure that there are always this many agents available to run new jobs. If you set this to 0, for example to conserve cost for a low volume of jobs, Azure Pipelines will start a VM only when it has a job.

While this is a good feature. You would probably like to have a more granular schedule.

Now you’re thinking, “I can just set it to 0 and the Agent Pool will scale once needed”.

Well keep in mind that it takes time for an agent to spin up. The Scale set instance actually has to get deployed. And this can take a while depending on SKU and Size of the image etc.

Now in a critical business, time = money. So you wan’t to make sure that you have a minimum number of agents on standby during business-hours. And then maybe a lower number during off-hours/night.

As for now this is not supported in Azure DevOps. So I wrote a PowerShell Script (see below), that is scheduled to run twice a day. 

The Solution

The Script will loop the Scale Set Agent Pools, and based on the current time, it will set the appropriate “number of agents to keep standby” based on a defined schedule.

The recommended way of setting up this orchestrator is via a Scheduled Build Pipeline.

The pre-requsite for this Orchestrator is the following:

  • You need to have PAT Token for a user that is administrator on the actual Agent pool
  • Preferably you want to store your PAT in a Key vault or as protected build variable

Configure your Build Pipeline:

Set your Build Schedule:

Use the script below in your Build Pipeline.

Make sure to replace the values accordingly

                    Set "number of agents to keep standby" on Self-Hosed Azure DevOps Agent Pool
                    The Script will loop the Scale Set Agent Pools, and based on the current time, it will set the appropriate "number of agents to keep standby" based on a defined schedule.
                    Asif Mithawala

    Version 1 - 2021-03-04
        First version of script created. The Script will loop the Scale Set Agent Pools, and based on the current time, it will set the appropriate "number of agents to keep standby" based on a defined schedule.
        0 is = sunday and saturday is = 6 in the schedule
    [array]$AgentPools = @(
        @{name = "Ubuntu"; id = 1; schedule = '{"TzId":"W. Europe Standard Time","0":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"1":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"2":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"3":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"4":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"5":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"6":{"S":"6","Sidle":"2","E":"21","Eidle":"2"}}' },
        @{name = "Windows"; id = 2; schedule = '{"TzId":"W. Europe Standard Time","0":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"1":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"2":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"3":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"4":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"5":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"6":{"S":"6","Sidle":"2","E":"21","Eidle":"2"}}' }

# Create a Linebreak Variable for Nice Write-Output
$linebreak = "-" * 150

# Fetch secret from KeyVault
$PATToken = [System.Runtime.InteropServices.Marshal]::PtrToStringBSTR([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR((Get-AzKeyVaultSecret -VaultName my-vault -Name 'DevOpsAgent-PAT').SecretValue))

# Setup security and headers
$DevOpsAccount = "{org}"
$creds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($("user:$($PATToken)")))
$encodedAuthValue = "Basic $creds"
$acceptHeaderValue = "application/json;api-version=3.0-preview"
$headers = @{Authorization = $encodedAuthValue; Accept = $acceptHeaderValue }

# Get All VMSS Pools
$ElasticPoolsUrl = "$($DevOpsAccount)/_apis/distributedtask/elasticpools?api-version=6.1-preview.1"
$ElasticPools = Invoke-RestMethod -Method GET -UseBasicParsing -Headers $headers -Uri $ElasticPoolsUrl

foreach ($AgentPool in $AgentPools) {

    Write-Output "$linebreak"

    Write-Output "Checking AgentPool: $($" 

    $schedule = ConvertFrom-Json $AgentPool.Schedule 

    $poolTz = [System.TimeZoneInfo]::FindSystemTimeZoneById($schedule.TzId)
    $utcCurrentTime = [datetime]::UtcNow
    $poolTzCurrentTime = [System.TimeZoneInfo]::ConvertTimeFromUtc($utcCurrentTime, $poolTz)

    $startTime = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).S)

    $endTime = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).E)

    $startTimeIdle = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).Sidle)

    $endTimeIdle = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).Eidle)

    Write-Output "Identified Start Time: $startTime and End Time: $endTime"

    Write-Output "Identified Start Time Number of idle Agents: $startTimeIdle and End Time Number of idle Agents: $endTimeIdle"

    Write-Output "Checking current config..."

    $ElasticPool = $ElasticPools.value | Where-Object { $_.poolId -eq $ }

    Write-Output $ElasticPool

    if (($poolTzCurrentTime.Hour -ge $startTime) -and ($poolTzCurrentTime.Hour -lt $endTime)) {

        Write-Output "Check if Start Time Number of idle Agents needs to be updated..."
        Write-Output "Expected Number of idle Agents should be: $startTimeIdle"
        Write-Output "Number of idle Agents currently configured is: $($ElasticPool.desiredIdle)"

        if ($startTimeIdle -eq $ElasticPool.desiredIdle) {
            Write-Output "No update is currently needed"
        else {
            Write-Output "Will update number of idle Agents from: $($ElasticPool.desiredIdle) to new value: $startTimeIdle"

            $Body = @"
	"desiredIdle": $startTimeIdle
            $URI = "$($DevOpsAccount)/_apis/distributedtask/elasticpools/$($ElasticPool.poolId)?api-version=6.1-preview.1"

            try {
                $response = Invoke-RestMethod -Uri $URI -headers $headers -Method PATCH -Body $Body -ContentType "application/json" -TimeoutSec 180 -ErrorAction:Stop
            catch {
                write-Error $_.ErrorDetails
            Write-Output $response

    if (($poolTzCurrentTime.Hour -le $startTime) -or ($poolTzCurrentTime.Hour -ge $endTime)) {

        Write-Output "Check if End Time Number of idle Agents needs to be updated..."
        Write-Output "Expected Number of idle Agents should be: $endTimeIdle"
        Write-Output "Number of idle Agents currently configured is: $($ElasticPool.desiredIdle)"

        if ($endTimeIdle -eq $ElasticPool.desiredIdle) {
            Write-Output "No update is currently needed"
        else {
            Write-Output "Will update number of idle Agents from: $($ElasticPool.desiredIdle) to new value: $endTimeIdle"

            $Body = @"
	"desiredIdle": $endTimeIdle
            $URI = "$($DevOpsAccount)/_apis/distributedtask/elasticpools/$($ElasticPool.poolId)?api-version=6.1-preview.1"

            try {
                $response = Invoke-RestMethod -Uri $URI -headers $headers -Method PATCH -Body $Body -ContentType "application/json" -TimeoutSec 180 -ErrorAction:Stop
            catch {
                write-Error $_.ErrorDetails
            Write-Output $response

Same shirt… Different time

From time to time you get to see these amazing transformation pictures with before and after results in very short times.
Of course, it’s possible to transform in a short time but it’s not common, and nothing that you should expect or be disappointed about. Building muscles takes time! (It’s 17 years between the two pictures).

See this as a chance to invest in your body, mind and soul. You’ll get so much more from being physically active. It will help you with:

  • Feeling happier
  • Increasing your energy levels
  • Relaxation and sleep quality
  • Increase your self-confidence
  • Etc.

This blog post is about weightlifting, becoming stronger and eventually gaining bigger muscles.

With that said, let’s get started!

I started working out at the gym as soon as I turned 16. Before that I was not allowed to go to the gym due to age-limit restrictions at my local gym. I’ve been training at the gym regularly since then in addition to other sports and activities (recently A Swedish Classic).

The main reason that made me continue to work out at the gym is the flexibility. You can attend the gym whenever it fits you – time wise!

During this time I have:

  • Tried different approaches with gaining strength and muscle
  • Been injured several times
  • Over-trained (resulting in negative development)
  • Stayed in the same routine
  • Assumed that heavy weights always mean better results
  • Set sky-high goals, going too hard -> burned out
  • Messed with food and sleep

And I’ve learned…learned a lot during this time. I felt that this is the perfect opportunity to share some of my insights and experiences.

I’ll go ahead and start with the most important thing.

It’s spelled: ROUTINE!!!

I have worked out three days a week, four days a week, five days a week and every day of the week. I have found it easiest to work out every day or at least from Monday – Friday as a daily routine in the mornings before going to work. By having this approach, I don’t need to think about whether I should hit the gym or not. I just do it, the same way as I work, eat and sleep. I see this as my personal time to invest in myself and I make sure that everyone else knows that this time is important for me as well.

How do I facilitate this routine?

  • I pack the gym bag the day before and place it in the hallway right by the door
  • I make my breakfast the day before
  • The time between when I wake up (05:30) until the time I leave home is less than 15-20 minutes. I put on my training clothes and have my breakfast on the way to the gym. By keeping this time short, I don’t get the time to feel and think – should I go back to sleep? Should I work out after work instead? Etc.
  • I always pick a gym near to the office where I’ll be working from.

This might sound extreme, but I promise you, if you stick to this routine for 10 weeks, you won’t go back! This setup might not fit everyone. But do it your way, the way that works out best for you.

Once you find your routine (!) don’t break it no matter what. This is often the time when most people stop training. They lose the routine and then it’s hard to get back…

How do you build muscle?

If you’re looking to start building muscle, getting bigger, and becoming stronger, these are the things you need to do:

  1. Lift heavy things (not always)
  2. Eat a diet based on your goals
  3. Get enough rest

You most likely need access to a gym with a free-weight section.

Body weight exercises can be fantastic for weight loss and keeping the muscle you already have, but if you’re serious about weight training you’ll need a gym with a squat rack, bench, barbells, and a spot to do pull ups, chin ups, and dips to be most efficient.

Stay away from machines, focus on dumbbells and barbell exercises.

Weight machines are a great starting point for beginners, but they don’t challenge the muscle as much as free weights do, in addition they could promote bad technique if not monitored. Use free weights instead, as they activate more muscles and promote natural movement.

If you are aiming to work out three days a week or less, focus on the larger muscles. Skip arms – you’ll have them trained anyway. The arms are secondary muscles in all upper body workouts.

How do I set up my training schedule?

I split my program to train each muscle group every week except for legs that I train two days a week. What is important here is that you group your training days in leg exercise, push exercise, pull exercise, and some core work.

So, if you train chest on Monday, don’t train shoulders on Tuesday. Both muscles involve push exercises. Train your back or legs in between.

Muscles grow when they’re resting, so take at least two days off between training the same muscle group to allow your muscles to recover and grow.

My weekly program is the following:

  • Monday: Chest & Triceps
  • Tuesday: Back & Biceps
  • Wednesday: Shoulders & Core (stomach, waist)
  • Thursday: Legs
  • Friday: Arms (Triceps & Biceps) OR Cardio
  • Saturday: Cardio
  • Sunday: Legs

I won’t cover any specific workout program in this blog, but please do reach out to me if you need some guidance depending on what goals you have.

Proper diet to gain muscle

Don’t overdo it. Try to have a balanced diet and eat often – don’t skip meals. Make sure that you get the proper amount of protein during your day. I try to aim at 2-3 grams of protein per kilogram of bodyweight.

And above all, enjoy your meals! If you train hard, don’t worry too much. These calories will help you gain strength and muscles!

Any other weight-lifting tips?

Warm-up before exercising – don’t walk into a gym and start lifting heavy weights directly.

Focus on your position and technique – if you’re doing a bodyweight squat incorrectly, you might develop bad habits and hurt yourself. Start with a light weight and make sure you are doing the exercise properly. There is no shame in starting with just the bar. You can always add more weight next week if this week is to easy.

Don’t over-train – more does not mean better in weightlifting. You don’t need to spend two hours in the gym, you don’t need to do 15 different kinds of chest exercises.

Work your biggest muscles – while everyone likes perfectly sculpted abs and biceps, your chest, legs and back contain some of your body’s biggest muscles. Don’t neglect them.

Get enough sleep – it helps your muscles recover and repair themselves and grow. Most adults need between 7–9 hours of sleep each night.

Have a bedtime snack – eating 30 grams of a protein-packed snack like quark or casein before bed supports the metabolism and muscle recovery.

Follow a routine, have a plan – once again, the best thing you can do is have a plan to follow, stick to it and results will follow!

I hope this blog post inspired you to start working out at the gym. Good luck!