Asif Mithawala

Background

Azure Pipelines announced the general availability of scale-set agents in september 2020.

Azure virtual machine scale set agents are a form of self-hosted agents that can be autoscaled to meet your demands. This elasticity reduces your need to run dedicated agents all the time. Unlike Microsoft-hosted agents, you have flexibility over the size and the image of machines on which agents run.

Read more here: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops

The top benefits of using Scale-set Agents are:

• Out-of-the-box auto scaling
• Automatically tear down virtual machines after every use

You don’t longer need to worry about a build that leaves credentials or other sensitive information behind in an Agent.

Or a build that messes the config of the Agent. Well, you get a new fresh Agent every time.

The Problem

One of the settings, that you can set is how many “Number of agents to keep on standby”

Azure Pipelines will automatically scale-down the number of agents, but will ensure that there are always this many agents available to run new jobs. If you set this to 0, for example to conserve cost for a low volume of jobs, Azure Pipelines will start a VM only when it has a job.

While this is a good feature. You would probably like to have a more granular schedule.

Now you’re thinking, “I can just set it to 0 and the Agent Pool will scale once needed”.

Well keep in mind that it takes time for an agent to spin up. The Scale set instance actually has to get deployed. And this can take a while depending on SKU and Size of the image etc.

Now in a critical business, time = money. So you wan’t to make sure that you have a minimum number of agents on standby during business-hours. And then maybe a lower number during off-hours/night.

As for now this is not supported in Azure DevOps. So I wrote a PowerShell Script (see below), that is scheduled to run twice a day. 

The Solution

The Script will loop the Scale Set Agent Pools, and based on the current time, it will set the appropriate “number of agents to keep standby” based on a defined schedule.

The recommended way of setting up this orchestrator is via a Scheduled Build Pipeline.

The pre-requsite for this Orchestrator is the following:

• You need to have PAT Token for a user that is administrator on the actual Agent pool
• Preferably you want to store your PAT in a Key vault or as protected build variable

Configure your Build Pipeline:

Set your Build Schedule:

Use the script below in your Build Pipeline.

Make sure to replace the values accordingly

<#
    .SYNOPSIS
                    Set "number of agents to keep standby" on Self-Hosed Azure DevOps Agent Pool
    .DESCRIPTION
                    The Script will loop the Scale Set Agent Pools, and based on the current time, it will set the appropriate "number of agents to keep standby" based on a defined schedule.
    .AUTHOR
                    Asif Mithawala

    .VERSION HISTORY
    Version 1 - 2021-03-04
        First version of script created. The Script will loop the Scale Set Agent Pools, and based on the current time, it will set the appropriate "number of agents to keep standby" based on a defined schedule.
        0 is = sunday and saturday is = 6 in the schedule
#>
param(
    [array]$AgentPools = @(
        @{name = "Ubuntu"; id = 1; schedule = '{"TzId":"W. Europe Standard Time","0":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"1":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"2":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"3":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"4":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"5":{"S":"6","Sidle":"10","E":"21","Eidle":"2"},"6":{"S":"6","Sidle":"2","E":"21","Eidle":"2"}}' },
        @{name = "Windows"; id = 2; schedule = '{"TzId":"W. Europe Standard Time","0":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"1":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"2":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"3":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"4":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"5":{"S":"6","Sidle":"2","E":"21","Eidle":"2"},"6":{"S":"6","Sidle":"2","E":"21","Eidle":"2"}}' }
    )
)

# Create a Linebreak Variable for Nice Write-Output
$linebreak = "-" * 150

# Fetch secret from KeyVault
$PATToken = [System.Runtime.InteropServices.Marshal]::PtrToStringBSTR([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR((Get-AzKeyVaultSecret -VaultName my-vault -Name 'DevOpsAgent-PAT').SecretValue))

# Setup security and headers
$DevOpsAccount = "https://dev.azure.com/{org}"
$creds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($("user:$($PATToken)")))
$encodedAuthValue = "Basic $creds"
$acceptHeaderValue = "application/json;api-version=3.0-preview"
$headers = @{Authorization = $encodedAuthValue; Accept = $acceptHeaderValue }

# Get All VMSS Pools
$ElasticPoolsUrl = "$($DevOpsAccount)/_apis/distributedtask/elasticpools?api-version=6.1-preview.1"
$ElasticPools = Invoke-RestMethod -Method GET -UseBasicParsing -Headers $headers -Uri $ElasticPoolsUrl

foreach ($AgentPool in $AgentPools) {

    Write-Output "$linebreak"

    Write-Output "Checking AgentPool: $($AgentPool.name)" 

    $schedule = ConvertFrom-Json $AgentPool.Schedule 

    $poolTz = [System.TimeZoneInfo]::FindSystemTimeZoneById($schedule.TzId)
    $utcCurrentTime = [datetime]::UtcNow
    $poolTzCurrentTime = [System.TimeZoneInfo]::ConvertTimeFromUtc($utcCurrentTime, $poolTz)

    $startTime = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).S)

    $endTime = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).E)

    $startTimeIdle = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).Sidle)

    $endTimeIdle = [int]::Parse($schedule.($poolTzCurrentTime.DayOfWeek.value__).Eidle)

    Write-Output "Identified Start Time: $startTime and End Time: $endTime"

    Write-Output "Identified Start Time Number of idle Agents: $startTimeIdle and End Time Number of idle Agents: $endTimeIdle"

    Write-Output "Checking current config..."

    $ElasticPool = $ElasticPools.value | Where-Object { $_.poolId -eq $AgentPool.id }

    Write-Output $ElasticPool

    if (($poolTzCurrentTime.Hour -ge $startTime) -and ($poolTzCurrentTime.Hour -lt $endTime)) {

        Write-Output "Check if Start Time Number of idle Agents needs to be updated..."
        Write-Output "Expected Number of idle Agents should be: $startTimeIdle"
        Write-Output "Number of idle Agents currently configured is: $($ElasticPool.desiredIdle)"

        if ($startTimeIdle -eq $ElasticPool.desiredIdle) {
            Write-Output "No update is currently needed"
        }
        else {
            Write-Output "Will update number of idle Agents from: $($ElasticPool.desiredIdle) to new value: $startTimeIdle"

            $Body = @"
{
	"desiredIdle": $startTimeIdle
}
"@
            $URI = "$($DevOpsAccount)/_apis/distributedtask/elasticpools/$($ElasticPool.poolId)?api-version=6.1-preview.1"

            try {
                $response = Invoke-RestMethod -Uri $URI -headers $headers -Method PATCH -Body $Body -ContentType "application/json" -TimeoutSec 180 -ErrorAction:Stop
            }
            catch {
                write-Error $_.ErrorDetails
            }
            Write-Output $response
        }
    }

    if (($poolTzCurrentTime.Hour -le $startTime) -or ($poolTzCurrentTime.Hour -ge $endTime)) {

        Write-Output "Check if End Time Number of idle Agents needs to be updated..."
        Write-Output "Expected Number of idle Agents should be: $endTimeIdle"
        Write-Output "Number of idle Agents currently configured is: $($ElasticPool.desiredIdle)"

        if ($endTimeIdle -eq $ElasticPool.desiredIdle) {
            Write-Output "No update is currently needed"
        }
        else {
            Write-Output "Will update number of idle Agents from: $($ElasticPool.desiredIdle) to new value: $endTimeIdle"

            $Body = @"
{
	"desiredIdle": $endTimeIdle
}
"@
            $URI = "$($DevOpsAccount)/_apis/distributedtask/elasticpools/$($ElasticPool.poolId)?api-version=6.1-preview.1"

            try {
                $response = Invoke-RestMethod -Uri $URI -headers $headers -Method PATCH -Body $Body -ContentType "application/json" -TimeoutSec 180 -ErrorAction:Stop
            }
            catch {
                write-Error $_.ErrorDetails
            }
            Write-Output $response
        }
    }
}