Azure ARM Templates and Testing with Pester

I have been recently working with Azure Resource Manager (ARM) templates and using Pester for testing. My friend Sunny showed me a very basic example of using Pester for testing an ARM template that is available as part of a template for VM Scale Sets managed by Azure Automation DSC. The pester test script provided with this template does a few things:

• Tests that an azuredeploy.json file exists
• Tests that an azuredeploy.parameters.json file exists
• Tests that a metadata.json file exists
• Tests that the azuredeploy.json has the expected properties

This is a good start but in this post I will walk through some additional types of tests that you can run and also some gotchas I found with the example in the Azure Quickstart templates Github repo.

Checking for expected properties in a JSON file

The example in the Azure Quickstart templates Github repo uses the code below to check for expected properties:

It "Converts from JSON and has the expected properties" {
            $expectedProperties = '$schema',
                                  'contentVersion',
                                  'parameters',
                                  'variables',
                                  'resources',                                
                                  'outputs'
            $templateProperties = (get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue) | Get-Member -MemberType NoteProperty | % Name
            $templateProperties | Should Be $expectedProperties
        }

There is a problem with this code in that the order in which the properties are returned through the line with the ConvertFrom-Json cmdlet may not match the order used by the expectedProperties variable. This issue can be solved by simply sorting the properties when you store them in the expectedProperties variable and also after the call to Get-Member.

It "Converts from JSON and has the expected properties" {
             $expectedProperties = '$schema',
                                  'contentVersion',
                                  'parameters',
                                  'variables',
                                  'resources',                                
                                  'outputs' | Sort-Object
            $templateProperties = (get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue) | Get-Member -MemberType NoteProperty | Sort-Object -Property NoteProperty | % Name
            $templateProperties | Should Be $expectedProperties
        }

Dealing with multiple parameter files

Another shortcoming of the example is that it assumes only one parameter file per template, so how do you deal with multiple parameter files? e.g. azuredeploy.parameters.dev.json,  azuredeploy.parameters.test.json

First we need to modify the test that checks for the existence of parameter files to allow for multiple files like so:

It "Has a parameters file" {        
            "$here\azuredeploy.parameters*.json" | Should Exist
}

Next we need to deal with multiple parameter files when checking if parameter files have the expected properties. To do this at the top of the test script we create an array hashes of all the parameter files.

$ParameterFileTestCases = @()
ForEach( $File in (dir "$here\azuredeploy.parameters*.json" | select -ExpandProperty Name) )
{
    $ParameterFileTestCases += @{ ParameterFile = $File }
}

Then we put the tests for parameter files in a separate context block and use TestCases parameter for a It block.

 Context "Parameter File Syntax" {
        It "Parameter file  contains the expected properties" -TestCases $ParameterFileTestCases {
            Param( $ParameterFile )
            $expectedProperties = '$schema',
                                  'contentVersion',
                                  'parameters' | Sort-Object
            $templateFileProperties = (get-content "$here\$ParameterFile" | ConvertFrom-Json -ErrorAction SilentlyContinue) | Get-Member -MemberType NoteProperty | Sort-Object -Property NoteProperty | % Name
            $templateFileProperties | Should Be $expectedProperties
        }
    }

Testing a resource has the expected properties

We can extend the method used to check that a azuredeploy.json template file has the expected resources to also check that the resource has the expected properties. In the example below, we first check that a the azuredeploy.json contains a virtual network resource, then we check the virtual network has properties for address space, DHCP options and subnets.

# Check that the template contains Microsoft.Network/virtualNetworks resource type
It "Creates the expected resources" {
    $expectedResources = 'Microsoft.Network/virtualNetworks'
    $templateResources = (get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue).Resources.type
    $templateResources | Should Be $expectedResources
}
        
# Check that the VNET defined in the template has properties that define the address space, dhcpOptions (which is used to configure DNS servers) and subnets
It "Has a VNET with the expected properties" {
    $expectedVNETProperties = 'addressSpace',
                            'dhcpOptions',
                            'subnets' | Sort-Object
    $templateVNETProperties = ((get-content "$here\azuredeploy.json" | ConvertFrom-Json -ErrorAction SilentlyContinue).Resources | ? { $_.type -eq 'Microsoft.Network/virtualNetworks' }).properties | gm -MemberType NoteProperty | Sort-Object -Property Name | % Name
    $templateVNETProperties | Should Be $expectedVNETProperties
}

Validating Templates

Another test we can add as part of our Pester testing script is to use the Test-AzureResourceGroupDeployment cmdlet to validate the template with each parameter file.  This requires creating a resource group.

When creating a resource group you should try to randomise part of the resource group name to avoid clashes, so for example you could use something like:

$ShortGUID = ([system.guid]::newguid().guid).Substring(0,5)
$TempValidationRG = "$ShortGUID-Pester-Validation-RG"

Here we use Pester-Validation-RG to easily identify what the purpose of the resource group is. We then prefix this with the first 5 characters from a GUID - to avoid clashes in the event you have multiple users or automated tests running at the same time in the same subscription.

We can then use the BeforeAll block to create the resource group before running the tests and the AfterAll block to delete it after all tests have run.

BeforeAll {
    New-AzureRmResourceGroup -Name $TempValidationRG -Location $Location
}

AfterAll {
    Remove-AzureRmResourceGroup $TempValidationRG -Force
}

We then run Test-AzureResourceGroupDeployment with the template and each parameter file in turn uses the TestCases parameter for the It block.

Context "Template Validation" {
        
    It "Template $here\azuredeploy.json and parameter file  passes validation" -TestCases $ParameterFileTestCases {
        Param( $ParameterFile )
        # Complete mode - will deploy everything in the template from scratch. If the resource group already contains things (or even items that are not in the template) they will be deleted first.
        # If it passes validation no output is returned, hence we test for NullOrEmpty
        $ValidationResult = Test-AzureRmResourceGroupDeployment -ResourceGroupName $TempValidationRG -Mode Complete -TemplateFile "$here\azuredeploy.json" -TemplateParameterFile "$here\$ParameterFile"
        $ValidationResult | Should BeNullOrEmpty
    }
}

There are few things to note with this:

• It obviously requires that we create a resource group - because although the Test-AzureResourceGroupDeployment cmdlet doesn't actually create the resources in the template it requires a resource group in order to use it.
• While there is an AfterAll block block that deletes the temporary resource group that is created to validate the template, if you Ctrl-C the test script or there is some other problem e.g. such as a corrupted test group stack it may not clean up your temporary resource group.
• Note the deployment of the template can still fail - this simply checks that the schema for each of the resources is correct and that the parameter file is correct. Deployments can still fail for other reasons and the parameter file may still be wrong e.g. we specify a subnet address prefix in the parameter file that does not fall within the VNET address spaces
• This will increase the time it takes for the tests to run because creating and deleting a resource group, even if it's empty takes a little time.