Content

Objective
Create Repos and add JMeter script
Create a service connection
Create a key vault
Create a Variable Group in Azure DevOps
Steps to Setup Release Pipeline

Objective

We are using JMeter to test how an application handles concurrent users. We want to run this JMeter script automatically at regular intervals without human involvement. When we use Azure Pipelines for Continuous testing, there is a reduction in the risk of errors that can happen when a human performs a process repetitively. Also, this helps us to detect issues sooner by running tests automatically.

Diagram

Scenario

Credentials should be stored in the Azure Key Vault and that is where it should be retrieved each time the load testing runs. In this case, there is access token for the API calls, which should be generated every 24 hours.

Create Repos and add JMeter script

Upload the JMeter script that has worked in GUI mode. Curl command to generate the access token is embedded in the script. The client id and client secret should not be hard coded in the script and are substituted with a variable name. Inside the script, the portion where client id/secret is assigned is replaced by a function with variable name (substitute of client id/secret) as its parameter.

Create Service Connection

To connect Azure Pipelines to Azure Key Vault (in Azure Portal), we create service connection and specify the service principal. “Service Connection” is a configuration that securely stores information required to connect and authenticate to external services or resources. The service principal specifies the resources and the access levels that are available over the connection.

Create a key vault and Grant Azure DevOps Access to Key Vault

In Azure Portal, create a key vault by providing a name, subscription, resource group and location for the vault.  To access the data from the vault, provide permissions to the above service principal for authentication in the pipeline. Once the service is provisioned, select the key vault and add the new secret.

Create a Variable Group in Azure DevOps

To integrate an Azure Key Vault with your Azure DevOps pipeline using a service account, and create a variable group for secrets, follow these steps:

In Azure DevOps, go to ‘Pipelines’ > ‘Library’.

Click ‘+ Variable Group’. Name your variable group.

Link to an Azure Key Vault and select your subscription and the Key Vault.

Select ‘Authorize’ to connect Azure DevOps to your Key Vault.

Choose the secrets from your Key Vault to add to the variable group.

Steps to Setup Release Pipeline

To create the release pipeline, go to Releases page from the left menu of Azure DevOps then click on New pipeline button.
Select “Azure Repos Git” as source, Team project, Repository & Branch details as required and click Continue. After that select “Empty Job” as template. Specify Pipeline Name & Agent Pool.
Add an artifact & connect Azure repository – Empty Job.
When release creation is opened, Azure automatically adds a stage. Add tasks in the stage

Add JMeter – Install JMeter.
Add CommandLine. (CommandLine should have instructions to run the selected JMeter file, link the ClientID/secret secured in Azure Key Vault to the script and store the results of the load test in downloadable option.)
Add Upload Release Artifact

Save pipeline. Configure schedules on pipelines. Scheduling can be done using the schedule option near the artifact.
Create pipeline release and execute
Download logs and check the report.

JMeter script (In the above figure, this refers to place marked as “2”)

Bearer ${AuthAccessToken}

client_id=${__P(variable1,)}

client_secret=${__P(variable2,)}

Azure DevOps release Pipeline-> Stages -> Tasks -> CommandLine Script (In the above figure, this refers to place marked as “4”)

jmeter -Jvariable1=$(ABCD_ClientId) -Jvariable2=$(ABCD_ClientSecret) -n -t

$(System.DefaultWorkingDirectory)/path_to_dir/loadTestscript.jmx -l

$(System.DefaultWorkingDirectory)/path_to_dir /LoadTestResults.jtl -e -o

$(System.DefaultWorkingDirectory)/path_to_dir/LoadTestReports

Problem:

Ours is a special scenario where the secrets are to be stored in AzureKeyVault instead of ADO variables.

While there is an online solution for automating JMeter load test in AzureDevOps, solution for the above specific scenario is not present over online. Online sources suggest using ApacheGroovy functions inside JMeter script for encryption when using ADO variable for secret storage.  This did not work in our Windows environment plus we need credentials to be stored in Azure Key Vault.

Why do we insist on Azure Key Vault? Azure Key Vault offers better security and easy access management, and ADO variable groups offer the benefit of reusability of variable and easy maintenance.

Solution:

This solution was done by a combination of the method to call the variables (to hold secret) in 2 places:

inside JMeter script
Azure DevOps release Pipeline-> Stages -> Tasks -> CommandLine Script

​Content

Objective
Create Repos and add JMeter script
Create a service connection
Create a key vault
Create a Variable Group in Azure DevOps
Steps to Setup Release Pipeline

 
Objective
We are using JMeter to test how an application handles concurrent users. We want to run this JMeter script automatically at regular intervals without human involvement. When we use Azure Pipelines for Continuous testing, there is a reduction in the risk of errors that can happen when a human performs a process repetitively. Also, this helps us to detect issues sooner by running tests automatically.
 
Diagram
 
 
Scenario
Credentials should be stored in the Azure Key Vault and that is where it should be retrieved each time the load testing runs. In this case, there is access token for the API calls, which should be generated every 24 hours.
 
Create Repos and add JMeter script
Upload the JMeter script that has worked in GUI mode. Curl command to generate the access token is embedded in the script. The client id and client secret should not be hard coded in the script and are substituted with a variable name. Inside the script, the portion where client id/secret is assigned is replaced by a function with variable name (substitute of client id/secret) as its parameter.
 
Create Service Connection
To connect Azure Pipelines to Azure Key Vault (in Azure Portal), we create service connection and specify the service principal. “Service Connection” is a configuration that securely stores information required to connect and authenticate to external services or resources. The service principal specifies the resources and the access levels that are available over the connection.
 
Create a key vault and Grant Azure DevOps Access to Key Vault
In Azure Portal, create a key vault by providing a name, subscription, resource group and location for the vault.  To access the data from the vault, provide permissions to the above service principal for authentication in the pipeline. Once the service is provisioned, select the key vault and add the new secret.
 
Create a Variable Group in Azure DevOps
To integrate an Azure Key Vault with your Azure DevOps pipeline using a service account, and create a variable group for secrets, follow these steps:
In Azure DevOps, go to ‘Pipelines’ > ‘Library’.
Click ‘+ Variable Group’. Name your variable group.
Link to an Azure Key Vault and select your subscription and the Key Vault.
Select ‘Authorize’ to connect Azure DevOps to your Key Vault.
Choose the secrets from your Key Vault to add to the variable group.
 
Steps to Setup Release Pipeline

To create the release pipeline, go to Releases page from the left menu of Azure DevOps then click on New pipeline button.
Select “Azure Repos Git” as source, Team project, Repository & Branch details as required and click Continue. After that select “Empty Job” as template. Specify Pipeline Name & Agent Pool.
Add an artifact & connect Azure repository – Empty Job.
When release creation is opened, Azure automatically adds a stage. Add tasks in the stage

Add JMeter – Install JMeter.
Add CommandLine. (CommandLine should have instructions to run the selected JMeter file, link the ClientID/secret secured in Azure Key Vault to the script and store the results of the load test in downloadable option.)
Add Upload Release Artifact

Save pipeline. Configure schedules on pipelines. Scheduling can be done using the schedule option near the artifact.
Create pipeline release and execute
Download logs and check the report.

JMeter script (In the above figure, this refers to place marked as “2”)
Bearer ${AuthAccessToken}
client_id=${__P(variable1,)}
client_secret=${__P(variable2,)}
 
Azure DevOps release Pipeline-> Stages -> Tasks -> CommandLine Script (In the above figure, this refers to place marked as “4”)
jmeter -Jvariable1=$(ABCD_ClientId) -Jvariable2=$(ABCD_ClientSecret) -n -t
$(System.DefaultWorkingDirectory)/path_to_dir/loadTestscript.jmx -l
$(System.DefaultWorkingDirectory)/path_to_dir /LoadTestResults.jtl -e -o
$(System.DefaultWorkingDirectory)/path_to_dir/LoadTestReports
 
Problem:
Ours is a special scenario where the secrets are to be stored in AzureKeyVault instead of ADO variables.
While there is an online solution for automating JMeter load test in AzureDevOps, solution for the above specific scenario is not present over online. Online sources suggest using ApacheGroovy functions inside JMeter script for encryption when using ADO variable for secret storage.  This did not work in our Windows environment plus we need credentials to be stored in Azure Key Vault.
Why do we insist on Azure Key Vault? Azure Key Vault offers better security and easy access management, and ADO variable groups offer the benefit of reusability of variable and easy maintenance.
Solution:
This solution was done by a combination of the method to call the variables (to hold secret) in 2 places:

inside JMeter script
Azure DevOps release Pipeline-> Stages -> Tasks -> CommandLine Script  Read More

​