Azure Key Vault is a great resource to keep your secrets out of your repos and to follow the least-privilege methodology. One of the ways to leverage Key Vault through Azure DevOps is to use the “Replace Token” extension that replaces your specified token with the secrets from the Key Vault during the build process. This is a simple process for a standard Key vault, but this post will focus on how to leverage the same feature when you have your Azure Key Vault closed to “All networks” through the Key Vault firewall.

The Problem

As of this post, Azure DevOps is not one the “Trusted Microsoft Services” yet; there is a plan for it to be added, but it is not currently one of those services. This is why enabling the checkpoint above will not help us. We need to specifically identify the IPs Azure DevOps will communicate through and white-list them within the Key Vault, as shown above.

The Solution

To keep this as simple as possible, you will need to leverage a self-hosted agent instead of the Microsoft hosted agent pool. This is due to the fact that with the Microsoft hosted pool, there are lots of IPs that will need to be whitelisted and that list gets updated every Wednesday. It is not feasible to go with that approach. More details on how to set up your own AzDo agent pool can be found here.

With your own hosted agent, you will need to whitelist the IP of these agents and the IP of Azure DevOps services to set a variable group linked to your Key Vault, as seen and detailed below.

Setting up a Variable group that links secrets from Azure Key Vault.

Here we will set up a variable group that links secrets from the Key Vault. Note that the key vault we are linking to has the firewall enabled, and currently, no IPs whitelisted.

  • Go to Pipelines – Library
  • Click on + Variable group
  • Enable the checkbox that enables linking to Azure Key vault
  • You will need to leverage an existing or a new service principal to be able to talk to your Azure Subscription, where the Key Vault resides.
  • After adding the Key Vault, you need to add all the variables you need from the Keyvault into the variable group.
  • If you did not whitelist any IPs, you would run into the error below:

Sample text for query purposes:

“Failed to query service connection API: 'https://hypervmigratio98a5kv.vault.azure.net/secrets?api-version=2016-10-01'. Status Code: 'Forbidden', Response from server: '{"error":{"code":"Forbidden","message":"Client address is not authorized and caller is not a trusted service.\r\nClient address: 40.82.190.38\r\nCaller: appid=2c5a35cb-ef1f-4659-98a2-dca8761c0def;oid=f54370a6-bc53-4e02-8d73-63de4c317b54;iss=https://sts.windows.net/a1a2578a-8fd3-4595-bb18-7d17df8944b0/\r\nVault: hypervmigratio98a5kv;location=canadacentral","innererror":{"code":"ForbiddenByFirewall"}}}'

  • To bypass that error, you need to add a few IPs for the Azure DevOps Services to your key vault firewall. Quick tip: For a quick resolution, you can just add the IP that you are seeing in the error message, which in our case is 40.82.190.38.
  • For a more robust solution, you want to add all the IPs that the DevOps service could use to connect to the Key Vault in that region and those IPs can be found here: https://docs.microsoft.com/en-us/azure/devops/migrate/migration-import?view=azure-devops
  • Search for Azure DevOps services and add all the IPs listed there for your region. You can see in the screenshot below that for the Canadian region, the IP we saw in the message above is listed.

By following the steps above, you have now white-listed the Azure DevOps service and should be able to add all the needed variables into the variable group.

Setting up a sample pipeline to test

The most common task used to leverage Key Vault in our pipeline is the token replace task. Using that task, you can test your configuration while making sure that you are using your own agent pool and that that agent’s public IP address is whitelisted.

In summary, to leverage Azure Key Vaults with firewall enabled within Azure DevOps, you need to whitelist:

  1. Your region’s Azure DevOps Services IP address, found in the link shared above.
    1. This is needed to be able to link your Keyvault secrets with AzDo variables.
  2. Your own hosted agent’s public IP address.
    1. This is needed so that the agent is able to access the key vault to extract those secrets.

Azure DevOps is a great tool as you can see, but when it comes to DevOps, it is not just about tools. A DevOps transformation starts with people followed closely by process then tools.
If you want to hear more about that, join us for our upcoming DevOps Webinar:

How to Transform with DevOps
September 24, 2020
10:00 a.m. – 10:45 a.m. EDT
Register Now

About The Author

Hammad Aslam is an Azure DevOps Consultant at New Signature, assisting customers with their digital transformation efforts within Azure. Hammad believes that your cloud migration journey is not complete without a proper DevOps strategy in-place.  He is a Software Engineering graduate currently working on his MBA. Hammad enjoys acting as an Azure & DevOps advocate while sharing best practices when it comes to DevOps and Architecture. In his free time, Hammad enjoys spending time with his family, binge-watching TV shows, playing soccer and catching up on the NBA.