Agent Pool Networking
ADO pipelines, agent pools are most likely to be containers hosted by Azure Kubernetes services for HMCTS.
Agent Capabilities can help narrow down the vNet hosting the container. First the AZP_Name follows a manual naming convention. Try removing the HMCTS at the beginning and adding VNET to the end. If the agent is actually a Kubernetes container, it will also state the Kubernetes_Port, with HMCTS being 10.0.0.1:443.
Virtual Networks
These are examples of some of the agent pool and vNet names
| Agent Pool | Virtual network |
|---|---|
| hmcts-ss-ptl | ss-ptl-vnet |
| hmcts-ss-ptl | ss-ptl-vnet |
| hmcts-ss-ptlsbox | ss-ptlsbox-vnet |
| hmcts-ss-ptlsbox | ss-ptlsbox-vnet |
Palo Alto Firewalls
Traffic from the agent pool to the Azure resource, flows through the Palo Alto if the infrastructure has been correctly configured.
Use the Palo Alto monitor utility to view the traffic flows. If there is nothing, the ADO agent is not sending traffic to Azure. Review the agent pool vNet security rules, routing and DNS configuration. Palo Alto policy may DENY the traffic, requiring a rule change. If the traffic is coming into Azure from the agent pool, use Azure portal Network Watch to troubleshoot.
ADO Connections to HMCTS Environments
HMCTS operates 3 main Azure environments (SBOX, NONPROD and PROD), which are isolated from each other. ADO agent pools are connected to only ONE of these environments. If your pipeline is reporting network errors and you are not observing ADO traffic in the Palo Alto. First suspect the wrong agent pool has been selected. Then start investigating the other links in the chain.
Agent Routing
If you vNET is new, you may need to update the route table aks-ptl-route-table, to enable the agents in K8s to reach your resources.
environment / 01-network /
Private DNS Zones and Private End Points
To enable ADO agents to resolve private DNS zones, each zone needs to be linked to the vNet contacting the Agent Pool.
Note once merged, this pipeline needs to be manually run to apply your changes. The network stage is early in the pipeline and can be selected via the “stages to run” option. Be sure you have selected the correct Environment for your ADO agent network.
Traffic will also have to be enabled between the agent pool vNet and the private endpoint vNet, by linking the vNets
Azure-Private-DNS-Private-Link
Public Network Services or Private
Some Azure resources (Storage Accounts, Databases, Purview, etc) can be configured as either public or private, with private creating additional requirements for agent pool access.
Example of private storage account
Pipeline will fail with an AuthorizationFailure error if only private network access is configured and access has not been granted.
RAW Error Message
2023-07-25T16:31:31.0612142Z Error: containers.Client#GetProperties: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailure" Message="This request is not authorized to perform this operation.\nRequestId:19ad4843-f01e-007b-3515-bfe0bc000000\nTime:2023-07-25T16:31:30.6786267Z"
2023-07-25T16:31:31.0613295Z
2023-07-25T16:31:31.0613887Z with module.sftp_cft_storage.azurerm_storage_container.container["dlrm"],
2023-07-25T16:31:31.0615351Z on .terraform/modules/sftp_cft_storage/containers.tf line 2, in resource "azurerm_storage_container" "container":
2023-07-25T16:31:31.0616205Z 2: resource "azurerm_storage_container" "container"
2023-07-25T16:31:31.0616683Z
Terraform code that adds agent pool vNets to the storage account: link
module "storage" {
for_each = { for account in local.flattened_storage_accounts : "${account.key}-${account.account_key}" => account }
source = "github.com/hmcts/cnp-module-storage-account?ref=master"
env = var.env
storage_account_name = "${each.value.account_key}${var.env}stor"
resource_group_name = azurerm_resource_group.this[each.value.key].name
location = azurerm_resource_group.this[each.value.key].location
account_kind = "StorageV2"
account_tier = each.value.account.tier
account_replication_type = each.value.account.replication_type
enable_hns = each.value.account.enable_hns
enable_sftp = each.value.account.enable_sftp
enable_nfs = each.value.account.enable_nfs
private_endpoint_subnet_id = "/subscriptions/${data.azurerm_subscription.this.subscription_id}/resourceGroups/${data.azurerm_resource_group.vnet_rg.name}/providers/Microsoft.Network/virtualNetworks/${data.azurerm_virtual_network.vnet.name}/subnets/${module.networking[each.value.key].subnet_names["${data.azurerm_virtual_network.vnet.name}-${each.value.key}"]}"
containers = each.value.account.containers
sa_subnets = [
data.azurerm_subnet.ssptl-00.id,
data.azurerm_subnet.ssptl-01.id
]
team_name = "Platform Operations"
team_contact = "#dtspo-orange"
common_tags = module.ctags.common_tags
}