Tools and configuration
CPP has separate delivery tooling and environment configuration from the Cloud Native Platform. Start with the existing repository that owns the concern instead of creating a second source of truth.
| Concern | Source of truth |
|---|---|
| Service source | The repository named by the service owner. Existing application repositories are not all in one source-control system. |
| Central image builds for established context services | cpp-docker-images |
| CCM AKS image versions | cpp.pipeline/aks-pipeline.versions.yml (requires VPN and Azure credentials to access) |
| CCM AKS deployment | cpp-aks-deploy |
| Reusable Azure DevOps pipeline steps | cpp-azure-devops-templates |
| Reusable Kubernetes charts | cpp-helm-chart |
| AKS operational automation | cpp-aks-ops |
| Virtual-machine automation | cpp-automation-ansible |
| Legacy Terraform automation | cpp-automation-terraform |
| Build-agent and virtual-machine images | cpp-packer-pipeline |
Deploying applications to Crime AKS
CPP applications configured for this pipeline, including CCM can be deployed to Crime AKS clusters using Helmsman through the cpp-aks-deploy pipeline.
Application entries and common, environment and stack values for deployment as well as pipeline configuration are maintained in the cpp-aks-deploy repository.
The pipeline is defined in aks-deploy.yaml.
cpp-aks-deploy pipeline supports dev, sit, ste, nft, prp, prd, and prx, selected through its environment parameter.
The deploy-service determines whether the pipeline runs the Helmsman apply step after planning. A plan-only run can still perform earlier setup actions, including namespace-related Helm upgrades, so while it can be useful as a preview before applying your changes, do not treat it as entirely side-effect free.
The pipeline selects its live or non-live agent pool and credential variable groups from its own environment variable files. It also uses shared steps from cpp-azure-devops-templates. Credentials are not pipeline parameters and do not need to be provided manually; see CPP Azure DevOps for the shared-library and service-connection model.
Some established CPP components will use different Azure DevOps or Jenkins jobs.
Deploying services to virtual machines, Alfresco or AMP is done through completely separate deployment jobs and not cpp-aks-deploy - refer to relevant component release runbook.
Live deployments are governed by Crime Release Management.
CCM applications have two deployment controllers: Helmsman through cpp-aks-deploy, and Flux through cpp-flux-config. Confirm which controller manages the service in the target environment and stack before changing deployment configuration:
apps/base/kustomization.yamlincludesidam-integration-servicein every configured environment, including PRD.apps/environments/ste/kustomization.yamladdscourtlistpublishing-service,listingcourtscheduler-serviceandgov-uk-notify-gateway-servicefor STE only.helmsman_set_default_variables.shforce-disables court-list publishing and listing-court scheduler on STE. It also disables court-list publishing on DEV CCM 01–10, SIT CCM 01, NFT CCM 01–02, PRP CCM 01 and PRD CCM 01.
The court-list publishing guard names Flux as its controller on those non-STE stacks, but the current Flux environment manifests include that service only in STE. Resolve that discrepancy with Platform Operations before deploying it on a non-STE stack. Do not override the guard or assume that disabling Helmsman proves a Flux release exists.
Trace the service through apps/environments, apps/overlays/stacks and clusters in cpp-flux-config to identify its target namespace and cluster. A service directory in apps/base/services alone does not select a deployment target. Follow starting a new component for the two routes.
Helmsman
Helmsman is a Helm Charts (k8s applications) as Code tool which allows you to automate the deployment/management of your Helm charts from version controlled code.
Its just a wrapper around helm and requires helm and helm-diff plugin for it work.
Helmsman gives a way to deploy and manage all the apps running in a namespace as a stack and maintain the state in version control. Helmsman provides this with something called Desired state file (DSF).
We can define multiple namespaces and apps along with their charts, values files and docker tags in DSF. When we apply the DSF helmsman will go though the desired state and check with what’s already deployed in the cluster if it sees a difference it will only apply the changes to a particular app and skip all other apps.
The DSF can be defined in toml or yaml files. We are using toml format just because its not sensitive with spaces and we can do our own way of indentation.
The functionalities we are currently using:
- Environment variables: Helmsman supports environment variables in DSF so we can templatise it.
- Plan before apply: We can do a plan-only Helmsman run to show what its going to do before we apply the changes.
- Protected flag: If set to true on an app it will prevent its deletion - this protects against an accidental mistake in the DSF, Helmsman will not delete the app.
- Parallel deploy: Helmsman supports parallel deployments and we can set number of parallel deployments on command line at deployment time.
Example DSF plain: helmsman.toml
context = devccm12
[metadata]
environment = devccm12
[settings]
slackWebhook = <HELMSMAN_SLACK_WEBHOOK>
[namespaces]
[namespaces.ns-dev-ccm-12]
protected = false
[apps]
[apps.<application>]
name = "<application>"
namespace = "<namespace>"
enabled = true
chart = "oci://<chart>"
version = "0.1.16"
valuesFiles = [
"./ansible/group_vars/<application>-service_values.yml",
"./ansible/group_vars/$environment/<application>-service_values.yml",
"./ansible/group_vars/$environment/$stack/<application>-service_values.yml"
]
wait = true
[apps.<application>-service.setString]
"image.tag" = "211601_7.0.8"
"image.pullPolicy" = "Always"
...
Example DSF templated: helmsman.toml:
context = "$stack"
[metadata]
environment = "$stack"
[settings]
slackWebhook = "$HELMSMAN_SLACK_WEBHOOK"
[namespaces]
[namespaces."$namespace"]
protected = $ns_protected
[apps]
[apps.<application>-service]
name = "<application>-service"
namespace = "$namespace"
enabled = $<application>_service_enabled
chart = "oci://$repo_url/charts/<chart-name>"
version = "$chart_version"
valuesFiles = [
"./ansible/group_vars/<application>-service_values.yml",
"./ansible/group_vars/$environment/<application>-service_values.yml",
"./ansible/group_vars/$environment/$stack/<application>-service_values.yml"
]
wait = $wait_for_service_to_be_up
[apps.<application>-service.setString]
"image.tag" = "$<application>_service_image_tag"
"image.pullPolicy" = "Always"
...
You can read more about Deploying to Crime AKS and Helmsman here.
Running and manual approval gates
Merging to main will not automatically run this pipeline; raising a PR will run pre-commit stage only - this validates the image-tag configuration and that selected application configurations are rendered correctly.
Manual runs (non pull-request) will run a Manual Approval job before executing the rest of the pipeline, this is essentially a manual approval gate to ensure changes have been reviewed and approved.
Current non-live configuration makes manual approval optional - it will wait for a minute for anyone to approve, once minute has passed it will resume running rest of the pipeline.
For live the configuration requires that a member of CPP DevOps SC approver group approves the run, if this is not done within five minutes the run is rejected and rest of the pipeline is not executed.
Docker images
Docker images for CPP apps and AKS-based ADO agents live in cpp-docker-images repository, they are built from docker-images pipeline.
This pipeline will build and publish an image into the Crime Azure Container registry, it can also optionally import a base image if specified in the parameters.
Helm charts
Helmsman deployment described in Deploying applications to Crime AKS section consumes helm charts defined in cpp-helm-chart repository.
For an application-only change, select an existing chart and put the service configuration in cpp-aks-deploy.
Charts in cpp-helm-chart will only need to be modified when the application needs a reusable Kubernetes capability that the selected chart does not provide, individual service environment values should not be stored there.
Crime AKS application secrets
Crime AKS has two secret integrations - they are selected per-service:
- HashiCorp Vault: Ansible Hashicorp Vault plugin is used to retrieve secrets using
lookup('vault', ...)expressions in cpp-aks-deploy values files, they read secrets while the deployment values are rendered. The ADO pipeline receives Vault connection details from its live or non-live variable group. - Azure Key Vault: a service with
secretProvider.create: truedeclares Key Vault secret mappings in its values. The shared springboot-app and wildfly-app charts create a Secrets Store CSISecretProviderClassso the pod can receive the mapped secrets at runtime.
Existing services (such as CCM) use both mechanisms - required secret, identity and access must exist before deployment.
When working with secrets you should only store the lookup expressions or key vault mappings in the code - never actual secret values.