Skip to main content

Current State (Azure DevOps + Confluence)

Current-state reference for the automated pipelines that start, stop, idle-suspend and cost-report on CRIME (CPP) non-production environments.

Contents

Overview

Non-production CRIME (CPP) environments cost money whether or not anyone is using them. The platform has a family of scheduled Azure DevOps pipelines that:

  1. Turn environments off automatically when idle, out of hours, or not tracked (“orphaned”), to save cost.
  2. Turn environments on on a schedule or on demand.
  3. Notify owners by email/Slack about usage and cost so they can intervene.
  4. Clean up environments nobody is tracking any more.

Two infrastructure models are in play for CPP stacks:

  • IAAS — VM-based stacks (Ansible over SSH against Azure VMs): Wildfly app servers, Artemis brokers, HAProxy load balancers, Elasticsearch.
  • AKS — Kubernetes-based stacks (kubectl/az against AKS clusters): deployments/statefulsets scaled via replica count, driven off cluster/namespace labels (stack=<name>).

Some stacks are fully migrated to AKS (“Full AKS”), others still run partly or fully on VMs — the tooling detects this per-stack and runs only the relevant path.

Source of truth: the Confluence page

Almost everything is driven off one Confluence page (https://tools.hmcts.net/confluence/display/EA/EA+Environment+management, page ID 1394672908). Its table lists, per stack: owner, status (incl. “Full AKS”), default/custom schedule icons, startup/shutdown time, weekend-running flag, and an optional start/end date window for temporary overrides.

The actual startup/shutdown behaviour for a given stack is configured in Confluence, not in code — the pipelines just read that page and act on it. A local backup file (confluenceoutputfile.txt, held in Azure Blob Storage) is used as a fallback if Confluence is unreachable.

Repo map

Repo Role
cpp-aks-ops Owns the scheduled/orchestrator pipeline definitions (the ones with schedules: blocks) and the Python scripts that talk to Confluence, Prometheus, Kubernetes, Slack and Azure Cost Management.
cpp-automation-ansible Owns the Ansible playbooks that actually start/stop IAAS VM services, plus the “Controlled Environment Startup/Shutdown” pipeline wrappers around them.
cpp-azure-devops-templates Shared pipeline step/template library used by the above — cluster detection, AKS scale up/down steps, Ansible step wrappers, and the two pipelines that combine PAAS+IAAS+AKS into one “full environment” start/stop (full-environment-start.yaml, full-environment-shutter.yaml).
cpp-aks-deploy App deployment pipeline (aks-deploy.yaml) for CPP services onto AKS. Not a startup/shutdown pipeline itself, but it owns the step that writes the replica-tracking ConfigMap the AKS scale-up logic depends on.

Pipeline catalog

All pipeline IDs are in Azure DevOps project cpp-apps (https://dev.azure.com/hmcts-cpp/cpp-apps/_build?definitionId=<id>).

ID Name Trigger Purpose
369 STE Usage Notify Daily 10:00 UTC Emails STE stack owners usage/age details. Currently failing.
376 STE Environment Costs Notification Manual only Emails owners their monthly Azure cost; posts a summary to Confluence.
379 ste_Auto_suspend Every 30 min, 09:00–04:00 UTC Detects idle STE stacks, alerts cpp_ste_auto_suspend on Slack, then shuts them down.
167 CleanUp STE Envs not in Confluence Daily 22:00 UTC Deletes AKS namespaces for STE stacks no longer listed in Confluence.
416 run-custom-auto-startup-shutdown 3x daily (STE / non-STE / all) The scheduler: reads Confluence, triggers 402/404 per stack.
402 cpp-full-environment-start Triggered by 416, or manual Starts one stack: PAAS → IAAS → AKS, whichever legs are requested.
404 cpp-full-environment-shutter Triggered by 416, or manual Stops one stack: AKS → IAAS → PAAS.
375 Kubernetes Scale Job Manual / triggered by 379 Generic AKS scale up/down job; also scales specific apps to set replica counts.
387 Controlled Environment Start-up Manual Ansible-only start, no AKS/PAAS legs — older alternative to 402.
397 Controlled Environment Shutdown Manual Ansible-only shutdown, no AKS/PAAS legs — older alternative to 404.
417 CPP PRP Controlled Environment Start Manual Live/prp wrapper around the same start template.
440 CPP PRP Controlled Environment Shutter Manual Disabled. Live/prp wrapper around the same shutter template.
333 CPP AKS DEPLOY Manual (app deploy) App deploy pipeline; optional step feeds the AKS scale-up restore logic.

Source files:

  • 369 → cpp-aks-ops/ste_usage_notify_mail.yaml
  • 376 → cpp-aks-ops/environment_costs.yaml
  • 379 → cpp-aks-ops/ste_auto_suspend.yaml
  • 167 → cpp-aks-ops/cleanUp_Ste_Not_in_Confluence.yaml
  • 416 → cpp-aks-ops/run_custom_auto_startup_shutdown.yaml
  • 402 → cpp-azure-devops-templates/pipelines/full-environment-start.yaml
  • 404 → cpp-azure-devops-templates/pipelines/full-environment-shutter.yaml
  • 375 → cpp-aks-ops/k8s_scale_job.yaml
  • 387 → cpp-automation-ansible/pipelines/env_startup_pipeline.yml
  • 397 → cpp-automation-ansible/pipelines/env_shutdown_pipeline.yml
  • 417 → cpp-aks-ops/prp_controlled_environment_start.yaml
  • 440 → cpp-aks-ops/prp_controlled_environment_shutter.yaml
  • 333 → cpp-aks-deploy/aks-deploy.yaml

Not fully investigated: prod_k8s_scale_job.yaml (“Nightly”/“Morning” scale job for prod-mirror stacks) exists in cpp-aks-ops but is marked # Under testing in the file itself, with hardcoded placeholder values — not yet live.

How it fits together

Confluence: "EA Environment management"
(owner, schedule, Full-AKS flag, dates)
        │  read
        ▼
[416] Scheduler (3x daily, per env group)
        │  per stack, per action
        ▼
[402] Start  or  [404] Shutter
        │
        ├─ PAAS → Postgres start/stop
        ├─ IAAS → Ansible playbooks (VMs)
        └─ AKS  → kubectl scale up/down
                  (restores replicas from the
                  replica-tracking ConfigMap)

[379] Auto-suspend (every 30 min)
        ├─ Prometheus idle check
        ├─ Slack alert (#cpp_ste_auto_suspend)
        └─ [375] Kubernetes Scale Job (stop)

[167] Confluence cleanup (daily)
        └─ deletes AKS namespaces not in Confluence

[369] Usage notify (daily)
        └─ emails stack owners (usage/age)

[376] Cost notify (manual)
        └─ emails owners + writes Confluence cost report

[cpp-aks-deploy] CPP AKS DEPLOY (manual)
        └─ optionally writes replica-tracking ConfigMap
           (used later by AKS scale-up)

What each job actually does

run_custom_auto_startup_shutdown (416) — the scheduler

  • Runs on three cron schedules that set which environment groups (STE / DEV / SIT / NFT) are in scope:
    • 2 1-2,4-23 * * * — STE only
    • 5 1-2,4-23 * * * — everything except STE
    • 52 22,23 * * * — all environments
  • Downloads the previous run’s state (STEData.json, confluenceoutputfile.txt) from Blob storage (samdvint01), so it knows which stacks it already started/stopped this run.
  • Runs scripts/generateCustomStartupShutdownLimit.py, which:
    • Fetches the Confluence page (or falls back to the cached backup file).
    • Parses per-stack startup/shutdown times, weekend flag, custom date window, and whether the stack is “Full AKS”.
    • Decides, for each stack, whether it should be started or stopped right now based on current UK time vs. its schedule, and whether today is a weekend it’s allowed to run on.
    • Writes <env>_aks_stacks.json (list of stacks fully on AKS) and updates STEData.json so it doesn’t repeat an action it already performed this run.
  • Uploads the updated state back to Blob storage, publishes the JSON as a pipeline artifact.
  • For each environment × action (Startup/Shutdown), conditionally runs a job that:
    • Reads the stack list for that action out of the JSON map.
    • For each stack, decides runAKS=true always, and runPAAS=true/runIAAS=true only if the stack is not in the “fully on AKS” list (i.e. still has VM components).
    • Triggers cpp-full-environment-start or cpp-full-environment-shutter per stack via az pipelines run, and polls each run to completion, failing the parent pipeline if any child run fails.

cpp-full-environment-start (402) / cpp-full-environment-shutter (404)

Single-stack start/stop orchestrators. Each has three independently-toggleable legs — runPAAS, runIAAS, runAKS — so a stack that’s fully on AKS only runs the AKS leg, while a stack with VM components runs all three.

PAAS leg — starts/stops the stack’s managed Postgres.

IAAS leg — runs Ansible against the stack’s VMs:

Order
Start Elasticsearch → Artemis (staggered, waits for “Server is now live”) → Wildfly (incl. WFM/Camunda, Olpa) → re-enable HAProxy backends → re-check Artemis
Stop Shutter site (HAProxy backends → maintenance, drain sessions) → stop Wildfly/HAProxy/Artemis/Elasticsearch → explicit stop_artemis step → power off VMs (deallocate)

AKS leg — scales the stack’s Kubernetes namespaces (identified by the stack=<name> label) up or down.

Other notes:

  • Production start/stop always needs a human: both pipelines gate on a manual approval stage if platform=live and the environment is/ends with prd.
  • Both pipelines read/write STEData.json in Blob storage to record whether the stack is currently started.

Kubernetes Scale Job (375) / k8s-scale-up.yaml / k8s-scale-down.yaml

Generic, reusable AKS scaling logic, callable standalone or from the pipelines above.

Scale down:

  • Finds namespaces labelled stack=<STACK>.
  • Scales deployments/statefulsets to 0 (or to a specified target/per-app map).
  • Patches HPA minReplicas instead of using kubectl scale where an HPA exists, so autoscaling doesn’t immediately fight the change.
  • Supports excluding specific namespaces and scaling only a named subset of apps.

Scale up:

  • Reads the replica-tracking ConfigMap per namespace to know what replica count to restore each app to.
  • Restores statefulsets before deployments.
  • Waits for rollout status on everything it touched.

Note: ste_auto_suspend (379) calls this pipeline directly (SCALE_TYPE=stop) for idle STE stacks, rather than going through the full 402/404 orchestration — auto-suspend only ever does the AKS leg, not IAAS/PAAS.

ste_Auto_suspend (379) — idle detection

  • Runs every 30 minutes, 09:00–04:00 UTC (skips the 05:00–08:00 window).
  • Lists all STE namespaces from the cluster, groups by stack.
  • Queries Prometheus (istio_requests_total for the ingress gateways) for request-rate over the last 2 hours, per destination namespace.
  • Also checks whether all pods in the stack’s ccm namespace have been running for less than 1 hour — used as a “this stack was just started, don’t suspend it yet” guard rather than a stronger heuristic.
  • If total request rate is ≤0 and the pod-age check passed, the stack is added to a shutdown list.
  • Posts a Slack warning to the cpp_ste_auto_suspend Slack channel listing the stacks about to be shut down.
  • Triggers Kubernetes Scale Job (SCALE_TYPE=stop) for each idle stack.

CleanUp STE Envs not in Confluence (167)

  • Daily at 22:00 UTC.
  • Lists all STE namespaces from the cluster, compares stack names against the Confluence table.
  • Any stack namespace not found in Confluence is deleted outright — destructive, unattended cleanup with no dry-run flag in the script as written.

STE Usage Notify (369) and STE Environment Costs Notification (376)

Both run the same underlying script (steMailNotify.py) with different env flags (STE_NOTIFY vs. TOTAL_COST_NOTIFY), joining the Confluence stack table with live cluster namespace data.

Usage Notify (daily 10:00 UTC, currently failing):

  • For each stack, checks if it’s off the “always tick / weekend light-off” default schedule, or its ccm namespace is >30 days old.
  • If so, emails the owner a usage/age notice.

Costs Notify (manual trigger only — no schedules: block despite the name suggesting it’s automated):

  • Queries Azure Cost Management for the previous calendar month’s cost per namespace.
  • Aggregates per owner across all their stacks.
  • Emails each owner their cost breakdown.
  • Writes/updates a monthly cost report page in Confluence (space EA, under “STE Environments Costs”).

Replica tracking — how AKS scale-up knows what to restore

k8s-scale-up.yaml restores each app to a specific replica count by reading a replica-tracking ConfigMap in the app’s namespace.

That ConfigMap is not written by the shutdown path. It’s written by a separate, conditional step in the app deployment pipeline, cpp-aks-deploy/aks-deploy.yaml (“CPP AKS DEPLOY”) — gated behind a pipeline parameter create_replica_configmap, which only runs when set to True for that deploy.

For each of the stack’s namespaces (main, IDAM, CDNS, web), for every Deployment/StatefulSet, it works out the desired replica count in this priority order:

  1. If an HPA exists, use its live minReplicas (cross-checked against the Helm chart’s autoscaling.minReplicas value, with a warning logged if they disagree).
  2. Otherwise, use the resource’s current live .spec.replicas, if that is > 0.
  3. If live replicas is 0 or missing (e.g. the environment is currently shut down at deploy time), fall back to the Helm chart’s replicaCount value.
  4. If nothing else is available, default to 1.

The result is written as a replica-tracking ConfigMap (one key per app, value = target replica count).

Implication: the replica count an app scales back up to is whatever was captured at the most recent deploy that had create_replica_configmap=True — not necessarily what was running immediately before the last shutdown. If a namespace has never had a deploy run with that flag set, k8s-scale-up.yaml finds no ConfigMap and silently skips restoring replicas for that namespace.

Known issues (current state)

  • 369 (STE Usage Notify) is currently failing — not yet root-caused.
  • 376 (STE Environment Costs Notification) has no active schedule in its pipeline YAML (trigger: none, no schedules: block) despite being framed as a recurring notification — confirm whether it’s actually triggered some other way, or if the schedule was lost.
  • prod_k8s_scale_job.yaml is explicitly marked “Under testing” with hardcoded placeholder values (prp standing in for prd) — not yet a live prod mechanism.
  • Replica-tracking ConfigMap is deploy-time, not shutdown-time (see above) — means replica counts on restart can drift from “what was actually running” if deploys with create_replica_configmap=True are infrequent or skipped for a namespace. There’s an open branch in cpp-aks-deploy (fix-replica-config-script) suggesting this area is already under active work.
  • This document does not yet cover live/prod (prp/prd) startup-shutdown mechanisms in detail — pipelines 417 (start) and 440 (shutter) reuse the same templates as 402/404 with the manual-approval gate, are manual-trigger only (no schedule), and 440 is currently disabled — confirm whether that’s intentional (e.g. prp shutdown handled another way) or something that needs re-enabling.
This page was last reviewed on 12 August 2026. It needs to be reviewed again on 12 November 2026 by the page owner platops-build-notices .
This page was set to be reviewed before 12 November 2026 by the page owner platops-build-notices. This might mean the content is out of date.