Flux Patching Example
This document covers a high level patching example for Flux.
Flux should be patched per environment because it is crucial for the operation of our clusters and deploying applications to them.
Review the renovate pull requests in the flux repos for breaking changes.
Cross check with the Flux releases page.
Upgrading flux includes upgrading the main flux software itself but also the various components that it deploys to perform its various functions.
You can find info about these components here.
These can be upgraded separately so it’s best to use the process detailed below to keep everything in sync.
NOTE: If there are API changes listed on the Flux release page, it may be a good idea to leave these until all environments are running the new Flux version to avoid conflicts. Once all environments are upgraded you can test the api changes on plum/toffee before applying to the rest of the apps.
To upgrade all of these together in a single environment, you can use the Flux CLI.
The generic process can be found on the Flux website including information on how to install the CLI tool on your machine.
Make sure to clone the flux repos and enter them on your terminal by running cd cnp-flux-config or cd sds-flux-config.
The specific command we need to target an environment, it is recommended to add the --version flag at the end as flux will default to your local configuration. e.g, sandbox, is:
flux install --export > apps/flux-system/sbox/base/gotk-components.yaml --version v2.4.0
This command will generate a file containing the latest versions of all the flux components.
You will also need to update the kustomization to point to this updated version instead of the existing version.
You can find a full example for CFT in this pull request.
The same process will work for SDS.
Testing everything is working and has been updated
If you describe some of the crds flux deploys you will see the updated versions:
kubectl describe crd helmreleases.helm.toolkit.fluxcd.io -n flux-system | grep app.kubernetes.io/version
You can get the list of crds deployed by flux by running:
kubectl get crds | grep fluxcd
You can also check the image version on the deployments or pods. The example below shows the source-controller deployment:
kubectl describe deploy source-controller -n flux-system | grep ghcr.io
The pods should have been redeployed as well, so their uptime should be very recent.
Non-Prod environments
To replicate the above procedure in a nonprod environment, update the flux install command to point to the environment you’re targeting and update the relevant kustomization file.
For example:
flux install --export > apps/flux-system/ithc/base/gotk-components.yaml
Prod environments
For PTL, there are two extra components that need to be deployed:
- Image reflector controller
- Image automation controller
These components are critical for ensuring updates to images are pushed to Git and synced to the clusters automatically.
To add these components on PTL, use this command:
flux install --components-extra image-reflector-controller,image-automation-controller --export > apps/flux-system/ptl/base/gotk-components.yaml
You can review this pull request and see the extra components are present.
To ensure that the image automation components are working, as well as the other checks you can perform, you can check the flux repo to see if updates are being pushed by fluxcdbot.
See how fluxcdbot was still working after the upgrade.
Now all environments, except prod, should be updated and you can move onto updating prod itself.
To do this, you can merge the renovate pull request. That pull request will update the base version of Flux that applies to all environments by default.
You can then remove all the previous changes you made per environment, except in PTL because it needs the extra components, and the base version will take over.
Upgrading Flux when there are API changes
Below is a step by step on how to upgrade flux when there are API changes included in the new version. The example is taken from SDS.
- Upgrade flux version in lower environments
- Test that has worked using commands mentioned above.
- Upgrade flux version in prod/ptl
- Test again using commands above.
- Upgrade API versions e.g. HelmRelease and ImageRepository
- Test all is still working after the upgrade. NOTE: You can also apply the API changes only to Plum/Toffee initially to test, before applying to other apps.