Jenkins
Instances
There are two sets of Jenkins instances, SDS and CFT, both have a sandbox instance and a production one.
SDS:
CFT:
Monitoring
Jenkins patching
⚠️ Note: Before starting, send notifications out ahead of time on #cloud-native-announce and #sds-cloud-native to let teams know patching will be occurring.
You should endeavour to patch Jenkins early in the morning before 8am on a weekday so we can identify issues and roll back, if required.
A change request is not necessary.
Building
The container image is built from cnp-jenkins-docker, it contains Jenkins controller version within its Dockerfile and plugin versions within plugins.txt - either could have been updated within a given image version.
Each merge to the main branch will run a GitHub workflow to build and publish the image to the ACR, consult Promoting a new Jenkins image for more details.
How Renovate is configured
Plugin and Jenkins Controller versions within the image are updated by Renovate through automatically raised and merged pull-requests, current Renovate configuration ensures plugins are updated in separate PR to the Jenkins Controller updates (so in practice this means they will be included in separate image versions).
Once cnp-jenkins-docker image version is promoted, Renovate will pick it up within its scheduled time and raise a PR to update a relevant Jenkins overlay file within sds-flux-config and cnp-flux-config repositories.
For any overlay files that apply to Sandbox Jenkins instances only, e.g. those in sbox folders: apps/jenkins/jenkins/sbox-intsvc in CNP or apps/jenkins/jenkins/ptlsbox in SDS Renovate will raise and automatically merge separate PRs so the Sandbox instances should always be on the latest version for:
- Jenkins Controller and plugins (latest
cnp-jenkins-dockerimage) - Jenkins Helm Chart (latest minor/patch versions only)
- K8S-Sidecar Helm Chart (latest minor/patch versions only)
Updating the version
Sandbox instances should already be on the latest available version before you start patching but double check this by reviewing the flux config files and going to "About" section of Jenkins site for both SDS and CND.
If they are not on the latest check if there are any unmerged update PRs opened by Renovate within the flux config repositories for Sandbox instances and merge them.
If latest version is already running on both instances check if there are any ongoing problems.
Run the plum and toffee pipelines (at least one of frontend or backend AND the infrastructure pipeline) to ensure there are no issues being caused by upgraded plugins or the Jenkins version itself.
If Renovate has not updated the Sandbox Jenkins instances to the version you want to patch to (usually latest) you should create a PR to do this manually and then test as described above.
When preparing for patching you can also use Jenkins Version Analyser agent skill to analyse the changes between current and target controller image version - it will create a report containing:
- All the intermediate versions of
cnp-jenkins-dockerimage betweencurrentandtarget - All the changes to Jenkins application or plugin versions contained within a specific image version
- Release notes for all changes (plugin and/or Jenkins updates)
- AI generated risk assessment for each image version based on release notes
This might help to catch any pitfalls, for example Jenkins controller updates that are incompatible with certain plugins, it is also handy to have all possible release notes and PR links displayed in a single place.
Checking for vulnerabilities
Check the settings page in CNP and SDS and confirm there are no remaining vulnerabilities in plugins or any other recommendations.
If there are recommendations or deprecations, this probably warrants a new PR to move to a newer version.
Sometimes there might not be a newer version to move to if new image within cnp-jenkins-docker has not yet been built with the required updates - in this case you could kick Renovate off from the Dashboard within repository issues tab to scan and create PRs to update plugins/jenkins version if newer ones are available. You can also update them manually.
If there are security vulnerabilities, follow the advice from maintainers. If a fix is not yet possible, set a reminder to revisit this in a few days.
If it’s not a security vulnerability, raise a tech improvement ticket for it to be dealt with by a DevEx Domain squad.
Proceed to patch the PTL environment and perform the same steps.
If there are updates to Helm charts, you should ensure that these have been tested out on Sandbox first to confirm if any changes to code are needed.
Once you’ve confirmed Sandbox Jenkins instances are working using the same steps from above, you can proceed to upgrading the helm charts in PTL.
See example of potential breaking changes that can occur when upgrading the helm charts to new major versions.
Configuration
All instances are 100% managed as code, you should never have to edit any configuration manually.
It’s configured with the Configuration as Code plugin using the integration configured in the official helm chart.
Configuration is in the Jenkins base for the corresponding flux repository.
Guides
- How Jenkins Agents Work
- Troubleshooting builds starting slowly
- Run command on all agents
- Downgrade a plugin
- Add packages to Jenkins Agents
- Fortify on Demand (FoD) scanning
- How SonarQube works on Jenkins
- Jenkins GitHub Apps