Artifactory Patching
Artifactory is utilized in the context of managing and updating software dependencies and configurations within a Kubernetes environment. These dependencies are cached and are utilized by teams if upstream issues arise. By bringing the cached dependencies closer to build agents, fewer teams are impacted. Artifactory can be disabled by following the example pr as a temporary bypass to unblock teams.
There are 3 additional files within artifactory: admin-pw.yaml, join-key.yaml and master-key.yaml. These files are necessary for Artifactory to operate, admin-pw.yaml is used to login to artifactory, join-key.yaml and master-key.yaml are used internally and are needed for the startup of Artifactory.
This documentation will guide you through checking and updating the self-hosted version of Artifactory being used. This process includes accessing Artifactory’s website to determine the latest version available, verifying that the Artifactory Helm Release has accepted the latest changes, and ensuring the pods are healthy and that Artifcatory is accessible.
Note: This activity is disruptive and will cause artifactory to stop working as the pods are redeployed in a statefulset. To minimize disruptions to teams the following paths can be taken:
- A: Complete the patch early in the morning, similar to a jenkins upgrade, to avoid disruptions.
- B: Temporarily disable Artifactory, following the example pr, before upgrading.
If any issues arise during the patching process, troubleshooting steps are provided at the bottom of this page. These steps involve identifying and resolving any errors or conflicts that may occur, such as missing namespaces or configuration discrepancies.
Artifactory Patching Process
1. Updating CFT PTLSBOX
• Create a new file named artifactory-sbox.yaml in the artifactory directory, add the following and update to the latest version.
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: artifactory-oss
namespace: artifactory
spec:
chart:
spec:
chart: artifactory-oss
version: 107.90.9 <- update to latest version
Note: The layout of Artifactory versioning includes 10 at the beginning. This version would be 7.90.9
• Make the changes to the bottom of kustomization.yaml file found in sbox-intsvc directory.
...
patchesStrategicMerge:
- ../../artifactory/artifactory-sbox.yaml
• Submit your PR and ensure it gets reviewed and merged.
2. Check new version on CFT PTLSBOX
• Connect to CFT PTLSBOX cluster
• Run the command below and ensure the Helm Release is showing True and is running the latest chart version.
kubectl get hr -n artifactory
• Run the following commands ensuring the pod is healthy and has applied new version. Note: version number found at “artifactory-oss”.
kubectl get pods -n artifactory
kubectl describe pods -n artifactory artifactory-oss-0
• For further confirmation, ensure that the grep command below returns the chart version that you have applied.
kubectl describe pods -n artifactory artifactory-oss-0 | grep chart=artifactory-107.90.9
• Ensure you can access Artifacory website. Note: you will need to be connected to F5 VPN to access this link.
3. Update CFT PTL
• Remove artifactory-sbox.yaml file and its corresponding code from kustomization.yaml made in step 1.
• Update the version in artifactory.yaml to latest version eg, (7.90.9) <- ensure 10 is in front of the version.
...
chart:
spec:
chart: artifactory-oss
version: 107.90.9
...
• Submit your PR and ensure it gets reviewed and merged.
• Repeat step 2 for ‘CFT PTL cluster’ checking the Helm release is on the new version and ensuring the pods are healthy.
Related Links
Artifactory Note: you will need to be connected to F5 VPN to access this link.
Artifactory Release Information
Troubleshooting
How to view logs of artifactory
To view the logs while artifactory is booting up / running and to spot any errors, enter the following commands below. Note: The default logs are from the router container.
kubectl logs -n artifactory artifactory-oss-0
- Tip: To keep watching logs use
-fat the end.
• To view all containers in a pod and to get container specific logs enter the following commands.
kubectl get pods -n artifactory artifactory-oss-0 -o jsonpath='{.spec.containers[*].name}'
kubectl logs -n artifactory artifactory-oss-0 -c <container-name-here>
- Tip: The main containers to view are
routerandartifactory, here you will see most errors likely to occur.
Missing required services
[jfrou] [WARN] [217db729f7674ef0] [local_topology_helper.go:68] [main] [] - Missing required services: [jfmd]
• You will see this error upon startup of artifactory as all services startup, if they do not clear after 60 - 90 seconds investigate further. This often occurs when one of the containers is not healthy, usually the router service. These errors most likely point to Master / Join key issues as the router cannot establish connection with the internal DB.
Database errors
[jfrt ] [ERROR] [ee4c12af0700e30b] [o.a.s.d.i.DbConnectionUtils:86] [Catalina-utility-2 ] - Cannot start the application with a database other than PostgreSQL. For more information, see JFrog documentation
• This error is unlikely to appear, if it does it means that the artifactory configuration has not successfully applied and is using other DB than the one specified. Restart artifactory and watch the startup logs of the container and ensure this error does not appear.
To watch the containers enter the command below.
kubectl logs -n artifactory -c artifactory artifactory-oss-0
Join / Master key errors
Join / Master key errors can be a false error and point to a separate issue. Investigate the artifactory container first before attempting to look at other areas of artifactory as this can be a bootstrapping error upon startup. Artifactory masks most issues with join-key or master-key so it is a good case to check artifactory container first. More info can be found in this GitHub comment where a more in depth explanation can be found, but is not relevant to this documentation.
[jfrou] Cluster join: Access Service ping failed, will retry. Error: cluster join: error from service registry on ping:
• Join key issues are found before master key issues arise, these are shown upon startup as the internal database boots and usually disappear after 30 seconds. If join-key missing is still showing investigate containers themselves if the key has been picked up by artifactory correctly.
[jfmd ] [ERROR] - Failed resolving master key: failed resolving 'shared.security.masterKey' key; file does not exist: /opt/jfrog/artifactory/var/etc/security/master.key
• Master key is also used for the database, for internal communication purposes. If master-key issues are still showing ensure artifactory has these in the containers.
Check keys in container
kubectl exec -it artifactory-oss-0 -n artifactory -c router -- /bin/sh
• Once inside the container cd to this directory, list the files and ensure join.key and master.key are there.
cd var/etc/security/
ls -la
How to fix
After investigating artifactory containers further and if the key issues are is persisting, restart the artifactory service by deleting the pvc, statefulset and the pod of artifactory namespace. If the service is still not restarting successfully, then a full rebuild of the secret keys is most likely needed.
Recreating new keys
This will cover creating a new key, sops encrypt new key, rename file and update the cnp flux repo in a new pr with the new key. Note: The following example uses master-key replace this with join-key when creating that key.
• Before we start we need a random string to be used by our key. Copy its output as we need it later.
openssl rand -hex 32
kubectl create secret generic master-key -n artifactory --from-literal=master-key='<rand -hex 32 output>' --type=Opaque -o yaml --dry-run=client > file.enc.yaml
• Note: Ensure you have the correct key vault sops link, search for your cluster in azure portal and copy the sops key uri. The Sops key uri is found in objects -> keys -> sops-key and is under Key Identifier.
sops --encrypt --azure-kv <sops-key-uri> --encrypted-regex "^(data|stringData)$" --in-place file.enc.yaml
• Rename the file so that artifactory can use the new key.
mv file.enc.yaml master-key.yaml
• Follow the steps again for join-key.yaml.
• Submit your PR with the new keys and ensure it gets reviewed and merged.
• Restart artifactory by deleting the pvc, statefulset and the pod of artifactory namespace. Monitor artifactory as it boots up and ensure the errors are gone.
• If the new set of keys and restart of artifactory has not started up correctly reach out to a member of your team for assistance.