Profile a Java application running in an AKS cluster
This page describes how to profile a java application which is deployed in one of our AKS clusters.
Prerequisites
- Your user will need to have permissions to run kubectl exec and to ssh to a node. (Subscription contributor role works)
- Install kubectl node-shell which in turn requires krew
- Something to open .jfr files with like Java Mission Control
Setup
You’ll need the following bits of information during this exercise:
1. namespace (hopefully you just know this or browse kubectl get namespaces)
2. pod name (find it in kubectl get pods -n $namespace)
3. container name (kubectl get pods -n $namespace $podName -o=jsonpath='{.spec.containers[0].name}')
4. node name (kubectl get pods -n $namespace $podName -o=jsonpath='{.spec.nodeName}')
Example
namespace=rpe
podName=rpe-send-letter-service-java-7fc6fffdf9-qvfd6
containerName=$(kubectl get pods -n $namespace $podName -o=jsonpath='{.spec.containers[0].name}')
nodeName=$(kubectl get pods -n $namespace $podName -o=jsonpath='{.spec.nodeName}')
Running Java Flight Recorder (JFR)
After completing the setup above you can run
kubectl exec $podName -n $namespace -- jattach 1 jcmd "JFR.start name=profile1"
The output will look something like this
Connected to remote JVM
Response code = 0
Started recording 1. No limit specified, using maxsize=250MB as default.
Use jcmd 1 JFR.dump name=1 filename=FILEPATH to copy recording data to file.
At this point you can run any tests you want to capture any data in the profile report.
Once complete, you can dump the data to a file on the container:
kubectl exec $podName -n $namespace -- jattach 1 jcmd JFR.dump
If you don’t specify a name in the command, a time-stamped filename will be generated.
The output will look something like this
Connected to remote JVM
Response code = 0
Dumped recording, 4.5 MB written to:
/opt/app/hotspot-pid-1-2022_01_20_14_32_49.jfr
Make a note of the filename for retrieving later
Run kubectl exec $podName -n $namespace -- jattach 1 jcmd "JFR.stop name=profile1"
to stop the recording. You can use kubectl exec $podName -n $namespace -- jattach 1 jcmd JFR.check
to list all in-flight recordings
Downloading the file
As our images don’t have a shell installed or tar (can’t use kubectl cp) it’s a little bit more involved to get the
JFR report.
To figure out where the file is, you need to use nsenter to get a shell onto the node and find the mount point for the specific container you need.
kubectl node-shell $nodeName -n admin
spawning "nsenter-x8uq64" on "aks-linux-81166528-vmss00005t"
If you don't see a command prompt, try pressing enter.
root@aks-linux-81166528-vmss00005T:/#
Make a note of the name of the spawned instance (in this case nsenter-x8uq64).
At this point you’ll need your variables from earlier
namespace=rpe
containerName=rpe-send-letter-service-java
pod=rpe-send-letter-service-java-6fb55dd776-87b6f
You’ll also need some additional information out of containerd using crictl.
podId=$(crictl pods --namespace $namespace --name $pod --state Ready --quiet)
containerId=$(crictl ps -p $podId --no-trunc --label io.kubernetes.container.name=$containerName -q)
targetPid=$(crictl inspect $containerId | jq .info.pid)
Now we can use nsenter to create us a shell at the mount point for the container
nsenter --target $targetPid --uts --net --pid unshare --mount-proc sh -c "cd /run/containerd/io.containerd.runtime.v2.task/k8s.io/$containerId/rootfs/opt/app && exec bash"
groups: cannot find name for group ID 11
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
When we list the files you can see the hotspot JFR file which was dumped by jattach earlier.
root@rpe-send-letter-service-java-6fb55dd776-87b6f:/run/containerd/io.containerd.runtime.v2.task/k8s.io/9704b2c0b9b559001eb23156a06e40d25e10ceeffc4849bb51b1688025da6f61/rootfs/opt/app# ls
AI-Agent.xml hotspot-pid-1-2022_01_20_14_32_49.jfr
applicationinsights-agent-2.5.1.jar send-letter-service.jar
To download this file we need to go back to a terminal on our local machine and run kubectl cp to copy from the spawned instance
(nsenter-x8uq64) using the path we just discovered above
(/run/containerd/io.containerd.runtime.v2.task/k8s.io/9704b2c0b9b559001eb23156a06e40d25e10ceeffc4849bb51b1688025da6f61/rootfs/opt/app)
appended by the file name hotspot-pid-1-2022_01_20_14_32_49.jfr like so:
kubectl cp admin/nsenter-x8uq64:/run/containerd/io.containerd.runtime.v2.task/k8s.io/9704b2c0b9b559001eb23156a06e40d25e10ceeffc4849bb51b1688025da6f61/rootfs/opt/app/hotspot-pid-1-2022_01_20_14_32_49.jfr ./hotspot-pid-1-2022_01_20_14_32_49.jfr
tar: Removing leading `/' from member names
You can now open this file locally using Java Mission Control and terminate your connection to the nsenter pod.