Dynatrace private locations
What are private locations?
You can run your Dynatrace synthetic monitors from a private Synthetic location, which is a location in your private network infrastructure where you install one or more Synthetic-enabled ActiveGate instances.
How does this affect monitoring?
We have HTTP and browser monitors set up in dynatrace to check applications are reachable and displaying the expected pages.
If you add a new private dynatrace activegate, it will need added to the HMCTS Private Location so it can receive monitoring data.
You can find the locations setup in Dynatrace by going to the portal, clicking Synthetic from the left hand pane.
This will show you the synthetic monitors setup on our tenant.
On the left hand side, under Monitored locations, you will HMCTS Private Location.
This private location refers to the private activegates we have installed in one of our Azure subscriptions as a Virtual Machine Scale Set.
How can I add a new activegate to the private location?
Unfortunately, you cannot do this yourself as you probably won’t have permission.
Reach out to one of our contacts at Dynatrace to get this completed.
Ask the Platform Operations team for info on who specifically to contact as this changes over time.
How can I manually trigger a monitor to check an application is working during an outage?
Different monitors will be setup to run at different frequencies but if you need to manually check a monitor to see if it has come back during an outage you can click On-demand execution.
This was needed when we removed our old private activegates and the new ones had mistakenly not been added to the private location.
Once the new activegates had been added, rather than waiting another 15 mins for the particular monitor to run again, we clicked On-demand execution and were able to quickly confirm the synthetic monitor was working again.
Fix DNS resolution issues
If the private VMs cannot resolve the hostnames of the apps being monitored, you may need to add the vnet as a link to the DNS zone.
To do this, raise a pull request in the azure-private-dns repo to add the dynatrace vnet as a link.
Here is an example pr.
Fix timeout issues
If the DNS is resolving but you’re getting a timeout error when running an on-demand execution, the issue could be firewall related.
Connect to the appropriate Palo Alto and check the Palo Alto monitor.
If the traffic is showing as denied, then you will need to ensure the traffic is allowed by ensuring the two networks are in the correct zone and there is a rule to allow the traffic.
This is managed in the hub-panorama-terraform repo.
If the traffic is allowed but it’s still not working, then it may be that there is no route back from the application to dynatrace.
In this case, you need to add a route to the appropriate route table for the application gateway.
Here is an example PR showing how to do this.
Dynatrace Private Activegates have stopped working
You may receive a PlatOps Help request from one of the Dynatrace team saying the private activegates have stopped working.
This is usually due to an outdated version of Chromium. Whoever raises the request will usually state this is the case.
If this is what’s happened, you can upgrade Chromium and install the latest activegate version manually.
Follow this guide to get connected to the activegates.
Then do the following:
Remove chromium
sudo snap remove chromiumUninstall the activegate
sudo /opt/dynatrace/gateway/uninstall.shReinstall chromium
sudo snap install chromiumDownload the latest activegate installer script
wget -O Dynatrace-ActiveGate-Linux-x86-azure.cft-<LATEST_VERSION>.sh "https://ebe20728.live.dynatrace.com/api/v1/deployment/installer/gateway/unix/latest?arch=x86&networkZone=azure.cft" --header="Authorization: Api-Token <API_TOKEN>"Reinstall the activegate
Dynatrace-ActiveGate-Linux-x86-azure.cft-<LATEST_VERSION>.sh --enable-synthetic --set-property=synthetic=private_location_id=SYNTHETIC_LOCATION-C0E9EDB41E72A6F7 --set-network-zone=azure.cft
You can get the API token and the latest version number from the Dynatrace team, i.e. whoever raised the ticket.
If you get an error about lack of space, you can try clearing down the journal file:
du -h /var/log -d 1
sudo journalctl --vacuum-time=10d
This will remove any logs older than 10 days.