Mailrelay Essentials
This runbook describes how Mailrelay is configured and deployed and the steps that are necessary to on-board new clients. The Mailrelay configuration can be found in the repo exim-relay.
How to On-board Clients
Each service will require a username and password to utilise the Mailrelay service and these will be stored in Key Vault (Dev: sds-mailrelay-dev, Prod: sds-mailrelay-prod). The Username will be the service name; the password should be randomly generated and be sufficiently complex.
1. Create a new secret in Key Vault and associate a complex password
2. Branch off the Flux repo, edit the configuration file mailrelay2.yaml then append the new service account to the section authKeyVaultSecrets.
The syntax is as follows:
- "my-service-user-account"
- "mailrelay-prod-user"
3. Create a new pull request and merge into the master branch once approved
4. Test access and connectivity as described in the section below Testing Mailrelay
What is Mailrelay and What is it used for?
Exim Mail Relay is a Mail Transfer Agent, its main purpose is to receive emails from a Mail User Agent (MUA) and relay the email to other MTAs or a Mail Delivery Agent .
Mailrelay AKS deployment
An Exim Mailrelay container is built via Docker and stored in the ACR Azure image repository. It’s based on the featherweight image Alpine as defined in the git repo Docker Exim Relay Image and is built via an Azure DevOps pipeline.
Exim Mailrelay is currently deployed on AKS in SS-dev-00 / SS-dev-01 / SS-prod-00 / SS-prod-01 as two replicas. It is monitored using Prometheus and Grafana. Alerts are sent to the following Slack channel #prometheus-alerting-prod , #prometheus-alerts, #prometheus-critical.
Modifying the exim.conf File
The exim.conf file specifies the configuration and behaviour of the mailrelay service in terms of security (hosts that can use the system to route mail, authentication), DNS settings (domain names, routing), protocols and many other settings. It’s created with makefile which collects together a group of related actions into a single build file that when executed compiles or performs some processing to create artifacts. Read more about makefile here: makefile tutorial.
Authentication Mechanisms
Client applications use TLS to authenticate to the Mailrelay server to be able to send emails. The server has inbound and outbound certs. Certificates are generated using the ACME Function App
The following certificates are installed in the Docker image which is the source of the mailrelay container:
dev-mailrelay-platform-hmcts-net
prod-mailrelay-platform-hmcts-net
dev-in.mailrelay.internal.platform.hmcts.net (self signed)
Deployment
1. Clone Exim-Relay
2. After making changes to the Mailrelay configuration have someone review the pull request.
- While making the Pull Request please ensure that the base repo is hmcts/ exim-relay and not luigibk/exim-relay
3. Pipeline
The pipeline will run after making a PR or merging to master and will build an image in the Azure Container Repository
- The Service connection used for the pipeline is
DTS SS Public Prod
- The Service connection used for the pipeline is
4. Configuring Shared Services Flux
In order to deploy to your latest image to the Dev or Production environment flux needs to know the image it needs to look for. In Azure DevOps
- Clone the sds-flux-config repo
- change the tag to the latest tag number in the patches file
- Add and Commit changes
- Review as required
- Merge branch with Master.
- After 5 to 10 minutes check the deployments have been updated on the cluster.
Testing Mailrelay
After making changes to Exim.conf you may need to test that emails are going through according to the authentication mechanism that you have set to be advertised by the host.
Test unauthenticated relay
1. Get the IP address and port number from the mailrelay/mailrelay2 namespace
kubectl get svc -n <namespace>
2. Spin up a temporary pod on the same namespace to send test email
kubectl run -it --rm --restart=Never -n admin busybox --image=radial/busyboxplus:curl
3. Send email from pcol to chosen email. Please note that each command below need to enter separately one at time, please see example below.
telnet <ip:port>
helo possessionclaim.gov.uk
mail from: noreply-pcol@hmcts.net
rcpt to: <recipient email>
data
Subject: test
test test test
.
- Note - For dev clusters, unauthenticated relay should be turned off and you should receive a 550 unauthenticated relay response
example
Test TLS connection using SWAKS
kubectl run my-shell -it --rm --restart=Never -n admin --image=ubuntu --command -- bash
apt update
apt install swaks
apt install telnet
swaks -a -tls -q HELO -s <ip>-au v1test -ap '<password'
Test StartTLS connection using OpenSSL
Turn user name and password to base 64
echo -ne '<password>' | base64
Connect to Mailrelay using StartTLS
openssl s_client -connect ip:port -starttls smtp
Send Email
```bash
helo possessionclaim.gov.uk
mail from: noreply-pcol@hmcts.net
rcpt to: <email>
data
Subject: test
test test test
.
Monitoring MailRelay
Pre requisites
- GitHub write access to Exim-Exporter - Platform Operations team has access
- Azure Devops access to Exim Exporter Pipeline - Platform Operations team has access
Prometheus
- The Prometheus server (Monitoring Namespace) is used to obtain metrics from the exim server and exports them using the Exim-Exporter tool
- The exim exporter is used to read and send exim relay metrics to Prometheus server
- The Alert Manager (Monitoring Namespace) takes these metrics and sends it to specified Slack Channels.
- The Prometheus alerts for mailrelay can be updated if current alerts needs to be updated
Troubleshooting Common Errors
Alerts are not being sent to Slack
- Check alert logs
kubectl exec --stdin --tty alertmanager-kube-prometheus-stack-alertmanager-0 -n monitoring -- /bin/sh
kubectl logs -f -c alertmanager alertmanager-kube-prometheus-stack-alertmanager-0 -n monitoring
kubectl logs -f -c alertmanager alertmanager-kube-prometheus-stack-alertmanager-0 -n monitoring > alertmanager-logs.log
- Make post request
wget --header "content-type: application/json" --post-data '[{"status": "firing","labels": {"alertname": "EximQueueLength","service": "mailrelay","severity": "warning","instance": "0"}}]' http://localhost:9093/api/v1/alert
Performance Testing
more to be added