Let's Encrypt Total Uptime

I’m waiting for our purchasing department to pay for our wildcard certs. I only have a few weeks left until they expire. Once the purchase goes through I’ll no doubt have to go through another validation process that involves waiting for the vendor to find our company on the internet and call me. Which of course is a problem because I don’t know when they are going to call and if they will get to me, especially when I’m working from home.

Anyway the whole situation makes me nervous so I was thinking about a fall back. How about EFF’s Let’s Encrypt! Let’s Encrypt is a revolutionary service. But there are a couple issues with the way I want things.

Let’s Encrypt has two validations options:

HTTP from a server reachable from the internet - many of the services I want to protect aren’t reachable from the internet.
And DNS - but Certbot doesn’t support my DNS provider, Total Uptime.

Certbot’s not the only game in town. Caddy can facilitate certs also but alas, no one has written a dns provider for Total Uptime, yet.

There is also gethttpsforfree.com, a nifty service that interactively facilitates the validation for generating certs using your browser. This works for wildcards but I don’t want to go through this process every 3 months.

Thankfully certbot provides validation hooks so you can script your own validation process. Perfect!

Their example cloudflare scripts were excellent starting points. They describe the surprisingly straight forward process well. It involves two scripts.

Example authenticator script

#!/bin/bash

# Get your API key from https://www.cloudflare.com/a/account/my-account
API_KEY="your-api-key"
EMAIL="your.email@example.com"

# Strip only the top domain to get the zone id
DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')

# Get the Cloudflare zone id
ZONE_EXTRA_PARAMS="status=active&page=1&per_page=20&order=status&direction=desc&match=all"
ZONE_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones?name=$DOMAIN&$ZONE_EXTRA_PARAMS" \
     -H     "X-Auth-Email: $EMAIL" \
     -H     "X-Auth-Key: $API_KEY" \
     -H     "Content-Type: application/json" | python -c "import sys,json;print(json.load(sys.stdin)['result'][0]['id'])")

# Create TXT record
CREATE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"
RECORD_ID=$(curl -s -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records" \
     -H     "X-Auth-Email: $EMAIL" \
     -H     "X-Auth-Key: $API_KEY" \
     -H     "Content-Type: application/json" \
     --data '{"type":"TXT","name":"'"$CREATE_DOMAIN"'","content":"'"$CERTBOT_VALIDATION"'","ttl":120}' \
             | python -c "import sys,json;print(json.load(sys.stdin)['result']['id'])")
# Save info for cleanup
if [ ! -d /tmp/CERTBOT_$CERTBOT_DOMAIN ];then
        mkdir -m 0700 /tmp/CERTBOT_$CERTBOT_DOMAIN
fi
echo $ZONE_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
echo $RECORD_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID

# Sleep to make sure the change has time to propagate over to DNS
sleep 25

Create a TXT record using the provided CERTBOT_DOMAIN and CERTBOT_VALIDATION environment variables.
Wait for propagation
After the script returns, certbot does it’s thing

When certbot is done it runs the cleanup script.

Example cleanup script

#!/bin/bash

# Get your API key from https://www.cloudflare.com/a/account/my-account
API_KEY="your-api-key"
EMAIL="your.email@example.com"

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID ]; then
        ZONE_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID)
        rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
fi

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID ]; then
        RECORD_ID=$(cat /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID)
        rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID
fi

# Remove the challenge TXT record from the zone
if [ -n "${ZONE_ID}" ]; then
    if [ -n "${RECORD_ID}" ]; then
        curl -s -X DELETE "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/$RECORD_ID" \
                -H "X-Auth-Email: $EMAIL" \
                -H "X-Auth-Key: $API_KEY" \
                -H "Content-Type: application/json"
    fi
fi

Delete the TXT records

Nothing is passed between the two scripts. API object IDs are saved to /tmp to avoid having to lookup the records during cleanup.

Now to make my own version of this for Total Uptime I was going to need a similar API. And to my delight I learned Total Uptime provides an API.

Here are the scripts that I came up with.

Total Uptime authenticator script

#!/bin/bash

TU_USERNAME=""
TU_PASSWORD=""
API_URL="https://api.totaluptime.com"

# Check vars
test -z "$TU_USERNAME" && \
  echo "ERROR: TU_USERNAME must be set" && exit 1
test -z "$TU_PASSWORD" && \
  echo "ERROR: TU_PASSWORD must be set" && exit 1

# Check deps
if ! which jq >/dev/null; then
  echo "ERROR: jq is required, https://stedolan.github.io/jq/" && exit 1
fi
if ! which curl >/dev/null; then
  echo "ERROR: curl is required, https://curl.se/" && exit 1
fi

# Strip only the top domain to get the zone id
DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')

# This must be a wildcard or top level domain
test -z "$DOMAIN" && \
  DOMAIN="$CERTBOT_DOMAIN"

# Get the Total Uptime domain id
QUERY_PARAMS="?searchField=domainName&searchString=$DOMAIN&searchOper=eq"
ZONE_ID=$(curl -s -u "$TU_USERNAME:$TU_PASSWORD" -X GET \
  -H "Accept: application/json" \
  "$API_URL/CloudDNS/Domain/All$QUERY_PARAMS" | jq -r '.rows[0].id')
test "$ZONE_ID" == "null" && \
  echo "ERROR: Failed to retrieve domain id from Total Uptime API" && exit 1

# Create TXT record
DOMAIN_HOST=$(echo "$CERTBOT_DOMAIN" | sed "s/$DOMAIN$//" | sed 's/\.$//')
CREATE_DOMAIN="_acme-challenge.$DOMAIN_HOST"
# Check if we are validating a wildcard domain
test -z "$DOMAIN_HOST" && \
  CREATE_DOMAIN="_acme-challenge"

# Check if TXT record already exists
QUERY_PARAMS="?searchField=txtHostName&searchString=$CREATE_DOMAIN&searchOper=eq"
RECORD_ID=$(curl -s -u "$TU_USERNAME:$TU_PASSWORD" -X GET \
  -H "Accept: application/json" \
  "$API_URL/CloudDNS/Domain/$ZONE_ID/TXTRecord/All$QUERY_PARAMS" | jq -r '.rows[0].id')
METHOD="PUT"
test "$RECORD_ID" != "null" && \
  RECORD_ID="/$RECORD_ID"
test "$RECORD_ID" == "null" && \
  RECORD_ID="" METHOD="POST"

# Create or update the TXT record
DATA="{\"txtHostName\":\"$CREATE_DOMAIN\",\"txtText\":\"$CERTBOT_VALIDATION\",\"txtTTL\":\"60\"}"
REQUEST_URI="$API_URL/CloudDNS/Domain/$ZONE_ID/TXTRecord$RECORD_ID"
RESPONSE=$(curl -s -u "$TU_USERNAME:$TU_PASSWORD" -X "$METHOD" \
  -H "Accept: application/json" "$REQUEST_URI" -d "$DATA")
RESPONSE_STATUS=$(echo "$RESPONSE" | jq -r '.status')
test "$RESPONSE_STATUS" != "Success" &&
  echo "ERROR: Failed to set TXT record using Total Uptime API, $REQUEST_URI, $DATA, $RESPONSE" && exit 1

# Set TXT record id
RECORD_ID=$(echo "$RESPONSE" | jq -r '.id')
test "$RECORD_ID" == "null" && \
  echo "ERROR: Failed to get record id after setting TXT record $CREATE_DOMAIN" && exit 1

# Save info for cleanup
test ! -d /tmp/CERTBOT_$CERTBOT_DOMAIN && \
  mkdir -m 0700 /tmp/CERTBOT_$CERTBOT_DOMAIN

echo $ZONE_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
echo $RECORD_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID

# Sleep to make sure the change has time to propagate over to DNS
sleep 300

Total Uptime cleanup script

#!/bin/bash

TU_USERNAME=""
TU_PASSWORD=""
API_URL="https://api.totaluptime.com"

# Check vars
test -z "$TU_USERNAME" && \
  echo "ERROR: TU_USERNAME must be set" && exit 1
test -z "$TU_PASSWORD" && \
  echo "ERROR: TU_PASSWORD must be set" && exit 1

# Check deps
if ! which jq >/dev/null; then
  echo "ERROR: jq is required, https://stedolan.github.io/jq/" && exit 1
fi
if ! which curl >/dev/null; then
  echo "ERROR: curl is required, https://curl.se/" && exit 1
fi

# Get zone and record ids
ZONE_ID=$(cat "/tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID")
RECORD_ID=$(cat "/tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID")
test -z "$ZONE_ID" && \
  echo "ERROR: Failed to get ZONE_ID" 1>&2 && exit 1
test -z "$RECORD_ID" && \
  echo "ERROR: Failed to get RECORD_ID" 1>&2 && exit 1

# Delete the TXT record
RESPONSE=$(curl -s -u "$TU_USERNAME:$TU_PASSWORD" -X "DELETE" \
  -H "Accept: application/json" \
  "$API_URL/CloudDNS/Domain/$ZONE_ID/TXTRecord/$RECORD_ID")

Some lessons learned in writing these.

Cerbot will schedule a job to renew certs automatically using the scripts provided.
Custom environment variables aren’t available in the auto renew scheduled task.
Relative paths to the scripts will not resolve in certbot’s scheduled task config file.
You can create a somewhat limited access API user in Total Uptime, but I would have liked to restrict the API user to only modify TXT records.
Snap provides some certbot daemon process. (I’ll have to look into this)

Originally, I was passing the username and password (no Total Uptime doesn’t have API keys) as environment variables. But to remain compatible with certbot’s scheduled tasks, I switched to hardcoded variables. I may modify this again to support environment variables passed through a container. But that’s a job for another day.

I detail how to create the API user and use the scripts in the project’s read me.

https://github.com/nickadam/letsencrypt-totaluptime

Now I have certs for free that will automatically update that contain all the wildcard domains I need! Thanks EFF!

Have fun!

Let’s Encrypt Total Uptime

Example authenticator script #

Example cleanup script #

Total Uptime authenticator script #

Total Uptime cleanup script #

Example authenticator script

Example cleanup script

Total Uptime authenticator script

Total Uptime cleanup script