When researching options before writing the code referred to in this article, I came across a comment from someone I understand to be involved in the admin/author side of certbot. While I can’t find the link any more, they basically replied to someone stating that certbot would never provide plugins for DNS providers, which made sense since there are so many. However, shortly after finishing off this software, I came across the shiny new release of certbot, which  has some pre-written plugins for, you guessed it, DNS providers.

These plugins are not yet packaged for Ubuntu, so while I tried to do a quick test, I ended up still using my new script. If you’re starting out with certbot 0.22.0 or higher and get their plugins, they are probably a better option however this stuff will still work and is an option for you. And I’ll keep the code live for anyone who wants it, if for no other reason than it’s something else others can pick as an example to learn things from.

Introduction

Many of you will have heard of Letsencrypt, a service that enables creation of SSL certificates for use on websites (and anywhere else technically) for free; cost being one of the barriers to wider adoption of secured websites. I use this service for several sites, including this one.

I had a particular issue when it came to certificate renewal time that wasn’t supported automatically. In this post, I’ll explain a little about Letsencrypt and its client application certbot, and about “challenges” which are how Letsencrypt verifies you should be given a certificate you ask for. I’ll only be dealing with the authentication side of certbot: while it and a number of other clients are able to install the certificates for you as well, I did this part manually to fit within my Nginx configuration the way I wanted, but there is plenty of information out there if you want to learn more about that side.

My Setup

If you’ve read my previous posts you’ll likely have an idea of this already, but a simple overview for those who haven’t.

I run my domains on servers hosted in AWS on Ubuntu servers. I use Nginx (Open Source Version) as a hosting and proxying platform. My websites are, for the most part, exposed via CloudFlare (Free Tier).

For me, these certificates are securing the leg between CloudFlare and my server so users don’t normally see them, but they form one leg of the end-to-end security chain..

Letsencrypt

Letsencrypt can be managed by a number of software clients, however, the main one and the one I use is called certbot. Certbot allows the issuing of new certificates and the renewal of existing ones; renewal being important because the main caveat of these certificates is that they are only valid for 90 days. And the key part of this process is validating ownership in a challenge/response style setup, which can be done 3 different challenge methods.

HTTP-01

Probably the most common or most easily achieved method of validating your domain is HTTP-01. It involves posting a specified file in a specified location on the website. There are plugins for certbot that make this really easy for a number of hosting setups, including Nginx which I run.

For me, this wasn’t so practical. As I mentioned the DNS wasn’t pointing to the new server yet, and I didn’t want to mess around putting stuff on the old servers. Also, I had an additional site that I wanted a certificate for but I wasn’t able to modify the content to support this method.

TLS-SNI-01

I haven’t actually heard of anyone using this in the wild, and I haven’t tried it as it’s not functional when using CDN in front of your servers.

DNS-01

So this is the one I chose. DNS-01 involves adding a TXT record to your DNS with the specified value. For me this was great, it didn’t matter where my website was pointed yet, or if I could modify files. Because I could edit the DNS this way I was able to ensure zero downtime getting the certificates issued and in place before I did the cutover.

The downside to DNS-01

DNS-01 got me going well, however, when it comes to renewal time it’s not something that can be automatically supported. HTTP-01 has various options and settings to dump files in a specified location to automate that, but DNS is much more diverse and not all providers even offer an API. I gave it some thought and confirmed for my self that I couldn’t change to HTTP-01, mainly as I had this system issue certificates for domains that didn’t point to this server still (yeah maybe I’m a special case, but it is what it is).

Certbot has hooks to do things before and after any validation steps, so I decided I’d make it automated myself!

Automating DNS-01 challenges with CloudFlare

CloudFlare offers a great API, even on its free tier, so I decided to write a hook to automate the necessary updates. Since I’m that way inclined too, I’ve made it freely available, hosted here.

Basically, it’s a NodeJS script that runs through the following logic:

• Given a domain and a validation code
• Get a list of the users’ domain zones from CloudFlare and find which one is appropriate for this domain
• Check that zone to see if we already have an ‘_acme-challenge’ TXT record for the domain
• If we don’t create it. If we do, update it. In both cases, set it to the validation code provided.
• Query DNS and see if it’s updated yet and if not, look at the TTL for the query and wait that long plus a 10-second buffer and try again up to a configurable number of times until it is updated.
• Done.

This basically sets up the new validation information and waits until it is deployed, then returns allowing certbot to do it’s check and provided that succeeds, get the newly issued certificates. Now, of course, I’m simplifying a few things. It’s not certbot doing ALL the validation etc, but between certbot and the server they work it out; the finer detail isn’t too important for us just here.

What you get in the end is the ability to run a command like the following, and have it automatically manage the DNS-01 challenge setups required, which means it can be put into a cron job and not require manual intervention every couple of months!

sudo certbot renew --manual --manual-auth-hook "/path/to/node /path/to/hook.js"

As it turns out, the hook can also be used for certonly certificate issuing; it creates and manages validation just as well as the renew does. There is more detail and examples for installation and usage in the readme file in the code repository so if you want to check it out that’s a good next place to look.

Summary

Wow, that was a lot longer way of saying “I made a script to support DNS-01 challenge automation on certbot manual renewals” than I expected…

Anyway, please feel free to check it out, have a look at the readme which has some helpful info on getting it going, and I’ll try to address any questions or issues that come up. It’s far from perfect, but it works well for me so for now, it’ll do. Perhaps in the future I’ll try and improve any error handling etc. but no doubt that’ll come if I start seeing more errors I need to handle!

If you manage a website and you don’t have it secured with SSL, DO IT!!! There really is no good reason not to anymore.

Thanks for reading!

The post Automating Letsencrypt renewals with DNS-01 challenges appeared first on Random Insanity.

Domains

I thought domains would be easy. I have a pretty good understanding of domain registration and DNS, having worked with them for years so I went looking for a new registrar. I needed to relocate registration and hosting of both domains, and my main requirements looking around were ease of management and price.

As I mentioned in my previous post, I’d already settled on Amazon Lightsail for my site hosting, so Amazon seemed a good option for DNS to keep it all together also.

While the web hosting was basically identical for the 2 sites, the domain portion has a little (only a little) variety.

RandomInsanity.net.nz

As the first of the 2 sites I moved, I jumped into AWS Route53 and set up my DNS settings. Quite easy, and as best I can tell, will at worst cost me a couple of bucks a month if there are lots of requests (I’m thinking spamming etc; I don’t expect my readership to be that large!).

I also used Route53 and transferred my domain hosting over. Pretty easy, although it gave me some messages about my current provider having to approve it etc. which in the end I suspect is a just half generic warning for some other registries that require that. Route53 wasn’t the cheapest registrar I’d found here, but for ease of management, I thought I’d just keep it all together.

The one thing I’ll note is to keep an eye on your email. They send you info and anything you need to do which may not always be obvious on the console.

SaferHomes.nz

After randominsanity.net.nz being so easy, I planned to so saferhomes.nz the same way. And then I hit the real world.

The easier part first: instead of the Route53 hosting of the domain records, I used the inbuilt DNS part of Lightsail which as best I can tell has no particular additional charge beyond the Lightsail plan I was on. Same general result but different management location. Cool, done.

And then I went to transfer the domain.

Despite .nz having been commonplace for New Zealand for a couple of years now, it seems Route53 does not support this top-level domain (the support .co.nz, .org.nz, .net.nz, but plain old .nz). Gandi, the registrar AWS state they use supports it, but AWS themselves don’t. Bugger.

At first, I thought I must be doing something wrong, but after a while, I’d found numerous forum posts asking AWS when they would support it, right back from when it first became available, with no commitment beyond “it’s on our backlog”. Amazon’s own documentation, when you dig deep enough, has a list of the TLD’s (Top Level Domains) that they support and .nz is not listed.

So, there goes my plan of keeping it all together. In the end, I have moved the registration to Domains4Less and then just entered the AWS name servers for the domain into the records. Not ideal, but it seems to be working well.

Email

The email was much easier in the end. I actually had saferhomes.nz using Google’s GSuite already so I left it where it was.

For randominsanity.net.nz, my initial thought was that I would just find a way to forward it all to a Gmail account I currently tie my personal address in with. However, until now, I’d been basically using a full email account on my ISP network, with Gmail just grabbing email using POP and sending to that user using SMTP. Given I wanted to kill that mailbox, that wasn’t going to work.

I probably spent a couple of hours looking at solutions. Ideally just finding somewhere that’d charge me bugger all to do a blind mail relay for my domain, but that wasn’t really a service in wide supply. I looked at Amazon services, using their Simple Email Service which short of using a lot of work (S3 buckets and Lambda functions) probably wasn’t going to do what I wanted nicely anyway.

In the end, I just bit the bullet and setup another GSuite account for it. It costs me $5 to have my mailbox. I could have used Amazon’s WorkMail feature for a similar price but it didn’t look as polished as Gmail/GSuite so I erred on staying with the option I was familiar with since I had no intention of moving the saferhomes email.

Summary

Well, that’s really it at present. Maybe there won’t be a part 3…

In short, this is what I’ve ended up with:

RandomInsanity.net.nz

• Domain Registration: Amazon Route53
• Domain Hosting: Amazon Route53
• Website Hosting: Amazon Lightsail
• Email Hosting: Google GSuite

SaferHomes.nz

• Domain Registration: Domains4Less
• Domain Hosting: Amazon Lightsail
• Website Hosting: Amazon Lightsail
• Email Hosting: Google GSuite

SSL was a bit of a mission, and I can’t say I’m 100% on exactly which bit resolved it but it can be done. Basically following various Bitnami documentation.

Outbound email required configuration of SMTP credentials in WordPress but once that was done it worked fine, for what I needed anyway.

Hopefully, someone might find this something resembling interesting, and maybe even helpful. I’ll try to provide updates as the journey continues or as my setup evolves (I may still try moving everything to a more home-grown EC2/RDS type setup, but we’ll see how we go).

The post Migrating WordPress to AWS Lightsail (and other related changes) – Part 2 appeared first on Random Insanity.

Besides the obvious Broadband, I’ve got 2 websites (well, more than 2 but only 2 I’m going to keep alive) and a number of domains, as well as some email hosting. Moving all this has been a bit of a mixed journey and although it’s still going on, I thought I’d start putting together a bit of the story, challenges, lessons, solutions etc. for anyone else who is looking at moving such things for themselves.

After getting part way through this article, I realised how long it was getting, so I’m going to break it down into parts. Part 1 will be websites, part 2 will cover the rest.

TL;DR

I moved 2 WordPress sites to Amazon Lightsail, mail to Google GSuite, and domains are still a work in progress with domain hosting being a mix of AWS Route53 and Lightsail, and domain registration being AWS Route53 and Domains4Less. Do you want to know why? Read!

The story

After 11 years at my current work, I’d just tended to keep all the things I could in our systems. Partially because I built said systems so had great control/flexibility/etc. and partially because, for the most part, it cost me nothing. So as part of leaving, rather than keeping everything in place and paying their advertised rates, I decided a move was necessary!

For the purpose of this post, I’ll concentrate on 2 domains. The first one “randominsanity.net.nz” (AKA, this site) is my personal stuff, blog, email, basic. The second is a business domain “saferhomes.nz” (A company I’m a co-founder of). For both, we’ll look at domain, website and email (not necessarily in that order mind you).

Websites

As the generally larger part of the project, I started looking for a new web host first off. Both the sites I’m dealing with are WordPress, currently hosted using Plesk on my employers’ web server.

I looked at various locations, both web hosting and straight out VPS, with varying cost and resources. I liked the VPS idea in that I could probably just run both sites (they are pretty low traffic) on one server; I’ve certainly got the sysadmin skills to run the server and manage all that so why pay for someone else doing that stuff! In the end VPS still looked like it would end up more expensive than just outright hosting in many instances.

Due to already evaluating AWS for another project recently, this was one of the places I looked, almost out of pure curiosity more than serious intent. While the pricing on AWS can be quite hard to piece together, best I could tell I could probably do what I needed within their free tier (at least for the first year) and costs wouldn’t be tooooo huge beyond that.

So, I could setup a server on EC2, run a MySQL database on RDS, and even do a bunch of my DNS on Route53 (we’ll talk more about DNS soon). Appealing, not overly difficult concept, and just enough new stuff to make it an interesting challenge. However, as one might imagine, with less than 4 weeks of employment left, a job hunt in the works, a mortgage and all the other bills one acquires as part of being an adult, I came to the conclusion that I wanted something a little faster and easier, with less thinking.

Still, AWS had strong appeal as I wanted to do more there in future. Then I spotted Amazon Lightsail. Almost a sub-service of AWS, accounts linked but designed to be nicely functional for those not dealing with the normal AWS console. Lightsail touts a low-end plan of $5 (US) per month and the ability to deploy a pre-configured WordPress site, which I took an educated guess I’d be able to migrate my current sites overtop of without too much effort.

So Lightsail it was.

I started with my personal domain as a guinea pig first (experiments are rarely good on business domains). A new AWS account was easy to setup, and jumping over to Lightsail I had a generic WordPress site going in about 5 minutes flat.

Lightsail gives you a dedicated IP address for your container (I consider their deployed site/server as a container – maybe I’ve just been playing in the docker space for too long) which you can use to access your website before sorting any DNS; a nice way to be able to prep the site before migration.

The first gotcha I realised in Lightsail is that although you get a public IP, it’s my understanding (from reading rather than having his the issue) is that the IP they allocate is dynamic, which I assume means either that a new IP may get allocated of you restart the container, or (and this is probably more likely) that it’s a little like EC2 where you could kill and redeploy the container but you’d lose the ability to get the same IP again. While the latter wasn’t a major likelihood for me, in case it was the former, I made use of the easy spot in Lightsail to create a Static IP which I can then point to my container and use henceforth. A quick note for young players, while the Static IP is free (for up to 5), it will get charged if you don’t have it attached to a container!!

Okay, so I have WordPress running and accessible… now how to get all my content across?

I had a quick go with WordPress’s inbuilt export/import. That moved all my posts across just fine, but then the theme I was using no longer seemed to be available from WordPress. Cool, copy the theme directory over… nope, seems I’d done some customisation of the theme which I couldn’t recall, so the theme looked quite different out of the box.

Right, plan B (there’s always a plan B, and as is often the case, it was Google). Rather than reinvent the wheel, a quick Google confirmed I wasn’t the first to have this mission (Shock!), nor the first to document a little of the journey for the benefit of others. You can find the article I read here, but basically there is lovely plugin for WordPress (All-in-One WP Migration – love the creative naming) that does a really good and complete site export/import. Install the plugin at each end and follow your nose!

The one trick I did find during export was that since I was keeping the same domain, under the advanced options on the export screen I selected “Do not replace the email domain” as well as “Do not export spam comments”. Other than that it was plain sailing!!

Saferhomes.nz was pretty much identical. Although, while I chose the advanced export options on my first site, when it came to the saferhomes domain I forgot them and it caused a sufficient headache on the other end that I actually went back and re-did the export with them selected. If you’re changing domain I don’t expect it’ll be such an issue but it certainly helped for just moving the domain to another).

With saferhomes.nz though I have an additional challenge. Under Plesk I had it using Letsencrypt to automatically get and keep an SSL certificate up-to-date. Lightsail’s WordPress has no such easy feature, and it uses a custom server setup for WordPress and Apache from Bitnami. SSL is a must on this site, so I’m currently working on getting Letsencrypt running in this new landscape. It’s a work in progress, but I’ll ether make a new post on how I succeeded (yes, I’m backing myself to crack it) once I finish.

Other than that, 2 websites running on Lightsail and accessible via IP!

Wow, that’s a lot. Stay tuned (or go look) for parts 2 and probably 3 in due course.

The post Migrating WordPress to AWS Lightsail (and other related changes) – Part 1 appeared first on Random Insanity.