When researching options before writing the code referred to in this article, I came across a comment from someone I understand to be involved in the admin/author side of certbot. While I can’t find the link any more, they basically replied to someone stating that certbot would never provide plugins for DNS providers, which made sense since there are so many. However, shortly after finishing off this software, I came across the shiny new release of certbot, which  has some pre-written plugins for, you guessed it, DNS providers.

These plugins are not yet packaged for Ubuntu, so while I tried to do a quick test, I ended up still using my new script. If you’re starting out with certbot 0.22.0 or higher and get their plugins, they are probably a better option however this stuff will still work and is an option for you. And I’ll keep the code live for anyone who wants it, if for no other reason than it’s something else others can pick as an example to learn things from.

Introduction

Many of you will have heard of Letsencrypt, a service that enables creation of SSL certificates for use on websites (and anywhere else technically) for free; cost being one of the barriers to wider adoption of secured websites. I use this service for several sites, including this one.

I had a particular issue when it came to certificate renewal time that wasn’t supported automatically. In this post, I’ll explain a little about Letsencrypt and its client application certbot, and about “challenges” which are how Letsencrypt verifies you should be given a certificate you ask for. I’ll only be dealing with the authentication side of certbot: while it and a number of other clients are able to install the certificates for you as well, I did this part manually to fit within my Nginx configuration the way I wanted, but there is plenty of information out there if you want to learn more about that side.

My Setup

If you’ve read my previous posts you’ll likely have an idea of this already, but a simple overview for those who haven’t.

I run my domains on servers hosted in AWS on Ubuntu servers. I use Nginx (Open Source Version) as a hosting and proxying platform. My websites are, for the most part, exposed via CloudFlare (Free Tier).

For me, these certificates are securing the leg between CloudFlare and my server so users don’t normally see them, but they form one leg of the end-to-end security chain..

Letsencrypt

Letsencrypt can be managed by a number of software clients, however, the main one and the one I use is called certbot. Certbot allows the issuing of new certificates and the renewal of existing ones; renewal being important because the main caveat of these certificates is that they are only valid for 90 days. And the key part of this process is validating ownership in a challenge/response style setup, which can be done 3 different challenge methods.

HTTP-01

Probably the most common or most easily achieved method of validating your domain is HTTP-01. It involves posting a specified file in a specified location on the website. There are plugins for certbot that make this really easy for a number of hosting setups, including Nginx which I run.

For me, this wasn’t so practical. As I mentioned the DNS wasn’t pointing to the new server yet, and I didn’t want to mess around putting stuff on the old servers. Also, I had an additional site that I wanted a certificate for but I wasn’t able to modify the content to support this method.

TLS-SNI-01

I haven’t actually heard of anyone using this in the wild, and I haven’t tried it as it’s not functional when using CDN in front of your servers.

DNS-01

So this is the one I chose. DNS-01 involves adding a TXT record to your DNS with the specified value. For me this was great, it didn’t matter where my website was pointed yet, or if I could modify files. Because I could edit the DNS this way I was able to ensure zero downtime getting the certificates issued and in place before I did the cutover.

The downside to DNS-01

DNS-01 got me going well, however, when it comes to renewal time it’s not something that can be automatically supported. HTTP-01 has various options and settings to dump files in a specified location to automate that, but DNS is much more diverse and not all providers even offer an API. I gave it some thought and confirmed for my self that I couldn’t change to HTTP-01, mainly as I had this system issue certificates for domains that didn’t point to this server still (yeah maybe I’m a special case, but it is what it is).

Certbot has hooks to do things before and after any validation steps, so I decided I’d make it automated myself!

Automating DNS-01 challenges with CloudFlare

CloudFlare offers a great API, even on its free tier, so I decided to write a hook to automate the necessary updates. Since I’m that way inclined too, I’ve made it freely available, hosted here.

Basically, it’s a NodeJS script that runs through the following logic:

• Given a domain and a validation code
• Get a list of the users’ domain zones from CloudFlare and find which one is appropriate for this domain
• Check that zone to see if we already have an ‘_acme-challenge’ TXT record for the domain
• If we don’t create it. If we do, update it. In both cases, set it to the validation code provided.
• Query DNS and see if it’s updated yet and if not, look at the TTL for the query and wait that long plus a 10-second buffer and try again up to a configurable number of times until it is updated.
• Done.

This basically sets up the new validation information and waits until it is deployed, then returns allowing certbot to do it’s check and provided that succeeds, get the newly issued certificates. Now, of course, I’m simplifying a few things. It’s not certbot doing ALL the validation etc, but between certbot and the server they work it out; the finer detail isn’t too important for us just here.

What you get in the end is the ability to run a command like the following, and have it automatically manage the DNS-01 challenge setups required, which means it can be put into a cron job and not require manual intervention every couple of months!

sudo certbot renew --manual --manual-auth-hook "/path/to/node /path/to/hook.js"

As it turns out, the hook can also be used for certonly certificate issuing; it creates and manages validation just as well as the renew does. There is more detail and examples for installation and usage in the readme file in the code repository so if you want to check it out that’s a good next place to look.

Summary

Wow, that was a lot longer way of saying “I made a script to support DNS-01 challenge automation on certbot manual renewals” than I expected…

Anyway, please feel free to check it out, have a look at the readme which has some helpful info on getting it going, and I’ll try to address any questions or issues that come up. It’s far from perfect, but it works well for me so for now, it’ll do. Perhaps in the future I’ll try and improve any error handling etc. but no doubt that’ll come if I start seeing more errors I need to handle!

If you manage a website and you don’t have it secured with SSL, DO IT!!! There really is no good reason not to anymore.

Thanks for reading!

The post Automating Letsencrypt renewals with DNS-01 challenges appeared first on Random Insanity.

Domains

I thought domains would be easy. I have a pretty good understanding of domain registration and DNS, having worked with them for years so I went looking for a new registrar. I needed to relocate registration and hosting of both domains, and my main requirements looking around were ease of management and price.

As I mentioned in my previous post, I’d already settled on Amazon Lightsail for my site hosting, so Amazon seemed a good option for DNS to keep it all together also.

While the web hosting was basically identical for the 2 sites, the domain portion has a little (only a little) variety.

RandomInsanity.net.nz

As the first of the 2 sites I moved, I jumped into AWS Route53 and set up my DNS settings. Quite easy, and as best I can tell, will at worst cost me a couple of bucks a month if there are lots of requests (I’m thinking spamming etc; I don’t expect my readership to be that large!).

I also used Route53 and transferred my domain hosting over. Pretty easy, although it gave me some messages about my current provider having to approve it etc. which in the end I suspect is a just half generic warning for some other registries that require that. Route53 wasn’t the cheapest registrar I’d found here, but for ease of management, I thought I’d just keep it all together.

The one thing I’ll note is to keep an eye on your email. They send you info and anything you need to do which may not always be obvious on the console.

SaferHomes.nz

After randominsanity.net.nz being so easy, I planned to so saferhomes.nz the same way. And then I hit the real world.

The easier part first: instead of the Route53 hosting of the domain records, I used the inbuilt DNS part of Lightsail which as best I can tell has no particular additional charge beyond the Lightsail plan I was on. Same general result but different management location. Cool, done.

And then I went to transfer the domain.

Despite .nz having been commonplace for New Zealand for a couple of years now, it seems Route53 does not support this top-level domain (the support .co.nz, .org.nz, .net.nz, but plain old .nz). Gandi, the registrar AWS state they use supports it, but AWS themselves don’t. Bugger.

At first, I thought I must be doing something wrong, but after a while, I’d found numerous forum posts asking AWS when they would support it, right back from when it first became available, with no commitment beyond “it’s on our backlog”. Amazon’s own documentation, when you dig deep enough, has a list of the TLD’s (Top Level Domains) that they support and .nz is not listed.

So, there goes my plan of keeping it all together. In the end, I have moved the registration to Domains4Less and then just entered the AWS name servers for the domain into the records. Not ideal, but it seems to be working well.

Email

The email was much easier in the end. I actually had saferhomes.nz using Google’s GSuite already so I left it where it was.

For randominsanity.net.nz, my initial thought was that I would just find a way to forward it all to a Gmail account I currently tie my personal address in with. However, until now, I’d been basically using a full email account on my ISP network, with Gmail just grabbing email using POP and sending to that user using SMTP. Given I wanted to kill that mailbox, that wasn’t going to work.

I probably spent a couple of hours looking at solutions. Ideally just finding somewhere that’d charge me bugger all to do a blind mail relay for my domain, but that wasn’t really a service in wide supply. I looked at Amazon services, using their Simple Email Service which short of using a lot of work (S3 buckets and Lambda functions) probably wasn’t going to do what I wanted nicely anyway.

In the end, I just bit the bullet and setup another GSuite account for it. It costs me $5 to have my mailbox. I could have used Amazon’s WorkMail feature for a similar price but it didn’t look as polished as Gmail/GSuite so I erred on staying with the option I was familiar with since I had no intention of moving the saferhomes email.

Summary

Well, that’s really it at present. Maybe there won’t be a part 3…

In short, this is what I’ve ended up with:

RandomInsanity.net.nz

• Domain Registration: Amazon Route53
• Domain Hosting: Amazon Route53
• Website Hosting: Amazon Lightsail
• Email Hosting: Google GSuite

SaferHomes.nz

• Domain Registration: Domains4Less
• Domain Hosting: Amazon Lightsail
• Website Hosting: Amazon Lightsail
• Email Hosting: Google GSuite

SSL was a bit of a mission, and I can’t say I’m 100% on exactly which bit resolved it but it can be done. Basically following various Bitnami documentation.

Outbound email required configuration of SMTP credentials in WordPress but once that was done it worked fine, for what I needed anyway.

Hopefully, someone might find this something resembling interesting, and maybe even helpful. I’ll try to provide updates as the journey continues or as my setup evolves (I may still try moving everything to a more home-grown EC2/RDS type setup, but we’ll see how we go).

The post Migrating WordPress to AWS Lightsail (and other related changes) – Part 2 appeared first on Random Insanity.

Besides the obvious Broadband, I’ve got 2 websites (well, more than 2 but only 2 I’m going to keep alive) and a number of domains, as well as some email hosting. Moving all this has been a bit of a mixed journey and although it’s still going on, I thought I’d start putting together a bit of the story, challenges, lessons, solutions etc. for anyone else who is looking at moving such things for themselves.

After getting part way through this article, I realised how long it was getting, so I’m going to break it down into parts. Part 1 will be websites, part 2 will cover the rest.

TL;DR

I moved 2 WordPress sites to Amazon Lightsail, mail to Google GSuite, and domains are still a work in progress with domain hosting being a mix of AWS Route53 and Lightsail, and domain registration being AWS Route53 and Domains4Less. Do you want to know why? Read!

The story

After 11 years at my current work, I’d just tended to keep all the things I could in our systems. Partially because I built said systems so had great control/flexibility/etc. and partially because, for the most part, it cost me nothing. So as part of leaving, rather than keeping everything in place and paying their advertised rates, I decided a move was necessary!

For the purpose of this post, I’ll concentrate on 2 domains. The first one “randominsanity.net.nz” (AKA, this site) is my personal stuff, blog, email, basic. The second is a business domain “saferhomes.nz” (A company I’m a co-founder of). For both, we’ll look at domain, website and email (not necessarily in that order mind you).

Websites

As the generally larger part of the project, I started looking for a new web host first off. Both the sites I’m dealing with are WordPress, currently hosted using Plesk on my employers’ web server.

I looked at various locations, both web hosting and straight out VPS, with varying cost and resources. I liked the VPS idea in that I could probably just run both sites (they are pretty low traffic) on one server; I’ve certainly got the sysadmin skills to run the server and manage all that so why pay for someone else doing that stuff! In the end VPS still looked like it would end up more expensive than just outright hosting in many instances.

Due to already evaluating AWS for another project recently, this was one of the places I looked, almost out of pure curiosity more than serious intent. While the pricing on AWS can be quite hard to piece together, best I could tell I could probably do what I needed within their free tier (at least for the first year) and costs wouldn’t be tooooo huge beyond that.

So, I could setup a server on EC2, run a MySQL database on RDS, and even do a bunch of my DNS on Route53 (we’ll talk more about DNS soon). Appealing, not overly difficult concept, and just enough new stuff to make it an interesting challenge. However, as one might imagine, with less than 4 weeks of employment left, a job hunt in the works, a mortgage and all the other bills one acquires as part of being an adult, I came to the conclusion that I wanted something a little faster and easier, with less thinking.

Still, AWS had strong appeal as I wanted to do more there in future. Then I spotted Amazon Lightsail. Almost a sub-service of AWS, accounts linked but designed to be nicely functional for those not dealing with the normal AWS console. Lightsail touts a low-end plan of $5 (US) per month and the ability to deploy a pre-configured WordPress site, which I took an educated guess I’d be able to migrate my current sites overtop of without too much effort.

So Lightsail it was.

I started with my personal domain as a guinea pig first (experiments are rarely good on business domains). A new AWS account was easy to setup, and jumping over to Lightsail I had a generic WordPress site going in about 5 minutes flat.

Lightsail gives you a dedicated IP address for your container (I consider their deployed site/server as a container – maybe I’ve just been playing in the docker space for too long) which you can use to access your website before sorting any DNS; a nice way to be able to prep the site before migration.

The first gotcha I realised in Lightsail is that although you get a public IP, it’s my understanding (from reading rather than having his the issue) is that the IP they allocate is dynamic, which I assume means either that a new IP may get allocated of you restart the container, or (and this is probably more likely) that it’s a little like EC2 where you could kill and redeploy the container but you’d lose the ability to get the same IP again. While the latter wasn’t a major likelihood for me, in case it was the former, I made use of the easy spot in Lightsail to create a Static IP which I can then point to my container and use henceforth. A quick note for young players, while the Static IP is free (for up to 5), it will get charged if you don’t have it attached to a container!!

Okay, so I have WordPress running and accessible… now how to get all my content across?

I had a quick go with WordPress’s inbuilt export/import. That moved all my posts across just fine, but then the theme I was using no longer seemed to be available from WordPress. Cool, copy the theme directory over… nope, seems I’d done some customisation of the theme which I couldn’t recall, so the theme looked quite different out of the box.

Right, plan B (there’s always a plan B, and as is often the case, it was Google). Rather than reinvent the wheel, a quick Google confirmed I wasn’t the first to have this mission (Shock!), nor the first to document a little of the journey for the benefit of others. You can find the article I read here, but basically there is lovely plugin for WordPress (All-in-One WP Migration – love the creative naming) that does a really good and complete site export/import. Install the plugin at each end and follow your nose!

The one trick I did find during export was that since I was keeping the same domain, under the advanced options on the export screen I selected “Do not replace the email domain” as well as “Do not export spam comments”. Other than that it was plain sailing!!

Saferhomes.nz was pretty much identical. Although, while I chose the advanced export options on my first site, when it came to the saferhomes domain I forgot them and it caused a sufficient headache on the other end that I actually went back and re-did the export with them selected. If you’re changing domain I don’t expect it’ll be such an issue but it certainly helped for just moving the domain to another).

With saferhomes.nz though I have an additional challenge. Under Plesk I had it using Letsencrypt to automatically get and keep an SSL certificate up-to-date. Lightsail’s WordPress has no such easy feature, and it uses a custom server setup for WordPress and Apache from Bitnami. SSL is a must on this site, so I’m currently working on getting Letsencrypt running in this new landscape. It’s a work in progress, but I’ll ether make a new post on how I succeeded (yes, I’m backing myself to crack it) once I finish.

Other than that, 2 websites running on Lightsail and accessible via IP!

Wow, that’s a lot. Stay tuned (or go look) for parts 2 and probably 3 in due course.

The post Migrating WordPress to AWS Lightsail (and other related changes) – Part 1 appeared first on Random Insanity.

That’s when the real fun started. Well, fun being a fairly loose term here…

Once activated, the card made the funds I’d already credited available. However, when I went to load it to Hulu (the whole reason I started down that path) it wouldn’t work. Eeek!!

I tried it out on US iTunes also, with no luck. Slightly off topic however, I found the option (which I’d probably just missed perviously) to continue with the iTunes account without a funds source loaded. With that job done, I’ve successfully been downloading free apps from the US store to my AppleTV (Gen4) and iPhone/iPad. There is a little overhead switching between NZ and US stores but thus far it’s meeting a need.

But back to Hulu. Plan B (or maybe that was Plan F+) turned into setting up a US PayPal account. This fortunately worked, and my new card is setup as a funding source to it. I then used the PayPal account as the funding source for Hulu. A very round about way, and not yet confirmed to be functional long-term however next billing date, in theory, Hulu will bill my PayPal account which will in turn use my card to fund it. We’ll of course see how well that works.

So the question as to why it didn’t work. Here, 3 things are worth noting:

1. Details on iTunes suggests that it’s failing there as Apple can actually compare some part of the billing address loaded against the card to details you give. So even if you tell it all the right details, and it’s a US issued card, it can still pick stuff isn’t matching.
2. Hulu doesn’t ask for anything more address wise than an area code. But, I found amongst some forum a note that Payoneer cards are issued from somewhere in Central America. So perhaps it’s looking more specifically for North America issuance. Anyway, as yet I haven’t been able to confirm.
3. I tried a couple of other sites that require US cards and that hadn’t worked with either my NZ Visa Debit or the Entropay card, and they worked. So there are numerous factors at work. As yet I can’t even begin to speculate all the variables and which is impacting where.

What’s next you may ask? Well, in due course I might setup another card and get it sent to a US address that will forward it on as mail, meaning there is a US address loaded against the card. Other than that, I’ll probably test a few more services over time; I’ve specifically been asked about some geo-restricted Amazon media services.

Other than that it’s been both a success and to a smaller extent a failure of an experiment. Well, some things I’d wanted to work didn’t work the way I wanted; you could argue that as a failure, or perhaps since it was all an experiment you could say it was a success that you got a result from said test.

And on that note I shall leave you to ponder experiments for yourself. Feel free to ask questions, I’ll answer any I can! =P

The post Payoneer – An internet payment journey – Part 3 appeared first on Random Insanity.

Basically, as some of you who know me or have read other bits of my blog will know, I subscribe to Hulu (among other services) which is a geo-restricted service locked down to the US.

Geo-restriction is pretty easy to overcome. As I mention in another article, there are a bunch of services out there to assist. I’ve used Unblock-US in the past but currently use MediaHint. It works great and has a browser plugin that helps when on the go (not at home with my permanent setup). However, Hulu requires a little more than looking like you’re accessing it from the US; it requires a US credit card also!

For around a year now I’ve been using Entropay, which gives you a virtual US credit card to run pre-pay. It also worked great for the last year (with the exception that my NZ based bank requires disabling some security to top it up as they deem it a suspicious site), however, the card has now hit its expiry date so I’ve gone to Hulu to update the card info… only to find it no longer accepts it! I’ve recently also had a go at setting up a US iTunes account with the same issue that iTunes wouldn’t accept the Entropay card as US based. I’ve no idea why, or what they use to determine such things, but it just didn’t work!

So, I’ve done some looking around to see what other services are out there. The list has certainly grown since I first looked! This time, I’m having a go at Payoneer. I decided I’d document a little of how it goes to help other people evaluating it also.

Thursday 11 – Morning

So far I’ve signed up on the Payoneer site and handed over a bunch of detail. It expects that you’re not in the US so you don’t have to fudge details for any of that, and apparently, they are now sending me a physical US issued MasterCard! They offer a service to help people receive payments from all over the world, and have a couple of options on how to make use of received funds – and this is one of them.

They ask for some “government issued ID” – I’ve given them a drivers licence. Although they took this detail and rapidly approved my account request, after I topped up the card (basically by requesting a payment from myself for some fictional service) I got another email asking for info on my business (service or product based, refund policy, website etc.) as well as a scanned copy of my licence. If I hadn’t heard enough good stuff about this place I’d be a little dubious already, but I’m confident enough that it’s legit that I’ve provided the necessary detail and am awaiting the deposit of my “received funds” in my US account.

I requested a payment of $20USD as the “seller”, and as the “buyer”, I had the option to either pay the transaction fee myself on top of the $20 or deduct it from the seller’s payment. I chose the former as it doesn’t really matter given I’m both parties, but for reference it was $0.62 fee, which I assume is some percentage calculation of the payment amount.

That’s as far as I’ve gotten so far. Now we wait… Or I wait anyway because you probably don’t care THAT much!

The post Payoneer – An internet payment journey – Part 1 appeared first on Random Insanity.

One of the greatest draw-cards for me was that it could control other devices that use 433MHz, which while it may not sound impressive, included some readily available “remote-control” power sockets. These sockets came with a remote control with a series of 4 on/off buttons – basically you plugged these into your wall socket, then something into them and you could use the remote to flick the switch on and off without going near it. This instantly appealed to me, and my plan was to put the lights of my 2 aquariums onto them, and use the NB rule capability to simply turn the lights on and off on a schedule!

This was a very meagre requirement and setup without any significant problems. But over time a few issues cropped up. The main one being that the NB device interacted with some cloud hosted brains to provide the management interface, and obviously some of the actuators as when the internet connection was down, it wouldn’t trigger anything. It also had an issue that (at least using its wireless connection which was necessary to get it close enough to the devices that needed to be controlled) if the connection dropped to the NB for any reason, it seemed to need to be rebooted to get it to reconnect.

The other real issue is no fault of NB but more a function of the communication style of 433MHz devices – sometimes the NB would tell the socket to turn on or off, but the socket for whatever reason wouldn’t get the hint. The NB had no idea if its command was followed or not; it was a fire-and-forget sort of communication.

Those issues aside, it ticked along for a fair while being my overpriced power timer (I should mention that before the NB I had 2 analogue power timers that did the job fine, they were just a little noisy and if the power was out for a period I had to manually readjust them so their view on time wasn’t offset). It was a geeky, lazy solution and conceptually I loved it.

Then it died.

A couple of months or so ago, the NB stopped doing anything. It wouldn’t even acquire an IP address from the network. And at the time I didn’t have the correct adapter to get a monitor on it to confirm my suspicion that it was just bricked. But I ignored it for a while and just started using the original remote for the sockets, which worked fine but meant that the on and off time for the lights was a little less scheduled, and if I went away, they got no lights (not a huge issue but I liked the idea of a stable rhythm for them!).

At some point along the timeline, I’d played a little attention to some other things in the wild around home automation, initially with the thought I could find something that wasn’t so reliant on internet connection to actually be able to control the thing right next to it.

In this process I got and played with a RaspberryPi (actually the Pi was inspired as an alternative to a computer to display an web based dashboard on a big screen TV for work) and liked it but found it had some limitations in its processing ability (it got bogged down easily). This led me to the BeagleBone Black (BB) – same concept but with a little more grunt, and coincidentally the board that the NB was built on top of. So I got a BB and it sat doing very little for a few weeks until I came across OpenHAB!

The remainder of this post will be about my journey with the BB and OpenHAB, where I am today with it and some of the fun I had along the way that might even help someone else!

I should mention that I’m terrible when it comes to research on these things too. A number of articles I found online that helped me with bits, probably held the answers to the questions I came up with along the way, but due to a terrible habit of skim reading along with a tendency to quickly locate just the bit I need at a point in time then lose track of the rest of the content by the time I hit my next hurdle, a review of some of them leads me to realise there were a bunch of hints that I’d just overlooked. Actually, a bunch of the things that I had read that just really made no sense to me at the time (mainly thanks to some specific terminology) now make perfect sense and show themselves to be answers – I was just to dumb to see it and ended up with a lot of trial and error to realise what they were actually saying.

Right, so OpenHAB is an open source home automation system basically. It’s software that can run on probably endless devices (Java based) and has the ability to interact with a whole raft of devices.

Today, I have it talking to my Philips Hue bulbs, as well as 2 Z-Wave power socket controllers, and a Fibaro motion sensor (which actually does motion, temperature, light metering and an accelerometer). All these along with the control stick required for the Z-Wave were sourced from Smart Things NZ, a Christchurch based supplier of lots of cool home automation stuff run by an extremely helpful guy called Ben Jones.

Some credit to Ben here. After a very disjointed email to him late one night, a conversation started and I found Ben to not just be a sales guy, but a real enthusiast, strong member of the home automation / OpenHAB community who was great with suggestions and eager to help. In fact I was led to his site by links to him from other people whom he’d helped in the past. He even happens to have written some of the bindings for OpenHAB that I was planning to use so overall helped me validate that the concept in my mind actually had some realism in its ability to happen in real life too.

Initially I set up my BB using the default operating system, and got OpenHAB running on it. I set it up with my Hue bulbs and had a play. Great, that’s fun, but I didn’t get much past that. When I tried to install a program called mosquitto to act as an MQTT server (for an iPhone app OwnTracks), I spent hours bashing my head against a brick wall. It was around this time that I started talking to Ben and he mentioned he used MQTT a lot in his setup, and he was running on Ubuntu. I’m pretty used to Debian and not to bad with the quite similar Ubuntu, and the BB can support either of these distros so I went with Ubuntu and flashed my BB, starting from scratch.

With that done, mosquitto was a 30 second apt-get install away and I had OpenTracks talking to it minutes after that.

Then my Z-Wave toys arrived. I concentrated on the power sockets first as getting the timed light control on my fish tanks was the main thing I knew I wanted to achieve. I had to reinstall OpenHAB, which went fine, then plugged in the Z-Stick, put the Z-Wave addons in the right directory and fired up OpenHAB.

Joining the sockets (the actual term for this is ‘including’, but I tend to call it pairing but it’s all the same thing) was as easy as a press on the Z-Stick (although I found that this does need to be done with the Z-Stick unplugged from the USB. Putting it into learn mode while plugged in was technically doable but disconnected it from the BB requiring a reboot). The doco for the Z-Stick actually gives a description of taking the Z-Stick to where you want the new device to be and pair it in there which made more sense to me after I’d got everything setup overall. The manuals with both the Z-Stick and the sockets had some good information even if some of it meant nothing to me at the time, but it was certainly enough to get started.

Anyway, I ramble…

Next, I added these devices into OpenHAB as items. This is pretty much where you define all the things you interact with; all your endpoints of sorts. Getting the Hue bulbs going in the first round had dragged me through the doco for these – the OpenHAB site has some good info on the basics and the doco for most bindings gives good detail and examples.

Problem… How the hell do I define the sockets as items? The Z-Wave binding (actually I think that might be another one of Ben’s) had great doco, so after reading that and another article^[2] (actually where I first found mention of Ben) I ended up stealing some code from Jan-Piet Mens’ article and turned it into this:

Switch Z_Socket1
   "Big Tank"
   (Lights)
   {zwave="3:1:command=SWITCH_BINARY"}

Switch Z_Socket2
   "Small Tank"
   (Lights)
   {zwave="4:1:command=SWITCH_BINARY"}

At this point, I was really just guessing which device was which (the 3 and 4 in those Z-Wave lines identifies which device it is), but during the OpenHAB bootup, I noticed it said there were 4 Z-Wave devices (I’d actually already paired the motion sensor at this point but I’ll detail that later) so I did some trial and error. For the trial and error bit, I had to get them on the sitemap so that I could actually do something with them. My initial sitemap was something like…

sitemap morse label="Main Menu" {
   Frame label="Fish tank lights" {
      Switch item=Z_Socket1 label="Big tank (900L)" icon="switch"
      Switch item=Z_Socket2 label="Small tank (215L)" icon="switch"
   }
}

W00t! It works. I can reliably switch them on and off, and if I do it using the button locally on the actual socket, when I reload the phone app, it shows the correctly updated state. The last bit to get these doing what I wanted was actually done after a munch of frigging around with the motion sensor (I’d thought, that was easy, I’ll just do the same for this and then it’s all in! – boy was I wrong). Here my first “rule” was born. The first one was simply, turn the two sockets (and hence the fish tank lights) on at 7am each day. Not too complex, no multiple criteria or anything like that. A little code acquisition from other examples ended up as:

rule "Switch fishtank lights on at 7am"
when
   Time cron "0 0 7 * * ?"
then
   sendCommand(Z_LightsTank1, ON)
   sendCommand(Z_LightsTank2, ON)
end

And what do you know? 7am the next morning, the lights came on. Easy as!! The lights off rule looks almost identical:

rule "Switch fishtank lights off at 8pm"
when
   Time cron "0 0 20 * * ?"
then
   sendCommand(Z_Socket1, OFF)
   sendCommand(Z_Socket2, OFF)
end

I’m still not 100% sure I’ve got the metering working for the feedback from the switches, but an example of my item for one for now is:

Number Z_Socket1_Power
   "Big Tank Lights [%.1f kWh]"
   (zPower)
   { zwave="3:1:command=METER,meter_scale=E_KWh,refresh_interval=600" }

Nothing to it! Then it was time to come back to the motion sensor. After a couple of evenings of frustration and large quantities of googling (I was determined to work it out without actually having to ask someone like Ben – I’m stubborn like that!) I worked out a couple of key things (these realisations came from reading a number of articles and forums, I’ll put a list of links that I found helpful at the end of the article as all of them added random bits to various pieces of the puzzle)

• Getting the motion sensor “included” is easy. But it’d different in that what it does automatically isn’t enough to get the information flowing back. I’ll give some more detail on that shortly.
• HABMin is a great addon to OpenHAB, and one I ended up using. But on the BB, it KILLS CPU, makes everything lag, causes messages to drop (which I was watching in the log) and actually made me think that the BB wasn’t going to be grunty enough! In the end, when I wasn’t accessing HABMin, CPU usage died down and everything was happy, so if you’re using a BB or other limited resource computer, try not to use HABMin when you don’t have to (Actually having it open in a webpage is all it takes! Having it installed is fine, just don’t open the page too often)
• Battery powered devices work a little differently – you need to be patient.

Patience was not my strong point. I ended up on the right track but got there too early first time around. Battery powered devices spend a bunch of their time sleeping. When they wake up (the motion sensor default was 7200 seconds), the check the queue for messages and then go back to sleep; a little like me on a Sunday. The outcome of this is that it wasn’t until I’d left it overnight that (as far as I can tell) it properly finished the inclusion and I could actually find the bits I thought I was looking for (the next paragraph) so if you’re having trouble, sleep on it! You can hurry it up a bit by “waking up” the device but I obviously hadn’t done that enough so I still suggest the overnight to sort things out. =P

As well as inclusion, the motion sensor requires you to manually sort out some association. Now that’s the term I didn’t understand totally at first when I read it which meant I skipped over that stuff and kept looking – right past the answer! So yes, you need to change the association, basically telling the motion sensor to send info back to the Z-Stick. To do this, you can use a number of apps (apparently) but I used HABMin for it. So, loading up HABMin, I… Chose Configuration along the top -> Chose Bindings down the bottom left side -> Click on the Z-Wave one -> Chose Devices from the stuff that loads up. This gives you (hopefully) a list of all the Z-Wave nodes you have (this would have saved me guessing about the switches) – 3 had a green light, but the motion sensor had a grey light. Expanding the one for the motion sensor, you will find something about Association Groups, and under that you’ll find Controller Updates. Now here’s another bit that after all the reading I was still confused about – some say you have to change all the things you see under Association, others say all under controller updates, and finally I found someone who mentioned the controller – of course, the one you need to update is whatever node your controller (the Z-Stick) is (probably node 1). So, change Node 1 from Non-Member to Member. Make sure it’s taken (I reloaded HABMin to double-check) and then close HABMin go sleep on it again. HABMin has no part in it after this and I suspect the load it was putting on things actually slowed the process down just because I had the page open.

The next day, I opened HABMin again and found a green light where the gray one had been. I think that rated as success. So, closing HABMin again, I mashed together some of the examples I’d found and ended up with this (note, due to me redoing the inclusion a couple of times, instead of its original Node 2, the motion sensor is Node 6):

Number FibEye1_Motion
   "Lounge Movement: [MAP(movement.map):%s]" (Fibaro,Motion)
   { zwave="6:1:command=sensor_binary,refresh_interval=600" }

Number FibEye1_Alarm
   "Lounge Alarm: [MAP(earthquake.map):%s]" (Fibaro,Alarm)
   { zwave="6:1:command=sensor_alarm,refresh_interval=600" }

Number FibEye1_Lux
   "Lounge Light: [%.2f Lux]" (Fibaro,Lux)
   { zwave="6:1:command=sensor_multilevel,sensor_type=3,refresh_interval=600" }

Number FibEye1_Battery
   "Fibaro Motion Sensor 1: [%d %%]" (Fibaro,Battery)
   { zwave="6:1:command=battery,refresh_interval=86400" }

Number FibEye1_Temp
   "Lounge Temp: [%.1f C]" (Fibaro,Temp)
   { zwave="6:1:command=sensor_multilevel,sensor_type=1,refresh_interval=600" }

The labels are a bit screwy but I’m actually moving away from defining labels in most of my items, and doing it in the sitemap instead. Speaking of sitemap, a snippet that shows some of that in action there is:

Frame label="Stats" {
   Text item=FibEye1_Battery icon=energy
   Text item=FibEye1_Temp icon=temperature
   Text item=FibEye1_Lux icon=sun
}

It took a while again for info to start coming through. The first time around, I hadn’t put a refresh_interval on the items… That wasn’t helpful, but equally setting too short a refresh time had a notable impact on CPU load, so in the end I settled for 600 seconds (5 minutes) and found that a good balance.

It was a long journey but the things I’ve learnt just from dealing with the motion sensor have given me a much better understanding of Z-Wave and I think will make it easier when (yes when, not if) I get some more devices to add in!

So with all those sensors in the mix, I now have 2 fish tanks with lights going automatically, a sensor that logs (I setup mongodb for persistence) and an interface on my phone where I can see/manipulate all the bits. I’ve added in the OwnTracks stuff but not had much luck yet – I haven’t spent much time to sort it all out but the main issue is updates often not getting back from phone to MQTT which is more a network issue than anything I think. I’ve also setup Network Health and used it for presence (can it see my phone’s wifi IP on the LAN) and monitoring my internet connection is up in general. And Charts have been fun too – I have charts for the power usage from the sockets, and for the light levels and temp from the motion sensor for example. I haven’t yet done anything based on the motion sensor itself but there are plenty of ideas in my head. Next “smart” rule is probably going to be around the Hue bulbs and the light level in the lounge at the time, but I’m still sorting out some niggles with the Hue binding overall. No doubt I’ll post some on that when I’m happy with it.

Expect that I’ll go over this article a few times and re-word, improve some references or just generally tweak bits. Some more images when I have the time perhaps. But this is a start to get it out there at least!

That list of links that I found helpful in one way or another (in no particular order)
https://github.com/openhab/openhab/wiki/Z-Wave-Binding
https://github.com/openhab/openhab/wiki/MQTT-Binding#transport-configuration
https://github.com/openhab/openhab/wiki/Mqttitude-Binding
http://jpmens.net/2014/01/14/a-story-of-home-automation/
http://forum.mysensors.org/topic/302/openhab-mqtt-tips-hints
http://blog.scurb.se/homeautomation/home-automation-an-early-case-study/
https://groups.google.com/forum/#!topic/openhab/8uMM-xT_WoE
https://groups.google.com/forum/#!msg/openhab/bb9bB71zgsM/J2bNY6dnZvsJ
https://groups.google.com/forum/#!topic/openhab/bb9bB71zgsM

Aaand a random selection of screenshots from the OpenHAB app on my phone at the moment…

[1] These were the older style NinjaBlocks; since then they have come up with NinjaSphere that I haven’t looked into much so anything comments I make may no longer be current.
[2] http://jpmens.net/2014/01/14/a-story-of-home-automation/

The post Home Automation, OpenHAB, Z-Wave appeared first on Random Insanity.

Ubuntu 13.04 (Codename Raring Ringtail)

So a wee while ago I decided I wanted to get a decent *nix operating system running at home again, because (let’s face it) running Windows all the time is enough to drive anyone crazy.

Don’t get me wrong, I’m not ditching Windows all together. I still need to keep it around for some apps, wether I like it or not.

After a bit of googling and with no small part played by reviews from Nixie Pixel (she’s got some great stuff on Twitter, Facebook and YouTube!) I settled on Ubuntu. It is similar to Debian in its terminal commands and it has a pretty good reputation as a nice stable desktop OS. Raring Ringtail (the codename for the latest version 13.04) was hot off the press too, so after a quick download and burning the ISO to disk I made space on one of my drives and booted ready to install.

It wasn’t without hiccups. My computer is self-built and Linux compatibility wasn’t a consideration at the time I got all the parts. The first issue I ran into was installing GRUB to the wrong drive… after several revisions of harddrive config in the machine, I managed to lose track of where the current boot loader was living. That fixed though, I got the install completed and booted Ubuntu!

Or at least tried to. For some unknown reason it just booted to black screens. No small amount of time buried in Google came up with various similar reports and a myriad of proposed fixes. I came to the assumption that it was all due to the graphics cards I was running (dual Nvidia GT250s), but after playing with config options in GRUB I managed to get limited access, and while snooping around I found an entry in fstab for the floppy drive.

Now I have no idea if this was actually related or not but I removed the entry as I have no floppy drive in my machine, and things suddenly started to play ball. This got me a clean boot with one screen working. I have four screens on my machine so this was the obvious next battle.

I thought this one should be simpler, given the Nvidia config app, but several aborted attempts to configure them this way got me two screens at best!

I resorted to writing the config file (xorg.conf) manually. I’ve never really played with any GUI on Linux before so this was new ground for me. Another dive into Google got me a bunch of samples and after more reboots than I can count I managed to get all four screens to show, and actually in the right arrangement as they sat on the desk!

But (there’s always a but) there was no menu bar, no dock or control options to be seen. And that’s where I’m currently at. Experimenting with finer changes to the xorg.conf file, trying Gnome, more Google madness, etc. It’s quite frustrating to have four screens working but unusable, or one (or two) screens working perfectly.

I don’t know if it’s a display manger issue, a config issue, a driver issue, or the little gremlin in the computer having a laugh, but I’m still working on it and determined to figure it out. When I do, I’ll post more…

In the mean time, have any of you had a similar battle? Any hints/tips/pointers for a guy who’s usually ssh’d into 5 different servers but basically a newbie when it comes to using nix as a desktop OS?

The post Raring Ringtail (Ubuntu 13.04) appeared first on Random Insanity.

Bits

In the digital world, everything breaks down to binary. Binary is just “base 2”, or 0’s and 1’s.

When it comes to measuring data, a binary 0 or 1 is a “bit” (short for binary digit). So think of bits as the building blocks of what we’re talking about.

You probably see more about bits than you think. Speeds for data transfer (how fast your download is going, or how fast your connection plan is) are typically represented in bits. But just like we deal with grams and kilograms to represent varying scales of weight (where a kilogram is just an easy term for 1000 grams) we do the same for bits.

Now there’s two ways that this can be done. In dealing with weight, we do things in tens, hundreds, thousands etc. (decimal) and we can do that for bit’s too.

So the same way kilo is a prefix for kilogram to represent 1000 grams, we use the kilo prefix to bit’s to represent 1000 bits. There’s another prefix for each increasing power of 1000. In the above table you see mega, giga and tera; it keeps going too with the next few being peta, exa, zetta and yotta tops out as 1000⁸ or 1,000,000,000,000,000,000,000,000 bits, or 1000 zettabits! Confused yet?

Well if you haven’t fallen asleep yet, let’s now look at the binary power version. This is the more pure version in my opinion but some people find the maths of 10’s easier.

These go higher too but you get the idea. The really confusing part is that the terms for the decimal versions used to be commonly used (and sometimes still are) when the decimal version is actually meant (even I usually mean 1024³ when I say Gigabit) so it’s often hard to know which is really intended. The units can also often be shortened too, so you’ll see Mb instead of Mbit (or Mibit), and the main thing to note is the lower case “b” which points to bit’s.

If you’re on a lovely 100/50 “UltraFast Broadband” fibre connection, the 100/50 really means you have a rated download speed of 100Mbit’s per second down, and 50Mbit’s per second up. A point that illustrates also that speed measurements represented by bits are also given against a unit of time (usually seconds). The speed you practically see on your connection varies significantly and certainly don’t expect to see every download or speedtest you see to show 100Mbit/s – as I said above there’s a bunch of info for another topic.

Bytes

A byte is a collection of 8 bit’s. Simple as that. The way those 8 bits are arranged defines what the byte represents. There are lots of different standards as to how that byte is interpreted but that’s another topic all together.

We do the grouping stuff here too, so we get Kilobyte, Megabyte, Gigabyte etc. We get the abbreviated units too, so KB, MB, GB (note the uppercase “B” here points to Byte’s instead of Bit’s). Technically the binary versions here get the abbreviations KiB, MiB, GiB etc., but again the terms get mixed up regardless of intention.

Bytes are usually not used to represent speed, but more commonly storage. So your harddrive in your computer will usually be measured in GB or more commonly now TB. For example I have several 2TB drives in my computer. If your internet connection has a data cap it too will usually be represented in byte’s instead of bits.

Some math

If you’re still with me, let’s do a little exercise to put what we’ve learnt to use.

Let’s work out if your downloads actually go at 20Mibit/s, how long will it take you to download a 4GiB file (for clarity, I’m going to use the binary version of both here).

First I’ll do some work on the speed to get it to a more useable number.

Times 20 by 60 to go from Mibit/s to Mibit/minute. 20 * 60 = 1200Mbit/minute.
Now, since there’s 8 bit’s in a byte and we measure our file in byte’s we can divide the Mibit’s/minute by 8 to get MiB/minute. 1200 / 8 = 150MiB/minute.

Now let’s get the file size from Gigabytes to Megabytes.
1 GiB = 1024 MiB so we times our 4GiB by 1024. 4 * 1024 = 4096MiB.

So, now we have a 4096MiB file, and we can download at 150MiB per minute.
4096 / 150 = 27.3 (minutes).

So all things going well (which they rarely do) you could download your file in a little under half an hour.

Another application of this sort of maths is working out how much data you will use while streaming video or music from the internet. I won’t put as much explanation around this one, but we’ll work out how much data I’ll use if I steam an hours’ worth of Netflix at high-definition which Netflix rate as about 5Mibit/s.

1 hour * 5Mibit/s = ???
60 minutes * 5Mibit/s = ???
60 minutes * (5 * 60)Mibit/minute = ???
60 minutes * 300Mibit/minute = ???
60 minutes * 300Mibit/minute = 18000Mibit’s
60 minutes * 300Mibit/minute = (18000 / 8)MiB’s
60 minutes * 300Mibit/minute = 2250MiB’s
60 minutes * 300Mibit/minute = (2250 / 1024)GiB’s
60 minutes * 300Mibit/minute = 2.2GiB’s

So for every hour you watched you’d use over 2 gigs of data. If your plan has a 100GB (or GiB depending on what they mean) quota that’s not too bad. If you have a 3GB quota… you’re going to run out of data pretty quick.

More reading? Wikipedia as usual has a tonne (another of those units of measurement) of information. You can start with their article on the Bit and go from there.

That’s enough for now. As usual, feel free to ask any questions below and I’ll try to help where I can. Any corrections are welcome also (Hey, I never claimed to be perfect!).

The post Bits and Bobs (or Bits and Bytes) appeared first on Random Insanity.

But… there’s still a lot of stuff that the states are keeping to themselves. Sites like Netflix, Hulu Plus and HBO GO (among others) that offer movie or TV streaming services in the US and UK for example, are not available in NZ. Netflix is one that’s received a lot of attention lately – for those of you who aren’t familiar with it, Netflix allows you to stream a large selection of movies on demand for a simple subscription fee, not a pay-per-movie. There’s streaming TV also, the options are vast.

Now, there’s been a lot of people posting details on ways to get around this geographic restriction, and even more speculating why it exits in the first place, but be it political, commercial or whatever other reason, get around it we shall. I decided it’s time to help piece together some of the bits around the net and hopefully give enough info for novice and advanced user alike.

I’m going to show you a few different scenarios to get yourself connected to all that content, but before we start there’s a few things to be aware of:

• Legality: The general consensus is that there’s no laws being broken here. Various Internet providers in NZ and further afield have even setup a service to do this for you. BUT I’m no lawyer. If it’s something you’re REALLY worried about, you should have a look around make your own mind up.
• The content is coming from the US. That’s a fair way away even in internet terms, which imposes some limitations. The main one you’ll notice with Netflix is that you might not get their full HD movies (See here for more info on connection requirements and remember it’s not your plan speed but your speed to their server in the states!). You can still stream those movies, they will just be delivered to you at standard definition which is still usually plenty for most people.
• Streaming movies uses your bandwidth. This is the same no matter where it comes from (as a general rule) but keep an eye on your data usage and take note of your internet plan!
• This works now, and has done so for quite some time. But as it’s not standard usage, providers like Netflix may make changes in the future that stop this from working. They haven’t made any noises about it, but it’s a possibility either directly or indirectly.
• Some will tell you this is what you do to replace Sky. If you’re a Sky subscriber (as I am) you need to have a look and see if just one or the other is the best option, or if you want to keep both for the things you want to watch. You might for example keep the basic Sky package for the channels it offers but biff your Sky Movies in favour of putting those funds towards Netflix. Basically, make your own mind up based on content and budget.

Lets start with probably the easiest scenario. Do you have an Apple TV? Yes? Excellent, you can get up and running pretty fast.

Netflix on Apple TV

Netflix on Apple TV

Difficulty: Low
Time required: 10 to 30 minutes for most users
Costs (at the moment anyway): $4.99/month (USD) for remote DNS, and $7.99/month (USD) for Netflix itself

This setup on the Apple TV will allow you to watch Netflix through it using what’s called a remote DNS, without messing with any of your other devices (e.g. Your desktop computer still won’t be able to access Netflix). This is good for someone who’s not that tech-savvy as there are sometimes unintended consequences of having all your devices use the remote DNS service.

What you should know before you start: Part of this setup requires you to basically tell your Apple TV that you live in the US. This works fine for our purpose but does have a side effect that you can’t use your NZ iTunes account from the Apple TV. Not a big issue for most, but if you watch movies purchased through Apple iTunes you need to decide if Netflix is a suitable replacement.

Still keen? Sweet! First head on over to Unblock us. It’s a service that helps you get access to the right information (it’s the remote DNS service I just mentioned). If you want to know more there’s info on the site, but in my opinion it’s one of the best ways to do it, so just do it. You can register for a free week trial, or $4.99/month (USD). They have instructions for lots of devices, but the version of it for Apple TV is here.

Note that you have to change both DNS and location settings to make it act like it’s in the US. Once that’s setup, you should see and be able to use the Netflix application on your Apple TV to connect.

Assuming you don’t have a Netflix account already (from time overseas or by registering an account at someone else’s place that has this going already), the Apple TV Netflix account should let you set one up. Just follow your nose, it should be self explanatory (I did mine over a computer but I’ll review the app method more if people are getting stuck). If you have to enter a postcode for your *cough* US Address *cough*, just use the good old 90210.

For those who aren’t just satisfied with getting it up and running but want to find out what else might be possible, Apple have a little marketing blurb that gives you an idea of what else might be out there which you can read here!

That’s it for now. Next on the list… well we’ll see. If there’s a particular device you’d like to get up and running or if you’re still having trouble, leave a comment down below and I’ll see what I can do to help.

Ciao!

The post Netflix and other US restricted content in NZ appeared first on Random Insanity.