Maori Geek

My New Site

Sun, 28 Jun 2026 21:35:54 +1200

Medium has been going down hill fast. I didn’t notice for a while, since I have been heads down coding, but the other day when I needed to share a post with someone, it didnt even load my blog.

I have moved all my blog over to github webpages, using hugo, medium-to-hugo and PaperMod theme.

Writing posts and sharing them is very motivating for me and finally medium broke that enough for me to move my decade old blog for the third time.

I Tried to Write a HTTP Service in Zig and Failed

Mon, 09 Mar 2026 23:54:47 +0000

I really like Zig. I have never wanted to write manually memory-managed code before. I don’t like C because it seems opaque and dangerous and I don’t like Rust for purely aesthetic reasons; but Zig is shiny and new and pretty and I can understand it intuitivly.

I wanted to write a small HTTP service in Zig (0.15.2) that used SQLite as a data store and could scale up. To do this I used the [http.zig](https://github.com/karlseguin/http.zig) and [zig.sqlite](https://github.com/vrischmann/zig-sqlite) libraries which are pretty stable. I spent a couple weeks on this project, but kind of hit a few walls all at once which has lead me to the decision to abadon the spike.

Being On Call for Type 1 Diabetes with Carer

Tue, 15 Jul 2025 23:34:11 +0000

Managing my 5yo son’s Type 1 Diabetes (T1D) at night is a lot like being on call as a software engineer. Both are absolute shite. Waking up to alarms at all hours, trying to fix something to only make it worse, not understanding why anything is happening, alarm fatigue, lost sleep, burnout, and the constant stress you will sleep through an emergency.

It is a bit guff to compare the two. Though, there are a some things that do apply to both, like good tools can reduce the burden.

Sqlite HTTP API: Zig vs Golang

Sat, 01 Mar 2025 03:12:16 +0000

I like shiny new things. Zig is very new, and very shiny, and very fast. I have no experience writing non-garbage collected languages, so I want to test it out and compare Zig with GoLang.


Benchmark
The benchmark is a HTTP API wrapper around SQLite:
`

/write
takes a
JSON
{
“name”
} and write is to the
DB

/read
returns
100
items
`
The SQL:
`
– TABLE
CREATE
TABLE
IF
NOT



Small, Light, Robust Phones for a Type 1 Diabetic Child
Thu, 13 Feb 2025 04:36:16 +0000
Our 4yo T1D son, Sam, needs to carry a phone everywhere he goes to act as his pancreas. This phone needs to:

run Android APS and a few other apps which need Android 11 or above (14 is preferred).
have good bluetooth, Wifi and cell signal. It needs to have constant communication with his Omnipod pump, Dexcom G7 and remote systems like NightScout.
be sturdy, light and small. It is strapped to a very active 4yo.
be easy to use. Clicking the wrong button can be pretty dangerous.
be very cheap. Things break, buy a couple phones as backups.

The phone doesn’t need fast graphics, large battery, big screen or good camera; the things most phones are marketed on. It just needs to work and not break when a 4yo falls off their bike.


Problems with Predicting Blood Glucose with Regression
Thu, 20 Jun 2024 04:25:03 +0000
I use an app called NightGuard to track my Type 1 Diabetic (T1D) son’s blood glucose levels (BGL) and alert me if he needs some food or insulin. I love NightGuard. It is one of the most important applications I use to manage T1D.
One of NightGuard’s best features is its predictive alerts. These alerts use regression models to try predict BGL and notify of future problems. In this post I am going to explore the way NightGuard uses regression and see if I can improve its outcomes.


Using Inkscape to make Individual Puzzle Pieces for Laser Cutting
Tue, 21 May 2024 23:03:14 +0000
My Laser Cutter bed is 300mm x 500mm, and I need to make a shape that is about 900m x 750mm. I could just use squares, but then the joinery gets complicated making sure everything lines up properly.
Using jigsaw pieces might be a fun way to make such large shapes, so here is how to do that.


Install “Lasercut Jigsaw” extension



Create the puzzle you want with Extensions > Render > Lasercut Jigsaw


Design: Blood Glucose Levels
Tue, 20 Feb 2024 02:51:43 +0000
I am building a Blood Glucose Display and just wanted to collect a few of other apps examples together to compare and get ideas:#### Dexcom

Libre

NightGuard

Nightscout

SugarPixel

I don’t 100% like any of them.
There are like two ways in which I use this information:

As a display to “glance” numbers and to alarm me if something is wrong. Like pull and push information.
Using the information to calculate treatments, once something is wrong, or I think something is going wrong I need to work out a remedy.

A lot of them contain graphs and other information that I don’t really need for either purpose.This is just a quick post about this to force my ideas into a box.


Managing a Type 1 Diabetic Preschooler with AndroidAPS
Sun, 18 Feb 2024 09:01:18 +0000
Sam, our 3.5yo T1D, just started preschool. In order to keep Sam safe with his BGL in a good range, reduce our stress and to help the staff at his new preschool we devised a plan that focused on:

making a custom Diabetes Action and Management Plans (called 504 plans in the US)
Automating with AndroidAPS’s SMB (super-micro-bolus) and Remotely controlling AAPS through SMS commands.

This post will go into each briefly to hopefully help anyone else going through similar stress.#### Action and Management Plan


How to implement JavaScript String.length in Golang
Thu, 14 Sep 2023 02:24:06 +0000
I needed to implement JavaScript .length in Golang, the solution was pretty fun so I will tell you how.For some strings Golang’s len function will do, e.g.
`
“Hello World!”
.
length
// JS     12
len
(
“Hello World!”
)

// Golang 12
`
The problems start when you include any unicode characters, e.g.
`
“👍”
.
length
// JS     2
len
(
“👍”
)

// Golang 4
`
This is because JavaScript is implemented in UTF-16 (or UCS-2) and Golang is in UTF-8.


Omnipod Dash + Dexcom G7 + AndroidAPS
Sun, 13 Aug 2023 09:18:31 +0000
I was asked by an Insulet (manufacturers of Omnipod) representative if I would like to share our T1D story with Pharmac (the government organisation that decided which medicines and pharmaceutical products are subsidised) via Diabetes New Zealand. Pharmac recently put out a “Request for Proposals” to fund CGMs and insulin pumps. I love the Omnipod and our Dexcom CGM and wish all New Zealand T1D’s would have access to them.
Below is the letter I wrote.To whomever will listen,


GoLang + SQLite on Fly.io with LiteFS: a Quick Benchmark
Mon, 17 Jul 2023 02:39:37 +0000
I am thinking about using GoLang and SQLite for a project. Looking around I found a presentation by Ben Johnson from Fly.io about LiteFS which is working towards a more scalable and reliable SQLite. This post is just about me getting something working with these technologies and stressing them a little.
Benchmark
I have a simple GoLang app with one route to inserts a row into a SQLite database. I want to test calling this endpoint when:


Moving to Android APS
Tue, 06 Jun 2023 04:46:26 +0000
In this post I am going to discuss moving our Type 1 Diabetic (T1D) 3yo son Sam from CamAPS to AndroidAPS; Why we moved, the logistics of the the change, our current setup, and the results we are seeing.

Sam’s Bedroom door with phone mount
Note: This is not medical advice or a recommendation, it is just our experience.#### Leaving CamAPS
To manage my sons Type 1 Diabetes (T1D) I need data. All the data I can get, and I need it to be accurate and timely. I need to know:


Layered Laser Cut Cardboard Art with Inkscape
Wed, 21 Dec 2022 03:04:25 +0000
I really love art by Martin Tomsky that is created with layers. I wanted to try something similar.
The source image:

Tui by Matt Binns [source]
The goal:

These are examples from previous attempts
Step 1: Preprocess
First we take the source image and use cut out the subject and adjust the colour to be very vibrant.
Note: Copy Subject in the photos app and New From Clipboard in macOS preview are useful here.


Automating my Son’s Pancreas with CamAPS and Dana-i Insulin Pump
Thu, 03 Nov 2022 19:00:35 +0000
This is the third post about my 2yo son’s Type 1 Diabetes (T1D). Previously, I wrote about his initial management coming out of the hospital, then his management with a Continuous Glucose Monitor (CGM). This post is about the last three months we have been using the Dana-i insulin pump with the CamAPS Hybrid-Closed-Loop (HCL) algorithm.
Obligatory: nothing here is medical advice.### Insulin Pump
The pump we chose was the Dana-i pump. I wrote about the reasons we picked that pump here, but basically the Dana-i is small, remotely controllable and can be used with the CamAPS hybrid-closed-loop.


6 Months as a Full Time Pancreas
Tue, 27 Sep 2022 07:50:11 +0000
Managing our Son’s Type 1 Diabetes with Multiple Daily Injections and Dexcom G6 Continuous Glucose Monitor
In a previous post, I wrote about my 18 month old son’s (Sam) Type 1 Diabetes (T1D) diagnosis and the first month when we were learning as much as possible and struggling. That post was received with so much love and positivity, I have decided to follow up with a 6 month recap.
Sam has now had diabetes for a quarter of his life (18–24 months). We have spent these first 6 months learning to be Sam’s pancreas using Multiple Daily Injections (MDI) and the Dexcom G6 Continuous Glucose Monitor (CGM). This post is about what we are doing and what we are focused on. It will hopefully be helpful for people in a similar situation and to serve as a reminder to ourselves how far we have come managing T1D.


What is HbA1c?
Mon, 19 Sep 2022 01:42:52 +0000
And what is NGSP, IFCC, GMI and eAG?
What is HbA1c? Glucose in the bloodstream will sometimes randomly link (glycate) with a red blood cell’s haemoglobin (Hb) creating “Glycated haemoglobin” or HbA1c. The more glucose there is in the bloodstream the more likely this linking will happen. A red blood cell is in circulation for about 120 days, so the amount of HbA1c in the bloodstream can be used to approximate average blood glucose level over that time.


My Awesome Year (+) At Clubhouse
Sat, 10 Sep 2022 00:41:42 +0000
While catching up with some friends in 2021, they nerd sniped me by talking about all the interesting scaling problems they were having at Clubhouse. I quickly found myself in a meeting with the Rohan, Clubhouse’s cofounder, discussing all the other novel challenges that such an ambitious product has. Before I knew it, I was on the infra team at Clubhouse.
I joined Clubhouse in June 2021 and have decided to leave in September 2022. This is a quick post about my work there.While at Clubhouse I mostly worked on:


Add a Dexcom Share Follower with curl
Wed, 03 Aug 2022 07:23:19 +0000
This is just a short set of commands that I used to add a dexcom share follower with curl. I am writing this mostly so I remember if I have to do it again, but also if someone else needs it.
This is based off the commands that listed here.
Note: use _https://share1.dexcom.com_ if you have a US account and _https://shareous1.dexcom.com_ for non-US account.Authenticate
SESSION_ID=$(curl -v -H "Accept: application/json" \   -H "Content-Type: application/json" \   -H "User-Agent: Dexcom Share/3.0.2.11 CFNetwork/672.0.2 Darwin/14.0.0" \   -X POST \   "https://shareous1.dexcom.com/ShareWebServices/Services/General/LoginPublisherAccountByName" -d '{   "accountName":"<username>",   "applicationId":"d8665ade-9673-4e27-9ff6-92db4ce13d13",   "password":"<password>"   }' | tr -d '"')


Which Insulin Pump to Choose?
Fri, 03 Jun 2022 03:04:37 +0000
A quick guide to pumps available in New Zealand
We are tired. Managing our 2 year old son’s (Sam) type 1 diabetes (T1D) with Multiple Daily Injections (MDI) is a lot of work. It’s normal to get up and give food or insulin multiple times a night, and it is necessary to give multiple injections at precise times during meals.
We need a better tool! The best tool available is an insulin pump. MDI is coarse, we give 8–10 giant doses of insulin per day. A pump, on the other hand, is fine grained, it can give hundreds of tiny doses a day.


The Unreasonable Math of Type 1 Diabetes
Thu, 17 Feb 2022 07:59:44 +0000
This is not medical advice, don’t base any treatments on this.
In January 2022, our 18 month old son, Sam, was diagnosed with Type 1 Diabetes (T1D). This was stressful, sad, and scary as we spent 5 days in hospital with him while he recovered from Diabetic Keto Acidosis (DKA). Within an hour of him being diagnosed a wonderful diabetes nurse gave us a literal backpack filled with books and information we needed to learn to keep him alive. We started to read and try to understand what it takes to manage T1D. Immediately the massive cognitive overhead it takes to just survive with this condition hit us.


My Troubles with Lua
Mon, 27 Sep 2021 10:32:50 +0000
I use Redis and Redis uses Lua as a scripting language. Today was my first day using Lua in anger, and I am still angry.My problem is that in Redis I have many sets of keys, e.g. s1 = {1,2,3} s2 = {3,4} and keys 1=a, 2=b, 3=c, 4=d. I want to return all values of all keys in the union of given sets, e.g. f(s1 s2) = a b c d


Game Of Life: CUDA vs Golang
Mon, 16 Aug 2021 09:30:55 +0000
I got a new graphics card because I want to play around with some machine learning using tensorflow and pytorch. Before I jump into all those high level concepts, with layers and layers of abstractions, I want to understand a little bit more about CUDA and how it works. I usually learn by doing, so I decided to do something pretty easy, implement Game Of Life (GOL) using CUDA.
GOL is good use case for CUDA. It is a simple mathematical operation that can be massively parallelised. In this post I get a pretty fast implementation of GOL working in CUDA, and compare it against another implementation in Golang. **TLDR: CUDA wins… by a lot.**CUDA lets you compile code to be run on a graphics card. You write in a C like language two types of functions, one for the CPU (host) and one for graphics card (device). The code starts with a main function executed on the CPU and can call out to device kernel functions declared with __global__.


Get an Email if Someone SSHs into Your Box
Mon, 19 Jul 2021 07:44:03 +0000
There are a couple different types of security; preventative will stop something bad from happening and detective will alert you when something bad does happen.
I have box that is SSH accessible on the internet that I want secure. There are lots of good sources for preventative SSH security (e.g. no passwords, tight config…) but not many for detective security. To be more secure (and decrease my paranoia) I want to be notified when someone SSH’s into that box from the wide internet, and I usually read my email.The first part is being able to send an email from a script. I could use any old SMPT server, but using an authorised one will decrease the chance my email goes straight to spam. I use Gmail’s SMTP server since it is pretty easy to setup.


Make Python’s DynamoDB client faster with this one simple trick
Sun, 11 Jul 2021 22:13:26 +0000
AWS DynamoDB is popular because it is super fast & scalable. However, when using the Python client boto3 to fetch a large number of documents we started to noticed some unexplained slowness. This was super annoying as some of our queries were taking 20s to process BUT the actual dynamo query returned in only 6s. So we wanted to find out where that 14s was going.Using dynamodb-local I put 250,000 basic items:
with table.batch_writer() as batch:   for i in range(250000):   batch.put_item(Item={   'id': i,   'updated_at': i,   'name': "some data",   'status': "some other stuff"   })


Python Decorator to Memoize Instance Methods
Wed, 30 Jun 2021 09:24:14 +0000
For the life of me I could not find a python decorator that memoizes methods on an instance. Something that replaces the pattern:
class A:   def x(self):   if self._x is not None:   return self._x   self._x = something_difficult()   return self._x
Sure there is [functools](https://docs.python.org/3.9/library/functools.html) [] (https://docs.python.org/3.9/library/functools.html)`[@cache](https://docs.python.org/3.9/library/functools.html)` [and] (https://docs.python.org/3.9/library/functools.html)`[@cached_property](https://docs.python.org/3.9/library/functools.html)` but these use a global lru_cache for everything. I want the cache to live and die with the instance.
I am still pretty new to python, but slowly getting the hang of it. That being said I came up with this:
`class memoize:

def init(self, func):

self.func = func


Calling Golang functions from Python with Bazel
Sun, 20 Jun 2021 09:01:42 +0000
So I have been doing a lot of Python and Django lately, but I miss Go. As an experiment I wanted to see how difficult it would be to directly call Golang functions from Python using my favourite build tool Bazel.
I found this post, which explains how to build and connect the two manually:
Python and Go : Part II - Extending Python With Go
Great now I am like 90% there!


5 Awesome Years at Coinbase
Sat, 03 Apr 2021 04:51:27 +0000
I joined Coinbase March 28th 2016; my last day is April 2nd 2021. This is a quick write up about my experience on joining and working in the infrastructure team for 5 years at Coinbase, growing from 100 to over 1000 employees, from 3.5 million users to more than 40 million, and from trading $3 billion total to now often trading that in a single day.

Coinbase in March 2016


Every Job I Have Had
Tue, 16 Mar 2021 18:58:51 +0000

Everyone has a different path to where they are today. At inflection points in my career I often think back to the jobs that have got me here. I just wanted to list all my paid jobs to reflect on that journey.Candy Mixer at Four Square (1995–96): A family friend owned a dairy (corner store for non-kiwis) and they needed someone to make the pre-packed $1–2 mixed candy bags. Paid $5 an hour and free candy.


How “go build” Works
Fri, 11 Sep 2020 17:17:24 +0000
How does go build compile the simplest Golang program? This post is here to answer that question.
The simplest go program (I can think of) is main.go:
package main``func main() {}
If we run go build main.go it outputs an executable main that is 1.1Mb and does nothing. What did go build do to do create such a useful binary?
go build has some args that are useful for seeing how it builds:


Learning to play Asteroids in Golang with NEAT
Sat, 29 Aug 2020 23:52:37 +0000
I am a huge fan of CodeBullet, an educational YouTuber that makes game playing AI bots. That is because computer science education more than just conveying complex ideas, it is also getting people excited using those ideas to solve interesting problems, something CodeBullet does very well.
Two years ago CodeBullet showed a bot learning to play Asteroids. I though I wonder if I can do that… in Golang. The first thing was build a desktop app in Golang, then it was to build an asteroids game. Finally, this post is about how I used the goNEAT implementation of NEAT to build an AI that can play my asteroids game.


Making Asteroids Game with Golang, Lorca/Webview and WASM
Thu, 20 Aug 2020 19:54:36 +0000
In my past few posts [1][2] I have written about building a desktop application using Golang with Lorca/Webview to run a WebAssembly (WASM) binary. Now, I want to actually try use these technologies in anger and produce a distributable desktop application.
I have chosen to make the game Asteroids (code here). It is reasonably complicated, fun and lets me play and learn more about Golangs [syscall/js](https://golang.org/pkg/syscall/js/) package and algorithms like Separating Axis Theorem (SAT) for collision detection. The goal is using Golang to build a single binary that can be downloaded onto different platforms (macOS, windows, linux) to play a fun game.


GoLang Desktop App with webview/Lorca, WASM and Bazel
Tue, 11 Aug 2020 06:40:07 +0000
On my quest towards building a GoLang Desktop application I found some useful frameworks, Lorca and Webview (which I wrote about in my previous post). These frameworks create a window which GoLang can inject HTML, CSS, and JavaScript to build the UI.
But I don’t want to write JavaScript(!) and deal with all the complexities that comes with it like npm, webpack, typescript… Fortunately, I can just compile GoLang to WebAssembly (WASM) and use that in place of JavaScript. WASM is a binary format that can be executed natively in most modern browsers. My previous post showed how to build a WASM web-app with the Bazel.


Golang Desktop App: Webview vs. Lorca vs. Electron
Thu, 06 Aug 2020 03:42:51 +0000
I want to build a local desktop Golang app, there are a few ways to do this:

Electron: bundled Node.js and the Chromium browser to create a packaged local web-app. Usable with Golang frameworks like go-app or go-astilectron.
Lorca: using the locally installed Chrome driving it using its dev-tools communication protocol.
Webview: create a native window with webview and render the app inside it using CGo bindings.

I have already written about building a simple electron app, so this post will go into how to build an app using Lorca and Webview, and then compare the three different options.


Building an Electron App with Bazel
Fri, 31 Jul 2020 06:01:46 +0000
I want to build a desktop app with:

Electron.js: A framework to build a desktop application using a Node.js and the Chromium browser
Bazel: a build system to quickly build, test and run applications

They seem like they might go well together, so let’s see. Note: we will be focusing on macOS only for simplicity.
All code is located at https://github.com/grahamjenson/bazel-electron
The Electron Application
An electron app (in macOS) is a folder with the electron binaries (Node.js Chromium, and libraries downloaded from https://github.com/electron/electron/releases) and three application files located at:
electron/Electron.app/Contents/Resources/app/   ├── package.json   ├── main.js   └── index.html


Longest Single Letter .com Domain Name
Wed, 22 Jul 2020 00:19:46 +0000

I was looking for a domain name, got bored at not finding any I liked, so started looking for other, more specific domain names. I tried aaaaaa.com but this was taken, then aaaaaaa.com , then aaaaaaaa.com and so on. I was surprised by how many of these were registered.
So I began to wonder how may a’s it would take before there was a domain name that I could register. Then I expanded that to wonder about b and c …


A Web App Using Bazel Golang WASM and Proto
Wed, 01 Apr 2020 01:03:40 +0000
Gary Bernhardt at PyCon 2014 talked about “The Birth and Death of Javascript”. He predicted, among other things, a giant exclusion zone around San Francisco, a world altering event in 2020, and that WASM would eventually take over the web. Let’s ignore the prior two prescient predictions and focus on the third, WASM taking over.
So, I want to build a simple web app by joining a bunch of new and fun technologies together:


Bazel: Why people love/hate it
Mon, 05 Aug 2019 13:04:44 +0000
In Bazel you put files into Rules and get files out, e.g:
pkg_tar(   name = "package",   extension = "tar.gz",   srcs = [:file1, :file2]   )
The pkg_tar rule takes :file1 and :file2 and spits out the tarball :package.tar.gz, which you can pass to another rule as input. So you input files into rules that output files into other rules, that output files into other rules… until eventually you get the file you want. This is a graph that can look like:


Git Log as JSON in Go
Tue, 23 Jul 2019 20:02:21 +0000
For reasons I wanted git log to be parsed in Go and it looks like the easiest way to do that is to output it as JSON and parse into Go structs.
First we need to define the format for the JSON (borrowed from here):
var GITFORMAT string = –pretty=format:{

“commit”: “%H”,

“parent”: “%P”,

“refs”: “%D”,

“subject”: “%s”,

“author”: { “name”: “%aN”, “email”: “%aE”, “date”: “%ad” },

“commiter”: { “name”: “%cN”, “email”: “%cE”, “date”: “%cd” }

},``


How to Digest a Docker Image
Thu, 11 Jul 2019 21:11:07 +0000
How to calculate and manipulate Docker SHA Digests


CloudFormation S3File and S3Zip Custom Resources
Fri, 21 Jun 2019 08:26:38 +0000
LearnLog
8:00am — CleanUp
Last time I was able to get the basics of custom resources in Fenrir working. Today I want to finish:

Uploading a S3File with custom resource.
Create Custom Resource S3ZipFile which will extract the ZIP file.
Also clean up some of the code.
Test out multi-account deploys with custom resources.

Using the Bifrost standard to build deployers provides a framework for building deployers that can work across AWS accounts by simply creating an assumable role in an AWS account. This make onboarding new accounts super easy. CloudFormation custom resources complicates this a bit because I want to keep this simple onboarding, so I don’t want to deploy a lambda.


CloudFormation Custom Resources in Fenrir
Tue, 18 Jun 2019 17:15:46 +0000
Learnlog
9:50am — Fenrir CloudFormation
I am taking today off from learning Bazel and instead I am looking at how to build a CloudFormation static site deployer for Fenrir.
Fenrir is a AWS SAM deployer that is basically sam deploy but in a Step Function. I want Fenrir to be able to deploy “full-stack” applications, including front end resources which can sit in S3 as a static assets. This would make it much easier for Coinbase engineering to work with serverless, as currently they have to deploy front end code separately from the serverless API’s that power them.


Bazel Docker Tests
Mon, 17 Jun 2019 20:33:05 +0000
Learnlog
7:30am — What to do today
In my previous learnlog I built a tree of docker images able to be changed and rebuilt.
What I want to try do today is write a tests on the built containers so that if they don’t meet some constraints the build will fail and their dependents won’t be built.
Let’s boot up the previous days project and start reading.
8:30am — Yak Shaving
Yak shaved by cleaning up some of the images by copying the build tree of the valid DockerHub images.


Bazel Docker Dependency Tree: LearnLog
Sat, 15 Jun 2019 07:29:09 +0000
Learnlog
9:45 am — What I want
What I want to do is use Google’s build tool Bazel to build a tree of Dockerfiles that are FROM each other and use Bazels dependency engine to automatically rebuild docker containers if a container they depend on changes. This is currently a manual job at many companies, and as they grow this becomes more tedious and in need of automating.
I already have Bazel installed from a previous failed spike using https://github.com/bazelbuild/rules_docker which are a bit overcomplicated and difficult to use for someone who is just learning (like me).


Introducing Fenrir: How Coinbase is Scaling Serverless Applications
Thu, 11 Apr 2019 13:08:59 +0000
Today we’re open sourcing Fenrir, an open source AWS SAM deployer. Learn how we’re using it to scale serverless applications at Coinbase.


YAML Custom Tags in Ruby
Wed, 06 Feb 2019 11:30:10 +0000

I wanted to parse an AWS SAM template file using Ruby. This format uses [intrinsic](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html) functions which can be YAML custom tags, e.g. [!Ref](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-ref.html) to reference a parameter or resource.
How do I parse a YAML custom tag in ruby? There are two options YAML.add_domain_type or YAML.add_tag.
Domain types are **** easier:
require "yaml"``YAML.add_domain_type("", "Ref") do |type, value|   value.upcase   end``out = YAML.safe_load("test: !Ref value")``puts out # {“test”=>”VALUE”}   puts out.to_yaml # test: VALUE


Hello SAM: AWS Golang Quickstart
Mon, 24 Dec 2018 20:44:25 +0000

Pre-Req: Must have Docker for Mac installed
I wanted to play with AWS SAM to understand how it works and compare to other tech like serverless.
Install and Test SAM
Use brew to install the aws-sam-cli:
``brew upgrade

brew update

brew tap aws/tap

brew install aws-sam-cli

sam –version

SAM CLI, version 0.10.0``

Now init a go SAM project:
cd $GOPATH/src/github.com/grahamjenson/   sam init --runtime go --name hello-sam    cd hello-sam
This creates a folder with:
hello-sam/   ├── Makefile                    <-- Make to automate build   ├── README.md                   <-- This instructions file   ├── hello-world                 <-- Source code for a lambda   │   ├── main.go                 <-- Lambda function code   │   └── main_test.go            <-- Unit tests   └── template.yaml


AWS Step Functions, State Machines, Bifrost, and Building Deployers
Tue, 20 Nov 2018 11:01:01 +0000
AWS Step Functions are hosted state-machines defined according to the Amazon States Language. To execute a Step function you send it JSON data which is given to an initial state to process then pass the output to another state. States are processed until a success or failure state is reached.
How a state processes its input and selects the next state depends on its Type. For example, a Task state can use a Lambda function to process the input, and a Choice state can select which state to go to next based on its input.


AWS API to get EC2 Instance Prices
Sat, 03 Nov 2018 21:46:49 +0000

AWS provides an API for everything! This includes the Pricing API to find out how much you can spend via their other APIs. This API can be difficult to use to answer questions like:

How much does an EC2 instance cost?

The difficulty comes because the API has a single endpoint [GetProducts](https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_pricing_GetProducts.html) that returns all prices for every service and product. That is a ton of data that requires lots of filtering to get what you want. You can download all the EC2 price data at once from a file Amazon hosts here. This file is 510Mb!


Assuming Roles in AWS with Go
Wed, 08 Aug 2018 14:46:19 +0000
So your organization has many AWS accounts, but you have services (like monitoring, deploying, security) that require access to resources across many/all of those accounts.
There are a few options for these services:

Deploy the service in each account: to add an account you need to recreate the entire service with all its resources, and then maintain all of that infrastructure in perpetuity.
Deploy the service in one account with access keys to other accounts: each account requires a user with a policy and access key that is given to the service. This can quickly get out of hand as the number of keys explode, both in maintaining the services access to them and security concerns around rolling them.
Deploy the service in one account that can assume roles into other accounts: the service requires a user, instance profile, or role that is trusted by roles in the other accounts. The service only needs to know the name of the role and the account ID to work.

The latter option has the least surface area to secure, requires the least amount of maintenance, and is the easiest to scale with the number of accounts. This post briefly goes over how to manage the assumed roles in a service written in Go.


GoLang- Raise Error if Unknown Field in JSON (with exceptions)
Tue, 24 Jul 2018 14:51:44 +0000
I want an Error to be raised during JSON unmarshalling if an undetected field is found. This is useful if you are trying to be extra careful, for example double checking the client has not mispeled any inputs. However I want to exclude some fields from this that might be used elsewhere.Model is the struct that we are unmarshalling. We must create a new type so that the unmarshalling function is not recursive (as explained below):
type XModel Model


Open Sourcing Coinbase’s Secure Deployment Pipeline
Tue, 22 May 2018 12:52:13 +0000
In 2017, Puppet and DORA (DevOps research and assessment) published their annual State of DevOps Report that collates more that six years of survey data about the cultural and technical impacts of DevOps. Analyzing over 27,000 responses they found that high performing engineering organizations have:

46x more frequent code deployments (on demand deployments)
440x faster lead time from commit to deploy (less than one hour)
96x faster mean time to recover from downtime (less than one hour)
5x lower change failure rate (0%-15%)

That is, high performing teams:


Selecting an Editor
Tue, 15 May 2018 12:09:09 +0000
When writing blog posts I have a few requirements:

Offline: I travel and commute which means that I have no internet frequently which is a great time to write distraction free.
Easy Publishing: I moved all my blogging to medium because I spent far too much time doing things on the blog that were not writing.
Syncing: I like to write on my laptop, but sometimes it is more convenient on my phone, like when I am waiting for a train.
Formatting: I like Markdown, Medium doesn’t use markdown. This is my biggest issue with Medium. Typically I will write a post in Markdown then manually convert it to Medium which is super annoying.Previously I have used Evernote which is great offline and syncing, but it has a different format to both medium and markdown making which makes it difficult to publish with.

I have also just used a plain text editor Sublime Text with GFM markdown plugin, then wrote content in Notes in iOS and moved it manually to my laptop. This is a lot of manual syncing, and once completed I still have to convert it to mediums formatting.


Slowly Baking Bread with AWS Step Functions
Thu, 12 Apr 2018 18:28:43 +0000

AWS Bread, recipe below
As developers we are pretty good at writing fast code because we put a lot of emphasis on that skill (especially in job interviews). Where we have a little more trouble is writing slow code, processes that don’t take milliseconds or seconds to run (e.g. a web request), but take minutes, hours, or days (e.g. data backup and migration) to complete.
As a process takes longer to complete, some qualities become much more important:


Gamifying Security Culture with PwnBot
Thu, 28 Dec 2017 20:00:34 +0000
Building security into your company culture is necessary but challenging. The first line of defense against most attacks are aware and vigilant people with an attitude of “see something, say something”.

attr. Tony Webster
Training employees on what is abnormal behavior and who to talk to when there is a potential problem can save your company from a lot of pain. For example, alerting security when a co-worker makes an unusual request because one of their accounts has been compromised by a hacker.


GitHub’s “Squash and Merge” doesn’t Squash and doesn’t Merge! Trade-offs with Merging
Tue, 26 Dec 2017 23:23:00 +0000
At Coinbase we use a GitHub-flow-ish workflow to collaborate on code and develop features:

Create a branch
Create/Edit/Delete/Rename files on branch
Create a Pull Request (PR) merging the branch into master
Get code reviewed and make requested changes
Merge the PR

However, we are still having discussions around the final step. How should we merge pull requests?
The options are:

Merge the pull request, git merge
Manually squash the branch with git rebase or git reset , force push, then merge
Use the “Squash and Merge” function in GitHub, basically git --squash merge branch

The core differences between these methods is how much friction it is for developers to use, and what historical information is left behind.


You need more than one AWS account: AWS bastions and assume-role
Tue, 17 Oct 2017 21:21:52 +0000

Bastion at Castel Sant’Angelo
You need more than one AWS account. This is to isolate production resources, manage limits (especially API rate limiting), handle costs, simplify compliance and security concerns, and restrict user access.
However, managing multiple AWS accounts can be difficult. To help with this AWS has built many useful features like organizations, consolidated billing, VPC peering, and descriptive IAM policies. Using these features effectively to structure and connect AWS accounts, a variety of patterns have been developed. One such pattern used by Coinbase is the AWS Bastion account.


Proof of Human
Wed, 13 Sep 2017 14:01:24 +0000

I am pretty sure that I am human. I am also pretty sure that my wife, my friends, my colleagues, and other people I have met in real life are also human.
I am less sure about disembodied entities on the internet, like you. If you are human, then welcome to this post and I would like to hear what you think about it in the comments. Before you post anything though, I need you to prove to me that you are human because I don’t want to read comments posted by evil-spammy-bots.


Turtles All The Way Down: Building Simple and Powerful Ruby DSLs
Fri, 17 Feb 2017 23:34:34 +0000
A Domain Specific Language (DSL) is a specialized way to clearly describe a problem domain. Ruby is a great language for creating DSLs because it lets developers decide how the language looks and is used. For example:
RSpec for writing tests:
FactoryGirl for mocking objects:
GeoEngineer for defining cloud resources:
This post will briefly describe how to use Ruby to create DSLs like the examples above.
Building Blocks
Ruby DSLs typically use functions that take a block given as an argument to build an instance of an domain object:


Treating Infrastructure Like Code
Tue, 10 Jan 2017 14:16:18 +0000

Zion National Park’s amazing geology
The goal of the infrastructure team at Coinbase is to provide self-service tooling to our engineers to empower them to rapidly develop, monitor, and optimize services with low risk. With this mission in mind, we are currently in the process of building a workflow for creating and managing our codified infrastructure resources that looks like:

Pull Request: an engineer submits a pull request to a repository with a new codified resource they want.
Validation: the new resource is automatically validated and follows our company standards for naming, tagging, and security.
Plan and Review: a plan describing the actions needed to be taken to apply a change is presented alongside the code change to be reviewed by an infrastructure team member.
Merge then Apply: if the plan is good, then the pull request can be merged and automatically applied to the cloud.

This workflow manages our codified infrastructure the same way we manage our code with GitHub flow; i.e. open a pull request, ensure the change is valid with tests, merge the change into the master branch, then apply the changes to the necessary environments. The main idea of this workflow is to improve collaboration between the infrastructure and engineering teams. This will also speed up the development and deployment of resources, and make sure we deliver what is actually needed.


Podcasts
Sat, 26 Nov 2016 16:56:58 +0000
While running or driving I enjoy listening to podcasts as they are a great way to passively learn and keep up with various technologies and trends, as well as be entertained. This post is a list of the podcasts I enjoy (or have enjoyed) listening to.
Programming
DevChat podcasts
Ruby Rogues: Ruby and programming focused panel podcast, I find the earlier episodes with the original crew were the the best and worth going back for a listen.


Keystone Metrics in DevOps: The 30 Day Project @ Coinbase
Fri, 07 Oct 2016 21:16:06 +0000
Alcoa Keystone
Charles Duhigg in his book Power of Habit discussed how Paul O’Neil, the CEO of Alcoa (Aluminum Company of America), was able to increase his company’s value by 27 billion by focusing on a goal unrelated the main company objectives, no workplace injuries. This is a keystone metric (habit), a broad goal for the entire company, not directly related to the company’s main goal but impactful and actionable. O’Neil chose safety as the single most important aspect because:


Self-Service DevOps @ Coinbase
Thu, 29 Sep 2016 18:28:59 +0000

A bad way to implement a currency exchange would be to have the customer pass their order to an employee who then manually checks the customer balances, decides whether the order is valid, then writes down the order in a big order book (a la the old New York stock exchange). This method would be slow, prone to human error, not scalable, and more customers would require more employees. A better way would be to give the tools to the customer (with appropriate validations and security) to create the orders themselves, making the exchange more efficient and less expensive to run.


Why are Ultrasound Machines So Expensive?
Mon, 22 Feb 2016 23:00:06 +0000

Name a technology that is more useful, more educational, more interesting, and more overpriced than a ultrasound machine. You can look inside of living things without the need for a powerful magnets or radioactivity and it is basically made from a speaker and microphone outputting to a screen.
Why doesn’t every high school biology class room have an ultrasound to show how muscles work, and hearts beat? Why don’t doctors have them immediately handy like a stethoscope or thermometer? Why can I not get one just because I am interested in how my injuries are healing? Probably because “a £20,000 [$30,000USD] scanner is generally classed as low cost.”


9 Things that Recommender Systems Should Do
Mon, 21 Dec 2015 21:36:52 +0000
Strands published a great vision paper about The Big promise of recommender systems. In this post I break down the paper into a few useful requirements; Recommender systems should:

be easy to integrate and easy to remove. Vendor lock-in is a negative so having a minimal impact on the clients system is a big plus
complete with internal marketing teams. Generated recommendations will compete for space and resources with internal marketing departments as their core goals are identical. So reuse the marketing metrics and methods in recommender systems to measure and drive real business value
first collect enough data to avoid the cold start problem. Don’t release the recommender systems functions till there is enough data to provide good recommendations
scale to the businesses needs, number of users and number of items
be a hybrid of approach as these create a robust solution that solves many problems of exclusive algorithms
be a balance between algorithms and UX. Focusing entirely on algorithms or UX will not create value, it must be a mixture of both
use implicit instead of explicit feedback. Explicit ratings or reviews can be manipulated, however implicit (e.g. if they actually bought or watched the item) are more difficult to manipulate and will provide better data
differentiate their products as recommender systems have become commoditized. If a system if difficult to evaluate and looks similar to other systems it will fail
have contextual awareness of where the recommendations are being shown. For example, recommendations on mobile are different to web and different to email

Check out my Good Enough Recommender (GER) to see how it can be used to implement the requirements in this list and check out the List of Recommender Systems and see how they implement these requirements


What We Learned Living in AirBnB’s for 6 months
Fri, 13 Nov 2015 11:00:29 +0000

Over the last 6 months my wife and I have been working and travelling around Europe using Brighton U.K. as a base. Before we left we looked at a few different ways of getting a roof over our head while travelling, our ideal solution would be low hassle, low responsibility, low liability, and easy to set up.
The original idea was to rent an apartment in Brighton but this can be very expensive, difficult to find and organise from overseas, and requires signing contracts, putting up bonds and other legal arrangements. In addition to these problems, the local law also puts a stupid paradox in the way of any traveller renting an apartment; To rent a British apartment you need British bank account and to get a British bank account you need a British address. From the HSBC site


New Version of HapiGER Recommender System
Tue, 13 Oct 2015 10:24:52 +0000

HapiGER, the Happy (Good Enough) Recommender System, has a new version!
This version:

is Simpler: with a smaller, better defined API
is More Configurable: can better define how to generate recommendations giving more control over quality and speed
has Added Functionality: can now generate similar things recommendations to find items that are similar to other items

Links:

HapiGER
HapiGER on npm
HapiGER source
GER Source
List of Recommender Systems



A Dodo Developer Doesn’t Want To Learn New Things
Thu, 08 Oct 2015 13:35:22 +0000
Why should you learn new things?
Because you can’t predict the future, and knowing is better than not knowing.
What you learn doesn’t need to be applicable, you don’t have to put it on your CV, it doesn’t need to be measured and quantified, you don’t need to tell anyone.
Learn anything; learn a toy, a gimmick, something too new or too old, something that is popular or something that is niche. It doesn’t matter if it is not supported, not serious or not professional. Learning is an exercise, so the worst that can happen is you get faster at learning the next time.


3 Optimisations that Improve Recommendations
Thu, 20 Aug 2015 15:01:38 +0000
In this post I will briefly describe three optimisations that GER (Good Enough Recommendations) uses to improve recommendations. These optimisations could be useful for other recommender systems as they have increased the quality of GER’s recommendations.
Separated Data
The two fundamental steps in recommender systems like GER are:

find a neighbourhood of similar people
recommend things that those people like

The similarity between people can vary widely from one set of items to another. For example, my friend and I have very similar tastes in action movies, but we don’t like the same drama movies. If all the data was combined, our similarity would be very weak. However, if we separated the data based on movie genre we would be very similar in the “action” genre, and dissimilar in the “drama” genre, now I can get some good “action movie” recommendations.


CoreOS: Fleet Service to Manage /etc/hosts
Sat, 11 Jul 2015 22:02:23 +0000
I want to run an application on my CoreOS clusters that uses hostnames to communicate between machines. This is a problem, because out of the box CoreOS machines cannot resolve hostnames of other machines in the cluster. So, I wrote a small fleet service that manages the /etc/hosts files on all the machines so they can correctly resolve each others hostnames. In this post I will briefly describe that service.
The Hosts Service
The hosts.service:
# hosts.service   [Unit]   Description=Hosts Manager   After=etcd2.service``[Service]   EnvironmentFile=/etc/environment   Restart=always``ExecStartPre=-/usr/bin/etcdctl mkdir /hosts``ExecStart=/bin/sh -c 'while true; do etcdctl watch --recursive /hosts; \   sleep 1;\   echo "127.0.0.1 localhost" > /etc/hosts; \   for i in $(etcdctl ls /hosts); do \   echo $(etcdctl get $i) $(echo $i | cut -c 8-); \   done >> /etc/hosts; \   done'``ExecStartPost=/usr/bin/etcdctl set /hosts/%H $PRIVATEIP   ExecStopPost=/usr/bin/etcdctl rm /hosts/%H``[X-Fleet]   Global=true


Testing Microservices with pmux and TravisCI
Tue, 09 Jun 2015 14:44:00 +0000
Separating an application into many small independently developed and deployed microservices that communicate over a thin layer (like http) has many benefits (see Fowler’s article). However, one of the main drawbacks of this architecture is the difficulty automating end-to-end tests for the application.
Newman’s Building Microservices asks two questions when end-to-end testing a microservices application:

Which versions of the services should we test?
Where are the tests written, to not to duplicate the effort for each service?

His solution is to have an external end-to-end test suite that can be run against many configurations of microservice versions. In this post, I present an implementation of Newman’s end-to-end microservices testing solution that uses the tool pmux and the continuous integration service TravisCI.


The Dysfunctional Fear of Code
Wed, 03 Jun 2015 13:59:00 +0000


I want you to think how deeply dysfunctional it is for you to be afraid of what you created — Uncle Bob Martin Architecture the Lost Years

When I sit down to some code that I haven’t touched in a while, I hesitate. This is weird; I wrote it, I created it, so why am I afraid of changing it? Because, the code works now and if I change something it might break.


Embeeded Javascript on the Tessel; Building a Modular Security Camera
Wed, 29 Apr 2015 12:22:00 +0000
I previously wrote an introduction to the Tessel here where I described a program that blinked out tweets in morse code. For my second project with the Tessel I wanted to use some of the modules that came with it to create a basic security device/camera to take pictures and alert me when something bad is happening.
Modules
The Tessel has 4 different ports to plug modules into, each labeled A, B, C and D. The four modules I decided to use for this project were:


Embedded Javascript on the Tessel; Twitter to Morse Code
Sat, 21 Mar 2015 20:15:00 +0000
I borrowed a Tessel from a friend @leighghunt to play with some embedded node.js. My first project is to listen to a twitter stream and blink it out as Morse code on the Tessel’s LEDs.
Getting Started
npm install -g tessel
Initialising the project:
npm init
I like CoffeeScript, so to test the Tessel out I adapted the demo code from here to a file called twitter_morse_code.coffee:
tessel = require('tessel')``led1 = tessel.led[0].output(1)   led2 = tessel.led[1].output(0)``setInterval( ->   console.log("I'm blinking! (Press CTRL + C to stop)")   led1.toggle()   led2.toggle()   , 100)


List of Recommender Systems
Mon, 16 Mar 2015 07:52:00 +0000
NOTE: I am also maintaining this post on github. This way I can accept pull requests for changes and additions :)
Recommender systems (or recommendation engines) are useful and interesting pieces of software. I wanted to compare other recommender systems to mine (HapiGER) but couldn’t find a decent list of them, so I decided to create one. In this post I will list the recommender systems that I have come across with links and some basic information about them. I intend on keeping this list up-to-date, so comment below if I am missing one or tweet me @grahamjenson.


Boot 2 Docker: How to set up Postgres, Elasticsearch and Redis on Mac OS X
Sat, 14 Feb 2015 06:27:00 +0000
In my previous post about using Vagrant to run Docker containers, I recommended not to use Boot2Docker. Since then my issues with Boot2Docker have been resolved and now it is my preferred way to use Docker in OSX.
Here is a quick tutorial on how to set up Postgres, Elasticsearch, and Redis as Docker containers using Boot2Docker on Mac OS X.
Boot2Docker for OSX
If you are using OSX and want to use Docker, then Boot2Docker is the recommended tool. It works by using a virtual machine to host Docker and letting the OSX command line ‘remotely’ call it with the docker command.


HapiGER: Recommendations Made Easy
Sat, 07 Feb 2015 01:01:00 +0000
I am proud to announce the beta version of HapiGER an open-source, easy to use, easy to integrate recommendations engine. It is built using the Good Enough Recommendations (GER) engine and the Hapi.js framework.
In this post I will describe how you can use HapiGER to generate recommendations for your users.
Install HapiGER
Install with npm
npm install -g hapiger
Start HapiGER
By default it will start with an in-memory event store (events are not persisted)
hapiger


accepts_nested_attributes_for is Creating New Records; Gotcha!
Thu, 29 Jan 2015 00:08:00 +0000
accepts_nested_attributes_for is a really powerful method in Rails because it allows a model to alter related models through itself. However, it has a pretty big gotcha.
An example using accepts_nested_attributes_for is where a user model which belongs_to an alias model, i.e.
#user.rb   class User < ActiveRecord::Base   end``#alias.rb   class Alias < ActiveRecord::Base   belongs_to :user   accepts_nested_attributes_for :user   end
This allows the Alias model to change the user by passing a hash key user_attributes i.e.
`Alias.first.user.name


Scalable Architecture DR CoN: Docker, Registrator, Consul, Consul Template and Nginx
Wed, 21 Jan 2015 06:53:00 +0000
Docker is great fun when you start building things by plugging useful containers together. Recently I have been playing with Consul and trying to plug things together to make a truly horizontally scalable web application architecture. Consul is a Service Discovery and Configuration application, made by HashiCorp the people who brought us Vagrant.
Previously I experimented using Consul by using SRV records (described here) to create a scalable architecture, but I found this approach a little complicated, and I am all about simple. Then I found Consul Template which links to Consul to update configurations and restart application when services come up or go down.


Big-Data: Quantity has a Quality All Its Own
Tue, 20 Jan 2015 03:02:00 +0000
I am not sure where the boundary between Data and Big Data exists, but I must be getting close. As the amount of data I deal with grows, my tools and processes have had to significantly adapt to many new challenges. This caused me to start to think about the qualities of working with Big Data as compared to physical laws.
In this post I will explore the challenges of working with Big Data using some analogies with scale, malleability and gravity. As with all analogies, these are not exact and just exploring a different way of thinking about a problem


Yeah, Nah: Movie Recommender Service
Mon, 15 Dec 2014 07:17:00 +0000

Yeah, Nah: New Zealand slang for yes, or possibly no e.g.

Yeah, Nah is a movie recommendation application built with the Good Enough Recommendation engine (GER), using themoviedb.org’s API for movie information, Hapi.js as its web framework, and Angular.js for front end code.
Source here


GER’s Anatomy: How to Generate Good Enough Recommendations
Wed, 03 Dec 2014 20:20:00 +0000
GER (Good Enough Recommendations) is a recommendations engine that could directly add value and increase user engagement for many existing applications. GER is an open source npm module that you could download and start using right now. However, you probably want to know how GER works and how to use it to get good recommendations out of it.
In this post I describe GER’s core model, its practical features and its limitations to help you use GER to get good enough recommendations.


A Search Box is a Single Question Survey; “What Do You Want?”
Mon, 10 Nov 2014 18:56:00 +0000
Search boxes are useful so they are everywhere! Yet they are treated with disdain, as secondary class elements, hidden away in the corners and pushed to the side of other ‘more important’ content. Only put there as a last resort for frustrated users who cannot navigate to their destination. As if to try and make users feel like they failed when they have to use the search box.
If you treat it your search box this badly it will never provide you its full potential value. The search box gives your users a way to tell you, in their own words, what they want and what they expect you to have. You do not need to guess these things, just read what they typed no further analysis required.


Testing Javascript with Mocha, Chai, and Sinon
Thu, 30 Oct 2014 06:54:00 +0000
These are some of the tools I use to test my Node.js code:

Mocha is a testing framework for describing and running tests
Chai is an assertion library
Sinon is a mocking and stubbing library

In this post I will give a brief introduction to each of these, with some basic examples and tips.
Mocha
Mocha.js is a test running framework. Install Mocha with npm install -g mocha. Run mocha to execute all the javascript test files in the test directory.


Docker Web Services with Consul
Mon, 29 Sep 2014 03:26:00 +0000
Do you need a solution to scale your web services both vertically and horizontally, with load balancing and health checking? Consul with Docker and Nginx can help!
In this post I will describe how to use Consul (a service registry) and Nginx (with srv-router) running in Docker containers to load balance across multiple services.
Note: For OSX users boot2docker is required
Consul
Consul is a service registry that uses the DNS protocol to return a list of healthy services, in a random order (for load balancing).


Replacing an Attribute with a One-to-Many Relationship in Rails
Wed, 24 Sep 2014 01:43:00 +0000
A common change to make to a Rails application is to extract an attribute from a model into a one-to-many relationship. This change can be made without causing a large amount of downtime, even if there a significant amount of records needing to be changed.
In this post, I will describe how to change a model to replace an attribute with a one-to-many relationship while minimising downtime and emphasising continuous deployment.


The Smallest Docker Web Service That Could
Mon, 08 Sep 2014 06:57:00 +0000
Docker is a great tool to test out new application architectures. To make sure that my architectures routed the right calls to the right places, I needed to have a simple containerised web service that just logged when it was called.
In this post, I am going to describe how to create a simple web service with Docker and Python, it is even small enough to fit in a tweet. This post will be brief.


Streaming directly into Postgres with Hapi.js and pg-copy-stream
Thu, 28 Aug 2014 07:07:00 +0000
When implementing the Good Enough Recommendations (GER) engine, a core requirement was to let users insert large amounts of data quickly in order to bootstrap the recommendations engine. Additionally, this bootstrapping should be available over HTTP, as this will become the primary channel for interaction with GER.
PostGres (which GER uses) has the COPY command that is “optimised for loading large numbers of rows” in various formats, and npm has the package pg-copy-streams that pass Node.js streams to COPY. This would work well with the Hapi.js web application framework which can turn an uploaded file into a Node.js stream without having to hold the entire file in memory or create a temporary file on disk.


Postgres Upsert (Update or Insert) in GER using Knex.js
Mon, 04 Aug 2014 10:40:00 +0000
While developing the Good Enough Recommendations (GER) engine, I needed to Upsert a record in Postgres. Upsert is a function that updates a record if it exists, or inserts the record if it doesn’t. However, Postgres doesn’t come with upsert out-of-the-box, so I had to find out how best to implement it.
In this post, I will describe two methods to upsert records in Postgres, multi-query and single-statement. Then I will describe how I compared them to select a method for GER to use.


Good Enough Recommendations with GER
Fri, 25 Jul 2014 09:01:00 +0000
Recommendation engines could be beneficial for many applications as they can directly add value and lead to greater engagement for users. However, there is significant overhead in implementing a custom solution and many off-the-shelf engines have overcomplicated APIs, or try to be infinitely scalable which is not needed by most applications.
In this post I introduce the Good Enough Recommendation (GER) engine. GER (pronounced like this) is built to be easily usable through a simple API, as well as being reasonably fast and scalable, to let developers focus on their applications and not a recommendation engine.


Waterfall to Agile: An Introduction to the Waterfall, Scrum and Kanban Software Method(ologies)
Mon, 07 Jul 2014 04:11:00 +0000
There was this time, known as the bad old days, where programmers were seen as the assembly line workers and mechanics. The perception was that programmers didn’t create, they were the people who merely put together and fixed software systems, systems that were really ‘created by’ people who wrote the specifications.
Giant requirements documents were dumped onto programmers desks, then they were told that most of the work had been done already and all they had to do was put it together. Then after slaving away building a system for months or years, they would find the thing they created was not what the customer wanted, and was essentially useless and/or fundamentally flawed, e.g. INCIS or Novapay.


Vagrant with Docker: How to set up Postgres, Elasticsearch and Redis on Mac OS X
Wed, 02 Jul 2014 00:45:00 +0000
After some time spent looking at Docker from afar, hearing everyone talk about how awesome it is and how all the cool kids are already **** using **** it. I decided to test drive Docker out by using it in my development environment. In this post I will describe how to set up Postgres, Elasticsearch, and Redis as Docker containers with Vagrant on Mac OS X.
What is Docker?
Docker uses lightweight containers to separate an application from the operating system it is running in. It puts the application in an isolated box that only exposes selected folders or ports required for that application to be used.


D3.js: Tips, Tricks and Tools for Creating and Working with Maps
Sun, 22 Jun 2014 10:25:00 +0000
Creating maps to be used in visualisations can be a very difficult task. Although I have written about creating visualisations using maps, I have not yet written how I created those maps which can be just as important.
In this post I will describe how to use a few tools (koordinates, QGIS, ogr2ogr and TopoJSON) to generate, edit and transform maps to be used in visualisations.
Note: TopoJSON command line tool has implemented much of the ogr2ogr functionality described in this post. There is also a free online tool mapshaper that can be used instead as well.


Marketing by being Useful: How You can be Yousful with Youtility
Sun, 15 Jun 2014 08:27:00 +0000

I recently picked up a copy of Youtility: why smart marketing is about help not hype by Jay Baer and found it interesting enough to blog about.
Youtility is about how to engage your customer by being useful to them, thus gaining the limited and valuable resource — their attention. In this post I will highlight the interesting parts of the book, and also try to convey the overall themes and content.


Become a Publisher with Redis and Ruby
Tue, 06 May 2014 09:30:00 +0000
Lately I have been learning Redis and experimenting with the features it provides. One such feature is its publisher/subscriber functionality that allows a publisher to push messages to many subscribers. This functionality is deceptively easy to use and a very handy tool to know, so in this post I will describe how to use the this Redis functionality in Ruby.
Note: If you want to learn about Redis Redis in Action
Sending Messages
There are many problems that can be simplified to the publish subscribe pattern (pub/sub). Twitter is a great example of a massive publish and subscribe service, where someone publishes or tweets a message to many subscribers or followers.


Enterprise Software and Building Infinite Staircases
Wed, 23 Apr 2014 23:29:00 +0000
Being new to an enterprise organisation, I found that reading Martin Fowlers book Patterns of Enterprise Application Architecture (PoEAA) a cathartic exercise. It described the confusing and often frustrating world I found myself in, and prescribed solutions without claiming to be a panacea.
Enterprise systems are difficult, and not only because they can be technically challenging. In this post I will briefly look at two problems of enterprise systems (illogical requirements and modelling objects across teams) and try to describe ways to mitigate them.


Scrolling Elasticsearch using Node.js and Promises
Mon, 14 Apr 2014 01:28:00 +0000
Elasticsearch is a great way to store lots of documents that need to be quickly searched and retrieved. In addition to a broad query API, Elasticsearch also provides scrolling functionality that lets you query the server and incrementally download the results. This can be really useful for processing any large result set, e.g. for reindexing.
Using this scrolling, however, can be difficult because it first requires setting up then repeatedly calling the server to access the results. In order to simplify scrolling, I have implemented a package called ElasticScroll using Node.js with the Q promises library. In this post I will describe how ElasticScroll works and how to use it.


D3.js Tutorial using HTML, Scales and Chili Peppers
Mon, 07 Apr 2014 07:52:00 +0000
If you want to create visualisations for the web, the tool you will want to learn is D3.js. It helps you build powerful visualisations that can express your data while remaining light-weight and flexible. However, D3 is one of those tools that has its own way of doing things that is a little bit different from other visualisation libraries. It has a philosophy that focuses on data and functions, and not visual elements. This may make it difficult to learn because to wrap your head around it you need an epiphany, where you just get how to use it.


Lessons from Science on How to Test your Code
Sun, 30 Mar 2014 18:57:00 +0000
The motto for the Royal Society is Nullius in verba, which translates to Take nobody’s word for it. This comes from a scientific culture of scepticism that does not care how renowned, well known, popular, or respected someone is, you should always demand evidence for any claim put forward.
I have been thinking about how scientific ideas like this are analogous in software engineering when writing tests. These days it is common to write tests to demonstrate that your code is working, no matter how good a programmer you claim to be. I think this is the software community rediscovering that they should Take nobody’s word for it, not even your own.


Using Q and JQuery Promises to Compose Complex Animations
Sun, 23 Mar 2014 00:42:00 +0000
I recently created a visualisation that showed the difference between rich and poor schools in New Zealand. This visualisation used a series of relatively complex animations to introduce and convey information to the viewer.
In this post I will describe how I used Q and JQuery promises to compose the complex animations and give examples of how using promises can be beneficial for such a visualisation.
Note: I am going to use CoffeeScript in this post for reasons described here. To learn CoffeeScript perhaps you could try The Little Book on CoffeeScript.


When to be a Software Architect
Mon, 17 Mar 2014 06:55:00 +0000

Many of the constraints I have in my day-to-day job as a developer come from this mysterious world of the software architect. After listening to Martin Fowler on the Ruby Rogues podcast talk about his book Patterns of Enterprise Application Architecture (PoEAA), I decided to pick up a copy (i.e. I ordered it from Amazon) and read a bit more into architecture. Not understanding the domain very well, I also watched a few talks and read a few different articles trying to get a feeling for the important aspects.


Why should you use CoffeeScript instead of JavaScript?
Wed, 12 Mar 2014 04:03:00 +0000

CoffeeScript is currently my favourite language to write in! That is because CoffeeScript contains three things that I like in a language:

easy function and lambda definitions
syntactically significant whitespace
straight forward class definitions

However, for all its benefits, CoffeeScript is complicated by its intertwined relationship with Javascript. This is because CoffeeScript is a language that doesn’t compile to binary or VM code, but it is transpiled to Javascript. So, to understand why you should use CoffeeScript you should probably understand why you should use it instead of Javascript.


The Difference Between Rich and Poor Schools in New Zealand
Thu, 06 Mar 2014 10:33:00 +0000
In New Zealand we split our schools into 10 groups called Socio-economic Deciles that each represent 10% of schools ranked by the relative poverty of the students family. That is, Decile 1 contains 10% of our schools where the poorest students attend up to Decile 10 that contains 10% of our schools where the wealthiest students attend.
New Zealand uses this measure to target funding to more needy schools and try and ensure that compulsory education is equal for all students, some of which may not be as privileged as others.


What is Ruby? It is fun and makes you happy!
Mon, 03 Mar 2014 04:23:00 +0000
I have been working as a Ruby Programmer for over a year. Now I am thinking about how much I have learnt and how happy it has made me.
In this post I will briefly go over the history and community of Ruby, then give a small example that I think demonstrates why Ruby is made for developer happiness. This is not a tutorial, more like an introduction to Ruby.
History



Graham’s Ph.D. Thesis: A Study of Software Component System Evolution
Sat, 22 Feb 2014 06:48:00 +0000
Since I spent 4.5 years completing my Ph.D. at Massey University, I have decided 1 year on to give an overview of what I actually did.
If you want my full thesis, you can download it here. Be warned, it is really long, really dense and an academic document. I am trying to make this post a bit more reader friendly.
So here it goes…
Introduction to Components and Evolution

In order to agree to talk, we just have to agree we are talking about roughly the same thing. The Feynman Lectures on Physics, Motion, Richard Feynman, 1961.


Testing promises in Node.js with Mocha, Chai and Sinon
Sat, 15 Feb 2014 22:34:00 +0000
To test my Rails projects I typically use rspec. I really enjoy the way it helps me layout and describe my tests.
So when I started writing my first node.js package (back-on-promise), I wanted a similar way in which to write my tests when testing promises. I decided to use mocha for running the tests, chai for test assertions, and sinon to mock and stub objects. In this post I will describe how to test with these tools in node.js, specifically looking at promises.


Envisioning Information with Edward R. Tufte
Fri, 07 Feb 2014 20:09:00 +0000
Envisioning Information was recommended to me by a friend as a way to improve how I think about and design visualisations. Although less popular than his other work Visual Display Quantitative Information (which is still on my to-read list), this book has many interesting examples and ideas on how to present complex information. Of particular interest to me, this book gives a large discussion on how cartographers design and present geographical maps. It also gives practical rules on how to design, colour and layout data in a visualisation.


JQuery Promises and Deferreds: I promise this will be short
Fri, 31 Jan 2014 11:13:00 +0000
I am always looking for beautiful solutions to complex problems, and recently I have been experimenting with promises to solve the ugly problem of asynchronous actions in javascript. Promises are a simple metaphor that make complex operations easy to understand. In this post I will describe what promises are, why they are beneficial, how to use them, and a project I have been working on called Back-on-Promise which integrates promises into backbone.js.


RPython: Compiling Python to C (for the speed)
Sun, 12 Jan 2014 21:37:00 +0000
SAT Solver
Last night decided that I am going to write a SAT solver. But I had some conflicting requirements:

I want it to be fast (SAT solvers are very resource heavy)
I want it to be modular (Easily replace parts for more efficiency)
I want the code to be very understandable.

SAT4J is the SAT solver I am most familiar with. It is as efficient as Java lets it be, it is very modular, and it is reasonably understandable on the surface, but the lower you get the more it is obfuscated.


RPython is not a Language: An introduction to the RPython language
Sun, 12 Jan 2014 20:41:00 +0000

The exact definition is “RPython is everything that our translation toolchain can accept” :)

The above quote is from the coding guidelines for RPython. RPython is not a typical language, in that it is not described by a syntax, but is defined by whether or not a tool chain can compile the code.
RPython is a subset of the Python language. That is, any RPython code can run in a Python interpreter. The difference is that you can compile RPython code, with the RPython tool-chain down to C code. So the advantage of RPython is speed after compilation, and the disadvantage is that you cannot use all of Python’s features.


How to Deploy Mahara on Heroku
Sun, 12 Jan 2014 20:26:00 +0000
Recently, I was in need of a testing environment for Mahara’s functionality. For those that do not know, Mahara is an ePortfolio and social networking system, written in PHP, and developed right here in New Zealand.
To test Mahara, I came up with three possible approaches:

download&install mahara from apt-get (with all its dependencies). Then setup and configure a local environment in which I would be working, as described here.
Download a virtual machine (maybe from here) and configure it to host Mahara.
Or create and deploy Mahara to my favourite app server, Heroku.

I tried for an hour or so to install Mahara locally, however I just ended up where I always do, Apache config hell.


Drawing Maps with D3.js and Other Geographical Fun
Sun, 12 Jan 2014 06:21:00 +0000
I recently decided to create some mapping visualisations. Mostly because using a map is an awesome way to present many data sets, and creating such visualisation is a skill I lacked.
So I looked around and found that D3.js has geographical features.
I had used D3.js in the past on projects like 100 companies, so I understood how to use it and could apply that knowledge to make visualisations with maps.
In this post I will go over a few examples of how to use D3’s geographical API to create visualisation with maps.