Posts

I want an Error to be raised during JSON unmarshalling if an undetected field is found. This is useful if you are trying to be extra careful, for example double checking the client has not mispeled any inputs. However I want to exclude some fields from this that might be used elsewhere.Model is the struct that we are unmarshalling. We must create a new type so that the unmarshalling function is not recursive (as explained below): type XModel Model ...

In 2017, Puppet and DORA (DevOps research and assessment) published their annual State of DevOps Report that collates more that six years of survey data about the cultural and technical impacts of DevOps. Analyzing over 27,000 responses they found that high performing engineering organizations have: 46x more frequent code deployments (on demand deployments) 440x faster lead time from commit to deploy (less than one hour) 96x faster mean time to recover from downtime (less than one hour) 5x lower change failure rate (0%-15%) That is, high performing teams: ...

When writing blog posts I have a few requirements: Offline: I travel and commute which means that I have no internet frequently which is a great time to write distraction free. Easy Publishing: I moved all my blogging to medium because I spent far too much time doing things on the blog that were not writing. Syncing: I like to write on my laptop, but sometimes it is more convenient on my phone, like when I am waiting for a train. Formatting: I like Markdown, Medium doesn’t use markdown. This is my biggest issue with Medium. Typically I will write a post in Markdown then manually convert it to Medium which is super annoying.Previously I have used Evernote which is great offline and syncing, but it has a different format to both medium and markdown making which makes it difficult to publish with. I have also just used a plain text editor Sublime Text with GFM markdown plugin, then wrote content in Notes in iOS and moved it manually to my laptop. This is a lot of manual syncing, and once completed I still have to convert it to mediums formatting. ...

AWS Bread, recipe below As developers we are pretty good at writing fast code because we put a lot of emphasis on that skill (especially in job interviews). Where we have a little more trouble is writing slow code, processes that don’t take milliseconds or seconds to run (e.g. a web request), but take minutes, hours, or days (e.g. data backup and migration) to complete. As a process takes longer to complete, some qualities become much more important: ...

Building security into your company culture is necessary but challenging. The first line of defense against most attacks are aware and vigilant people with an attitude of “see something, say something”. attr. Tony Webster Training employees on what is abnormal behavior and who to talk to when there is a potential problem can save your company from a lot of pain. For example, alerting security when a co-worker makes an unusual request because one of their accounts has been compromised by a hacker. ...

At Coinbase we use a GitHub-flow-ish workflow to collaborate on code and develop features: Create a branch Create/Edit/Delete/Rename files on branch Create a Pull Request (PR) merging the branch into master Get code reviewed and make requested changes Merge the PR However, we are still having discussions around the final step. How should we merge pull requests? The options are: Merge the pull request, git merge Manually squash the branch with git rebase or git reset , force push, then merge Use the “Squash and Merge” function in GitHub, basically git --squash merge branch The core differences between these methods is how much friction it is for developers to use, and what historical information is left behind. ...

Bastion at Castel Sant’Angelo You need more than one AWS account. This is to isolate production resources, manage limits (especially API rate limiting), handle costs, simplify compliance and security concerns, and restrict user access. However, managing multiple AWS accounts can be difficult. To help with this AWS has built many useful features like organizations, consolidated billing, VPC peering, and descriptive IAM policies. Using these features effectively to structure and connect AWS accounts, a variety of patterns have been developed. One such pattern used by Coinbase is the AWS Bastion account. ...

I am pretty sure that I am human. I am also pretty sure that my wife, my friends, my colleagues, and other people I have met in real life are also human. I am less sure about disembodied entities on the internet, like you. If you are human, then welcome to this post and I would like to hear what you think about it in the comments. Before you post anything though, I need you to prove to me that you are human because I don’t want to read comments posted by evil-spammy-bots. ...

A Domain Specific Language (DSL) is a specialized way to clearly describe a problem domain. Ruby is a great language for creating DSLs because it lets developers decide how the language looks and is used. For example: RSpec for writing tests: FactoryGirl for mocking objects: GeoEngineer for defining cloud resources: This post will briefly describe how to use Ruby to create DSLs like the examples above. Building Blocks Ruby DSLs typically use functions that take a block given as an argument to build an instance of an domain object: ...

Zion National Park’s amazing geology The goal of the infrastructure team at Coinbase is to provide self-service tooling to our engineers to empower them to rapidly develop, monitor, and optimize services with low risk. With this mission in mind, we are currently in the process of building a workflow for creating and managing our codified infrastructure resources that looks like: Pull Request: an engineer submits a pull request to a repository with a new codified resource they want. Validation: the new resource is automatically validated and follows our company standards for naming, tagging, and security. Plan and Review: a plan describing the actions needed to be taken to apply a change is presented alongside the code change to be reviewed by an infrastructure team member. Merge then Apply: if the plan is good, then the pull request can be merged and automatically applied to the cloud. This workflow manages our codified infrastructure the same way we manage our code with GitHub flow; i.e. open a pull request, ensure the change is valid with tests, merge the change into the master branch, then apply the changes to the necessary environments. The main idea of this workflow is to improve collaboration between the infrastructure and engineering teams. This will also speed up the development and deployment of resources, and make sure we deliver what is actually needed. ...

While running or driving I enjoy listening to podcasts as they are a great way to passively learn and keep up with various technologies and trends, as well as be entertained. This post is a list of the podcasts I enjoy (or have enjoyed) listening to. Programming DevChat podcasts Ruby Rogues: Ruby and programming focused panel podcast, I find the earlier episodes with the original crew were the the best and worth going back for a listen. ...

Alcoa Keystone Charles Duhigg in his book Power of Habit discussed how Paul O’Neil, the CEO of Alcoa (Aluminum Company of America), was able to increase his company’s value by 27 billion by focusing on a goal unrelated the main company objectives, no workplace injuries. This is a keystone metric (habit), a broad goal for the entire company, not directly related to the company’s main goal but impactful and actionable. O’Neil chose safety as the single most important aspect because: ...

A bad way to implement a currency exchange would be to have the customer pass their order to an employee who then manually checks the customer balances, decides whether the order is valid, then writes down the order in a big order book (a la the old New York stock exchange). This method would be slow, prone to human error, not scalable, and more customers would require more employees. A better way would be to give the tools to the customer (with appropriate validations and security) to create the orders themselves, making the exchange more efficient and less expensive to run. ...

Name a technology that is more useful, more educational, more interesting, and more overpriced than a ultrasound machine. You can look inside of living things without the need for a powerful magnets or radioactivity and it is basically made from a speaker and microphone outputting to a screen. Why doesn’t every high school biology class room have an ultrasound to show how muscles work, and hearts beat? Why don’t doctors have them immediately handy like a stethoscope or thermometer? Why can I not get one just because I am interested in how my injuries are healing? Probably because “a £20,000 [$30,000USD] scanner is generally classed as low cost.” ...

Strands published a great vision paper about The Big promise of recommender systems. In this post I break down the paper into a few useful requirements; Recommender systems should: be easy to integrate and easy to remove. Vendor lock-in is a negative so having a minimal impact on the clients system is a big plus complete with internal marketing teams. Generated recommendations will compete for space and resources with internal marketing departments as their core goals are identical. So reuse the marketing metrics and methods in recommender systems to measure and drive real business value first collect enough data to avoid the cold start problem. Don’t release the recommender systems functions till there is enough data to provide good recommendations scale to the businesses needs, number of users and number of items be a hybrid of approach as these create a robust solution that solves many problems of exclusive algorithms be a balance between algorithms and UX. Focusing entirely on algorithms or UX will not create value, it must be a mixture of both use implicit instead of explicit feedback. Explicit ratings or reviews can be manipulated, however implicit (e.g. if they actually bought or watched the item) are more difficult to manipulate and will provide better data differentiate their products as recommender systems have become commoditized. If a system if difficult to evaluate and looks similar to other systems it will fail have contextual awareness of where the recommendations are being shown. For example, recommendations on mobile are different to web and different to email Check out my Good Enough Recommender (GER) to see how it can be used to implement the requirements in this list and check out the List of Recommender Systems and see how they implement these requirements

Over the last 6 months my wife and I have been working and travelling around Europe using Brighton U.K. as a base. Before we left we looked at a few different ways of getting a roof over our head while travelling, our ideal solution would be low hassle, low responsibility, low liability, and easy to set up. The original idea was to rent an apartment in Brighton but this can be very expensive, difficult to find and organise from overseas, and requires signing contracts, putting up bonds and other legal arrangements. In addition to these problems, the local law also puts a stupid paradox in the way of any traveller renting an apartment; To rent a British apartment you need British bank account and to get a British bank account you need a British address. From the HSBC site ...

HapiGER, the Happy (Good Enough) Recommender System, has a new version! This version: is Simpler: with a smaller, better defined API is More Configurable: can better define how to generate recommendations giving more control over quality and speed has Added Functionality: can now generate similar things recommendations to find items that are similar to other items Links: HapiGER HapiGER on npm HapiGER source GER Source List of Recommender Systems

Why should you learn new things? Because you can’t predict the future, and knowing is better than not knowing. What you learn doesn’t need to be applicable, you don’t have to put it on your CV, it doesn’t need to be measured and quantified, you don’t need to tell anyone. Learn anything; learn a toy, a gimmick, something too new or too old, something that is popular or something that is niche. It doesn’t matter if it is not supported, not serious or not professional. Learning is an exercise, so the worst that can happen is you get faster at learning the next time. ...

In this post I will briefly describe three optimisations that GER (Good Enough Recommendations) uses to improve recommendations. These optimisations could be useful for other recommender systems as they have increased the quality of GER’s recommendations. Separated Data The two fundamental steps in recommender systems like GER are: find a neighbourhood of similar people recommend things that those people like The similarity between people can vary widely from one set of items to another. For example, my friend and I have very similar tastes in action movies, but we don’t like the same drama movies. If all the data was combined, our similarity would be very weak. However, if we separated the data based on movie genre we would be very similar in the “action” genre, and dissimilar in the “drama” genre, now I can get some good “action movie” recommendations. ...

I want to run an application on my CoreOS clusters that uses hostnames to communicate between machines. This is a problem, because out of the box CoreOS machines cannot resolve hostnames of other machines in the cluster. So, I wrote a small fleet service that manages the /etc/hosts files on all the machines so they can correctly resolve each others hostnames. In this post I will briefly describe that service. The Hosts Service The hosts.service: # hosts.service [Unit] Description=Hosts Manager After=etcd2.service``[Service] EnvironmentFile=/etc/environment Restart=always``ExecStartPre=-/usr/bin/etcdctl mkdir /hosts``ExecStart=/bin/sh -c 'while true; do etcdctl watch --recursive /hosts; \ sleep 1;\ echo "127.0.0.1 localhost" > /etc/hosts; \ for i in $(etcdctl ls /hosts); do \ echo $(etcdctl get $i) $(echo $i | cut -c 8-); \ done >> /etc/hosts; \ done'``ExecStartPost=/usr/bin/etcdctl set /hosts/%H $PRIVATEIP ExecStopPost=/usr/bin/etcdctl rm /hosts/%H``[X-Fleet] Global=true ...

Separating an application into many small independently developed and deployed microservices that communicate over a thin layer (like http) has many benefits (see Fowler’s article). However, one of the main drawbacks of this architecture is the difficulty automating end-to-end tests for the application. Newman’s Building Microservices asks two questions when end-to-end testing a microservices application: Which versions of the services should we test? Where are the tests written, to not to duplicate the effort for each service? His solution is to have an external end-to-end test suite that can be run against many configurations of microservice versions. In this post, I present an implementation of Newman’s end-to-end microservices testing solution that uses the tool pmux and the continuous integration service TravisCI. ...

I want you to think how deeply dysfunctional it is for you to be afraid of what you created — Uncle Bob Martin Architecture the Lost Years When I sit down to some code that I haven’t touched in a while, I hesitate. This is weird; I wrote it, I created it, so why am I afraid of changing it? Because, the code works now and if I change something it might break. ...

I previously wrote an introduction to the Tessel here where I described a program that blinked out tweets in morse code. For my second project with the Tessel I wanted to use some of the modules that came with it to create a basic security device/camera to take pictures and alert me when something bad is happening. Modules The Tessel has 4 different ports to plug modules into, each labeled A, B, C and D. The four modules I decided to use for this project were: ...

I borrowed a Tessel from a friend @leighghunt to play with some embedded node.js. My first project is to listen to a twitter stream and blink it out as Morse code on the Tessel’s LEDs. Getting Started npm install -g tessel Initialising the project: npm init I like CoffeeScript, so to test the Tessel out I adapted the demo code from here to a file called twitter_morse_code.coffee: tessel = require('tessel')``led1 = tessel.led[0].output(1) led2 = tessel.led[1].output(0)``setInterval( -> console.log("I'm blinking! (Press CTRL + C to stop)") led1.toggle() led2.toggle() , 100) ...

NOTE: I am also maintaining this post on github. This way I can accept pull requests for changes and additions :) Recommender systems (or recommendation engines) are useful and interesting pieces of software. I wanted to compare other recommender systems to mine (HapiGER) but couldn’t find a decent list of them, so I decided to create one. In this post I will list the recommender systems that I have come across with links and some basic information about them. I intend on keeping this list up-to-date, so comment below if I am missing one or tweet me @grahamjenson. ...

In my previous post about using Vagrant to run Docker containers, I recommended not to use Boot2Docker. Since then my issues with Boot2Docker have been resolved and now it is my preferred way to use Docker in OSX. Here is a quick tutorial on how to set up Postgres, Elasticsearch, and Redis as Docker containers using Boot2Docker on Mac OS X. Boot2Docker for OSX If you are using OSX and want to use Docker, then Boot2Docker is the recommended tool. It works by using a virtual machine to host Docker and letting the OSX command line ‘remotely’ call it with the docker command. ...

I am proud to announce the beta version of HapiGER an open-source, easy to use, easy to integrate recommendations engine. It is built using the Good Enough Recommendations (GER) engine and the Hapi.js framework. In this post I will describe how you can use HapiGER to generate recommendations for your users. Install HapiGER Install with npm npm install -g hapiger Start HapiGER By default it will start with an in-memory event store (events are not persisted) hapiger ...

accepts_nested_attributes_for is a really powerful method in Rails because it allows a model to alter related models through itself. However, it has a pretty big gotcha. An example using accepts_nested_attributes_for is where a user model which belongs_to an alias model, i.e. #user.rb class User < ActiveRecord::Base end``#alias.rb class Alias < ActiveRecord::Base belongs_to :user accepts_nested_attributes_for :user end This allows the Alias model to change the user by passing a hash key user_attributes i.e. `Alias.first.user.name ...

Docker is great fun when you start building things by plugging useful containers together. Recently I have been playing with Consul and trying to plug things together to make a truly horizontally scalable web application architecture. Consul is a Service Discovery and Configuration application, made by HashiCorp the people who brought us Vagrant. Previously I experimented using Consul by using SRV records (described here) to create a scalable architecture, but I found this approach a little complicated, and I am all about simple. Then I found Consul Template which links to Consul to update configurations and restart application when services come up or go down. ...

I am not sure where the boundary between Data and Big Data exists, but I must be getting close. As the amount of data I deal with grows, my tools and processes have had to significantly adapt to many new challenges. This caused me to start to think about the qualities of working with Big Data as compared to physical laws. In this post I will explore the challenges of working with Big Data using some analogies with scale, malleability and gravity. As with all analogies, these are not exact and just exploring a different way of thinking about a problem ...

Yeah, Nah: New Zealand slang for yes, or possibly no e.g. Yeah, Nah is a movie recommendation application built with the Good Enough Recommendation engine (GER), using themoviedb.org’s API for movie information, Hapi.js as its web framework, and Angular.js for front end code. Source here

GER (Good Enough Recommendations) is a recommendations engine that could directly add value and increase user engagement for many existing applications. GER is an open source npm module that you could download and start using right now. However, you probably want to know how GER works and how to use it to get good recommendations out of it. In this post I describe GER’s core model, its practical features and its limitations to help you use GER to get good enough recommendations. ...

Search boxes are useful so they are everywhere! Yet they are treated with disdain, as secondary class elements, hidden away in the corners and pushed to the side of other ‘more important’ content. Only put there as a last resort for frustrated users who cannot navigate to their destination. As if to try and make users feel like they failed when they have to use the search box. If you treat it your search box this badly it will never provide you its full potential value. The search box gives your users a way to tell you, in their own words, what they want and what they expect you to have. You do not need to guess these things, just read what they typed no further analysis required. ...

These are some of the tools I use to test my Node.js code: Mocha is a testing framework for describing and running tests Chai is an assertion library Sinon is a mocking and stubbing library In this post I will give a brief introduction to each of these, with some basic examples and tips. Mocha Mocha.js is a test running framework. Install Mocha with npm install -g mocha. Run mocha to execute all the javascript test files in the test directory. ...

Do you need a solution to scale your web services both vertically and horizontally, with load balancing and health checking? Consul with Docker and Nginx can help! In this post I will describe how to use Consul (a service registry) and Nginx (with srv-router) running in Docker containers to load balance across multiple services. Note: For OSX users boot2docker is required Consul Consul is a service registry that uses the DNS protocol to return a list of healthy services, in a random order (for load balancing). ...

A common change to make to a Rails application is to extract an attribute from a model into a one-to-many relationship. This change can be made without causing a large amount of downtime, even if there a significant amount of records needing to be changed. In this post, I will describe how to change a model to replace an attribute with a one-to-many relationship while minimising downtime and emphasising continuous deployment. ...

Docker is a great tool to test out new application architectures. To make sure that my architectures routed the right calls to the right places, I needed to have a simple containerised web service that just logged when it was called. In this post, I am going to describe how to create a simple web service with Docker and Python, it is even small enough to fit in a tweet. This post will be brief. ...

When implementing the Good Enough Recommendations (GER) engine, a core requirement was to let users insert large amounts of data quickly in order to bootstrap the recommendations engine. Additionally, this bootstrapping should be available over HTTP, as this will become the primary channel for interaction with GER. PostGres (which GER uses) has the COPY command that is “optimised for loading large numbers of rows” in various formats, and npm has the package pg-copy-streams that pass Node.js streams to COPY. This would work well with the Hapi.js web application framework which can turn an uploaded file into a Node.js stream without having to hold the entire file in memory or create a temporary file on disk. ...

While developing the Good Enough Recommendations (GER) engine, I needed to Upsert a record in Postgres. Upsert is a function that updates a record if it exists, or inserts the record if it doesn’t. However, Postgres doesn’t come with upsert out-of-the-box, so I had to find out how best to implement it. In this post, I will describe two methods to upsert records in Postgres, multi-query and single-statement. Then I will describe how I compared them to select a method for GER to use. ...

Recommendation engines could be beneficial for many applications as they can directly add value and lead to greater engagement for users. However, there is significant overhead in implementing a custom solution and many off-the-shelf engines have overcomplicated APIs, or try to be infinitely scalable which is not needed by most applications. In this post I introduce the Good Enough Recommendation (GER) engine. GER (pronounced like this) is built to be easily usable through a simple API, as well as being reasonably fast and scalable, to let developers focus on their applications and not a recommendation engine. ...

There was this time, known as the bad old days, where programmers were seen as the assembly line workers and mechanics. The perception was that programmers didn’t create, they were the people who merely put together and fixed software systems, systems that were really ‘created by’ people who wrote the specifications. Giant requirements documents were dumped onto programmers desks, then they were told that most of the work had been done already and all they had to do was put it together. Then after slaving away building a system for months or years, they would find the thing they created was not what the customer wanted, and was essentially useless and/or fundamentally flawed, e.g. INCIS or Novapay. ...

After some time spent looking at Docker from afar, hearing everyone talk about how awesome it is and how all the cool kids are already **** using **** it. I decided to test drive Docker out by using it in my development environment. In this post I will describe how to set up Postgres, Elasticsearch, and Redis as Docker containers with Vagrant on Mac OS X. What is Docker? Docker uses lightweight containers to separate an application from the operating system it is running in. It puts the application in an isolated box that only exposes selected folders or ports required for that application to be used. ...

Creating maps to be used in visualisations can be a very difficult task. Although I have written about creating visualisations using maps, I have not yet written how I created those maps which can be just as important. In this post I will describe how to use a few tools (koordinates, QGIS, ogr2ogr and TopoJSON) to generate, edit and transform maps to be used in visualisations. Note: TopoJSON command line tool has implemented much of the ogr2ogr functionality described in this post. There is also a free online tool mapshaper that can be used instead as well. ...

I recently picked up a copy of Youtility: why smart marketing is about help not hype by Jay Baer and found it interesting enough to blog about. Youtility is about how to engage your customer by being useful to them, thus gaining the limited and valuable resource — their attention. In this post I will highlight the interesting parts of the book, and also try to convey the overall themes and content. ...

Lately I have been learning Redis and experimenting with the features it provides. One such feature is its publisher/subscriber functionality that allows a publisher to push messages to many subscribers. This functionality is deceptively easy to use and a very handy tool to know, so in this post I will describe how to use the this Redis functionality in Ruby. Note: If you want to learn about Redis Redis in Action Sending Messages There are many problems that can be simplified to the publish subscribe pattern (pub/sub). Twitter is a great example of a massive publish and subscribe service, where someone publishes or tweets a message to many subscribers or followers. ...

Being new to an enterprise organisation, I found that reading Martin Fowlers book Patterns of Enterprise Application Architecture (PoEAA) a cathartic exercise. It described the confusing and often frustrating world I found myself in, and prescribed solutions without claiming to be a panacea. Enterprise systems are difficult, and not only because they can be technically challenging. In this post I will briefly look at two problems of enterprise systems (illogical requirements and modelling objects across teams) and try to describe ways to mitigate them. ...

Elasticsearch is a great way to store lots of documents that need to be quickly searched and retrieved. In addition to a broad query API, Elasticsearch also provides scrolling functionality that lets you query the server and incrementally download the results. This can be really useful for processing any large result set, e.g. for reindexing. Using this scrolling, however, can be difficult because it first requires setting up then repeatedly calling the server to access the results. In order to simplify scrolling, I have implemented a package called ElasticScroll using Node.js with the Q promises library. In this post I will describe how ElasticScroll works and how to use it. ...

If you want to create visualisations for the web, the tool you will want to learn is D3.js. It helps you build powerful visualisations that can express your data while remaining light-weight and flexible. However, D3 is one of those tools that has its own way of doing things that is a little bit different from other visualisation libraries. It has a philosophy that focuses on data and functions, and not visual elements. This may make it difficult to learn because to wrap your head around it you need an epiphany, where you just get how to use it. ...

The motto for the Royal Society is Nullius in verba, which translates to Take nobody’s word for it. This comes from a scientific culture of scepticism that does not care how renowned, well known, popular, or respected someone is, you should always demand evidence for any claim put forward. I have been thinking about how scientific ideas like this are analogous in software engineering when writing tests. These days it is common to write tests to demonstrate that your code is working, no matter how good a programmer you claim to be. I think this is the software community rediscovering that they should Take nobody’s word for it, not even your own. ...

I recently created a visualisation that showed the difference between rich and poor schools in New Zealand. This visualisation used a series of relatively complex animations to introduce and convey information to the viewer. In this post I will describe how I used Q and JQuery promises to compose the complex animations and give examples of how using promises can be beneficial for such a visualisation. Note: I am going to use CoffeeScript in this post for reasons described here. To learn CoffeeScript perhaps you could try The Little Book on CoffeeScript. ...