5 Awesome Years at Coinbase
I joined Coinbase March 28th 2016; my last day is April 2nd 2021. This is a quick write up about my experience on joining and working in the infrastructure team for 5 years at Coinbase, growing from 100 to over 1000 employees, from 3.5 million users to more than 40 million, and from trading $3 billion total to now often trading that in a single day.
In 2015, I got an email from a recruiter:
I may have responded because of the personal nature of the email (mentioning my GER project), but TBH I have no idea why I responded. I emailed back that I was more into DevOps work which they also had positions for. So we set up some online interviews.
I didn’t really need this job so I was super relaxed during the interviews, mostly I just had a good time solving interesting problems. One question was to implement the Mandelbrot set which I did in Ruby, and another was some data munging problem I did in Javascript. In one interview Brian Armstrong dropped by to ask a few questions, including if I was a Coinbase user. After saying “no” he sent me $15 of Bitcoin (he used to do that with everyone).
I think because I had fun and demonstrated some skill with multiple languages they decided to fly me (and my wife) to SF for a week long interview called a work-trial.
At the work-trial I was given a project to setup a third-party code-coverage tool for Ruby. That quickly got derailed when I realized the tool wouldn’t scale to Coinbase’s large codebase, so I started looking around for other problems to solve. I found a way to speed up some test times, worked with an engineer on a visualization, and also tried to help out (i.e. flailed around) during a few DDOS incidents.
Hanging out in a cool SF office with a ton of talented engineers building high stakes applications was exhilarating. There is nothing like being in a room full of experts working hard to solve an immediate problem. I think my willingness to jump in, help and to find useful things to do when stuck is why I got an offer at the end of the week.
Ultimately, I decided to join Coinbase for a few reasons. First, Rob Witoff (then director of infrastructure) laid out a vision of an organization I wanted to be a part of. Next, I saw Brian Armstrong & Fred Ehrsam as a good team; Brian seemed like a steady hand in rocky seas and Fred was so enthusiastic about crypto and the company, his optimism was infectious. Finally, every person I talked to, or worked with, was friendly and fully engaged with their work.
I don’t think I could have gotten that kind of insight (and might not have joined) if not for the work-trial. Unfortunately, work-trials don’t scale to a large company so they are not part of the interview process anymore.
Really the only thing that I wasn’t sure about was moving to the United States. New Zealand is home; family, friends, stability, safety... But San Francisco isn’t that far away, it’s only a 12 hour flight. I could get on a flight in the evening and have breakfast at home the next day.
I moved to the US in March 2016. It was a pretty stressful time. I started on the “infra & security” team that had 2 infra engineers. I liked the team and heavily relied on my new colleagues for advice about all the complexities of living in the US (SSN, banks, health insurance…) and for onboarding me into the company.
Within a month I was the only member left on the infra team. To be clear, I was not the first infra engineer, but for a short time I was the only one (with the exception of a wild Lian).
Being the only FTE infra engineer meant that I had a lot of say in building the new team. We interviewed a lot of great candidates and grew the team to 7 people in 18 months. I think helping build that team was probably my greatest impact and proudest achievement at Coinbase. They are all wonderful talented people and I would work with any of them again.
My US visa ran out after 18 months in September 2017, so I moved back to New Zealand for a bit. Then to the office in the UK (where I could get a visa) for a little over a year. As soon as I got another visa, I went back to San Francisco in September 2019. By the time I got back the US the team had grown from 7 to 30-ish. Then the pandemic. Now, after another year it’s 60-ish. That is crazy growth.
What did I do there?
On my first day at Coinbase Rob and I sat in a room and wrote down what my first project was “Decrease the time a developer takes to go from commit to production”. So I did that for 5 years.
I started by improving the way AWS resources were provisioned with GeoEngineer, a Ruby DSL that maps to Terraform with additional validations and Coinbase-specific logic. The goal was to decrease the time it takes to provision/manage resources for projects. In 2018, we started migrating away from the Ruby DSL to a YAML format called GPS to better standardize. It has been written and spoken about here, here, here, and here.
Next, I moved onto extending and building the code-review system Heimdall. Heimdall decides if a Git SHA in a repository was reviewed, allowing it to be deployed. I talked about Heimdall here, and here and it is discussed here (but it’s called Sauron).
I was the primary maintainer/builder for Codeflow, the paved-road for engineers to deploy code. If Codeflow was down, nothing would get released, so it had to be reliable and secure. I talk about Codeflow here.
We were having reliability issues with existing deployers and developers wanted more options, so I built and open-sourced Odin and Fenrir. Odin deploys to Auto Scaling Groups and Fenrir to serverless Lambda. I have talked about these deployers here, here and here and here and here.
Monorepo is the last big project I worked on at Coinbase. Codeflow requires one git repository per project, where a monorepo can have many projects in a repo. Currently the Coinbase monorepo has hundreds of projects implemented in Golang, Ruby, Python, and JavaScript. The monorepo has been written about here and here. A monorepo is a large multi-org effort involving many engineers adopting new technologies and working together. But, I am most proud that I got the first commit to start with 4c0ffee
I also helped out with scaling applications, managing blockr.io, automated documentation, slack integrations, CI servers, removing legacy services, responding to incidents, docker build pipelines, security scanning, disaster recovery planning, mentoring, interviewing… 5 years is a long time.
Why am I leaving?
I want to stay in New Zealand. I would love to continue to work at Coinbase, the people are amazing, the problems are interesting. However, moving back to the US isn’t an option right now. The distance is not a 12 hour flight anymore, now it’s a 2 week stint in managed isolation with an infant. That is too far to be from family and friends. This may only be the case for another 6 months-ish, but that is too long to live in limbo.
What is next
I think I am going to enjoy New Zealand with my friends and family for a few months before looking for something else. Whatever I do next, I will probably do for the same reasons I joined Coinbase; working with cool people on hard/fun problems.
This post in no way encapsulates all the fun and memories I had at Coinbase, which are too numerous to count.
To all the engineers, designers, managers, recruiters, operations, staff, and folks I worked with: Cheers for the awesome 5 years :)
