Learnlog
9:45 am — What I want
What I want to do is use Google’s build tool Bazel to build a tree of Dockerfiles that are FROM each other and use Bazels dependency engine to automatically rebuild docker containers if a container they depend on changes. This is currently a manual job at many companies, and as they grow this becomes more tedious and in need of automating.
I already have Bazel installed from a previous failed spike using https://github.com/bazelbuild/rules_docker which are a bit overcomplicated and difficult to use for someone who is just learning (like me).
Now it is time to read some docs.
9:55 am—So many docs
I have created an empty WORKSPACE and am trying to read the “getting started” docs. There are none (or few) for someone who is trying to create a new rule, this probably means either it is too complicated to get started with, or too easy to write docs for.
Bazel has lots of docs for everything though, but it does include some intimidating lines like
“Before creating or modifying any rule, make sure you are familiar with the evaluation model. You must understand the three phases of execution and the differences between macros and rules.”
Telling me that before I do anything, I better learn it all first. No, I learn by doing!
10:30am — Marcos or Rules
Macros are super simple rules that are immediately executed. Rules are loaded, analyzed, then executed only if necessary. So I want rules.
I have set up a folder structure:
WORKSPACE # emtpy rules/docker.bzl rules/BUILD # empty src/containers/ubuntu/Dockerfile # FROM ubuntu src/containers/ubuntu/BUILD
rules/docker.bzl is:
print(“here”) # want to see when loaded``def _docker_build(ctx): print(“Building “ + ctx.attr.name) # TODO``docker_build = rule( implementation = _docker_build, )
src/containers/ubuntu/BUILD is
load(“//rules:docker.bzl”, “docker_build”)``docker_build(name = “ubuntu”)
Running bazel build //src/containers/ubuntu prints the debug lines which is a good first step.
Now it is time to see if I can get it to call docker build .
11:30am — Bazel’s Weird attr Schema
So I have some more stuff working. I am getting to know Bazel’s weird schema for attrs.
I am missing something… although I call ctx.actions.run to execute docker and pass in a Dockerfile. When I run bazel build it doesn’t run.
docker.bzl is now:
print("here")``def _docker_build(ctx): name = ctx.attr.name dockerfile = ctx.attr.dockerfile.files.to_list()[0].path print("Building " + name) print("Dockerfile " + dockerfile) a = ctx.actions.declare_file(name + “.dockerout”) ctx.actions.run( executable = "docker", inputs = ctx.attr.dockerfile.files, arguments = [“build”, dockerfile], outputs = [a], ) print(a) return struct( docker_sha = “ubusntu”, )``docker_build = rule( implementation = _docker_build, attrs = { "dockerfile": attr.label(allow_files = True), "deps": attr.label_list(allow_files = True), }, )
with ubuntu/BUILD being:
load(“//rules:docker.bzl”, “docker_build”)``docker_build( name = “ubuntu”, dockerfile = “Dockerfile”, deps = [“Dockerfile”] )
OI am hoping to at least get an error back from trying to call docker build before lunch. I am missing something, like having to register the file explicitly as a dependency to make Bazel realize I want to build it.
I am pretty sure I just need to understand what this means:
“[The implementation] function does not run any external commands. Rather, it registers actions that will be used later during the execution phase”
12:00pm — Outputs
I needed to explicitly declare the outputs of the rule, otherwise why would it run 🤦♂
Also found this file dockerfile_build.bzl, which is pretty much exactly what I want, except it is a repositroy_rule (only usable in WORKSPACE). The goal at the moment is not to be 100% hermetic but to learn and build towards that. So cutting corners is fine for today.
Now when I run bazel build //src/containers/ubuntu:ubuntu I get the wonderful error docker failed: error executing command docker build src/containers/ubuntu/Dockerfile
Now Lunch
1:00pm — Toolchains?
After a Greek wrap at the Brighton street market, and reading a bit more from the docker_build.bzl file I think I need a toolchain.
1:45pm — Don’t need a toolchain!
Well that was 45 mins of wasted time.
The problem was the it was trying to execute docker build src/containers/ubuntu/Dockerfile which doesn’t work because docker build expects a folder arg not a Dockerfile 🤦♂. Using the folder instead, docker is now building. Just waiting for the docker build to finish.
2:00 pm— Reading docs
The most useful documentation page while writing rules https://docs.bazel.build/versions/master/skylark/lib/ctx.html
3:00pm — Understanding more
So I have a bunch more stuff working now. I decided to use a simple script to execute docker instead of calling the docker command directly. This is because there is no (easy?) way to output the STDOUT from docker using ctx.actions.run to a file.
I define the script using sh_binary in rules/BUILD :
package(default_visibility = [“//visibility:public”])``sh_binary( name = “docker”, srcs = [“docker.bash”], )
rules/docker.bash is:
#!/bin/bash output=$1 shift echo “docker $@” docker $@ > $output
docker.bzl is now
def _docker_build(ctx): name = ctx.attr.name folder = ctx.file.folder.path``ctx.actions.run( executable = ctx.executable._docker_tool, inputs = ctx.attr.folder.files, arguments = [ctx.outputs.dockerout.path, “build”, “-q”, “-t”, name, folder], outputs = [ctx.outputs.dockerout], )``return struct( docker_sha = “ubuntu”, )``docker_build = rule( implementation = _docker_build, attrs = { “folder”: attr.label( allow_single_file = True, mandatory = True, ), “_docker_tool”: attr.label( executable = True, cfg = “host”, allow_files = True, default = Label(“//rules:docker”), ), }, outputs = { “dockerout”: “%{name}.dockerout” }, )
and ubuntu BUILD file is :
load(“//rules:docker.bzl”, “docker_build”) docker_build(name = “ubuntu”, folder = “.”)
I am having a bit of trouble with the location that docker build is being executed from. I cannot find a method that easily returns the folder of the build directory, so am passing in the folder manually for now.
Now I am trying to make it rebuild when I change the Dockerfile
3:30pm — Bazel Magic
So Bazel is magic. I am still not sure what it is (or I am) doing, but I got the ubuntu image rebuilding when its Dockerfile is changed.
docker.bzl looks like this:
def _docker_build(ctx): name = ctx.attr.name folder = ctx.file.dockerfile.dirname print([f.path for f in ctx.attr.dockerfile.files])``ctx.actions.run( executable = ctx.executable._docker_tool, inputs = ctx.files.dockerfile, arguments = [ ctx.outputs.imagesha.path, “build”, “-q”, “-t”, name, “-f”, ctx.file.dockerfile.path, folder ], outputs = [ctx.outputs.imagesha], )``return struct( image_sha = ctx.file.dockerfile, )``docker_build = rule( implementation = _docker_build, attrs = { “dockerfile”: attr.label( allow_single_file = True, mandatory = True, ), “_docker_tool”: attr.label( executable = True, cfg = “host”, allow_files = True, default = Label(“//rules:docker”), ), }, outputs = { “imagesha”: “sha” }, )
with the ubuntu BUILD looking as docker_build(name = “ubuntu”, dockerfile = “Dockerfile”)
The next challenge is the hardest part. I want to make a ruby-2.6 image FROM the ubuntu image we are creating that will rebuild if its base ubuntu changes. The Docker tree part.
4:45 pm— Friday Drinks
I created a few more files src/containers/ruby-2.6/Dockerfile that builds a ruby container FROM ubuntu:bazel and src/containers/ruby-2.6/BUILD:
load(“//rules:docker.bzl”, “docker_build”)``docker_build( name = “ruby-2.6”, dockerfile = “Dockerfile”, froms = [“//src/containers/ubuntu”], )
The docker_build rule is now:
docker_build = rule( implementation = _docker_build, attrs = { “dockerfile”: attr.label( allow_single_file = True, mandatory = True, ), **“froms”: attr.label_list(),** “_docker_tool”: attr.label( executable = True, cfg = “host”, allow_files = True, default = Label(“//rules:docker”), ), }, outputs = { “imagesha”: “sha” }, )
This has added froms which is a list of other Bazel docker builds that are bases for this container.
I was hoping that just having the reference to its base ubuntu container would mean that when I changed the ubuntu Dockerfile it would rebuild the ruby container. This is not working, I am blocked. Now I am leaving to enjoy some Friday drinks.
9:45pm — I had an idea
Just putting a reference to a rule doesn’t automatically rebuild it. Maybe if you take the output of the rule and put it as input to the next rule that would work.
This worked! Here is the new _docker_build function:
def _docker_build(ctx): name = ctx.attr.name folder = ctx.file.dockerfile.dirname`` froms = [f.image_sha for f in ctx.attr.froms]`` ctx.actions.run( executable = ctx.executable._docker_tool, inputs = ctx.files.dockerfile + froms, arguments = [ ctx.outputs.imagesha.path, “build”, “-q”, “-t”, name + “:bazel”, “-f”, ctx.file.dockerfile.path, folder ], outputs = [ctx.outputs.imagesha], )`` return struct( image_name = name, image_sha = ctx.outputs.imagesha, )
The change here is that I output the image_sha file and use that as input to its child containers docker build call (even though it is not used). Bazel then must detect a change in that file and then rebuild the container.
This has the nice benefit of only rebuilding the depended on images if they actually are rebuilt. For example, adding a comment to the ubuntu Dockerfile will not change the sha, so not rebuild the base images.
This is the last feature necessary to build a tree of docker images so a successful spike into Bazel.
10:30pm — Pretty Graph
As a final little project I wanted to display the dependency graph with:
bazel query — noimplicit_deps ‘deps(//src/containers/ruby-2.6)’ — output graph > graph.in
This outputted a dot file which after some styling and dot -Tpng -Gdpi=300 -o graph.png graph.in resulted in:
End
The code is available at https://github.com/grahamjenson/bazel-docker-tree
I am not sure this was a helpful blog, I just wanted to try something new and document my learning experience with Bazel. I think it has worked though I am not sure how useful it would be.
I would like to spend more time with Bazel, trying to get pushing/caching working and integrating more with the existing rules. That is enough for today though :)