Do you need a solution to scale your web services both vertically and horizontally, with load balancing and health checking? Consul with Docker and Nginx can help!
In this post I will describe how to use Consul (a service registry) and Nginx (with srv-router) running in Docker containers to load balance across multiple services.
Note: For OSX users boot2docker is required
Consul
Consul is a service registry that uses the DNS protocol to return a list of healthy services, in a random order (for load balancing).
Consul can be run as a docker container with:
docker run -it -h node \ -p 8500:8500 \ -p 8600:53/udp \ progrium/consul \ -server \ -bootstrap \ -advertise $DOCKER_IP
This runs Consul using the progrium/consul image, mapping to the ports 8500 to the web interface, and 8600 to the DNS interface. node is the given hostname, and it is being started as a server in bootstrap mode.
Finally, DOCKER_IP is the address of the docker host and is used for advertising. If you are using boot2docker export DOCKER_IP=boot2docker ip will assign it correctly.
A service can be registered with HTTP:
curl -XPUT \ $DOCKER_IP:8500/v1/agent/service/register \ -d '{ "ID": "simple_instance_1", "Name":"simple", "Port": 8000, "tags": ["tag"] }'
This service can be discovered using the DNS query tool dig:
dig @$DOCKER_IP -p 8600 \ tag.simple.service.consul
Nginx with srv-router
To have multiple instances of a service running on a single machine they have to be exposed on different ports. DNS A records only contain an IP address, where SRV records contain the port of the service.
srv-router uses
SRV records to correctly redirect an incoming request to the address and port. srv-router is a Docker container that runs an Nginx server modified with a Lua script to route requests using SRV records.
To run srv-router:
docker run -it -p 80:80 \ --net host \ -e "NS_IP=$DOCKER_IP" \ -e "NS_PORT=8600" \ -e "TARGET=simple.service.consul" \ -e "DOMAINS=$DOCKER_IP" \ vlipco/srv-router
This will start the srv-router on port 80.
— net host gives the container the same network interface as the Docker host to give it access to all other containers. The NS_IP and NS_PORT point towards the Consul server.
The srv-router when called will query Consul for home.simple.service.consul, then route to the address and port that is returned. This is the tags namespace in Consul, so each consul service must have the home tag.
Start to Finish
Using a simple Docker web-service described here, lets set up a “server” that uses load-balances across multiple services using Consul and srv-router.
First, lets run two servers on port 8001 and 8002:
docker run -it -p 8001:80 python/server docker run -it -p 8002:80 python/server
Now lets start Consul:
docker run -it -h node -p 8500:8500 -p 8600:53/udp progrium/consul -server -bootstrap -advertise $DOCKER_IP
Now srv-router:
docker run -it -p 80:80 --net host -e "NS_IP=$DOCKER_IP" -e "NS_PORT=8600" -e "TARGET=simple.service.consul" -e "DOMAINS=$DOCKER_IP" vlipco/srv-router
Finally, let register the services:
curl -XPUT \ $DOCKER_IP:8500/v1/agent/service/register \ -d '{ "ID": "simple_instance_1", "Name":"simple", "Port": 8001, "tags": ["home"] }'``curl -XPUT \ $DOCKER_IP:8500/v1/agent/service/register \ -d '{ "ID": "simple_instance_2", "Name":"simple", "Port": 8002, "tags": ["home"] }'
Lets check Consul has the services with:
dig @$DOCKER_IP -p 8600 \ home.simple.service.consul SRV
Finally, lets call the services with curl $DOCKER_IP.
What should happen is:
- srv-router gets the request and asks Consul for the SRV records for home.simple.service.consul
- Consul returns in a random order the two SRV records for simple_instance_1 on port 8001 and simple_instance_2 on port 8002.
- srv-router then routes the request to one of those services
- then the service handles the request and returns a response
Future Things and Stuff
In this post I did not describe how Consul can do health checking of the services, or how to streamline registering services with registrator. I will definitely go over these aspects in further posts.
Using Docker to contain and distribute services and Consul with srv-router to load-balance across them could greatly reduce growing pains of services. In conclusion, Docker is awesome.