I want to run an application on my CoreOS clusters that uses hostnames to communicate between machines. This is a problem, because out of the box CoreOS machines cannot resolve hostnames of other machines in the cluster. So, I wrote a small fleet service that manages the /etc/hosts files on all the machines so they can correctly resolve each others hostnames. In this post I will briefly describe that service.
The Hosts Service
The hosts.service:
# hosts.service
[Unit]
Description=Hosts Manager
After=etcd2.service
[Service]
EnvironmentFile=/etc/environment
Restart=always
ExecStartPre=-/usr/bin/etcdctl mkdir /hosts
ExecStart=/bin/sh -c 'while true; do etcdctl watch --recursive /hosts; \
sleep 1;\
echo "127.0.0.1 localhost" > /etc/hosts; \
for i in $(etcdctl ls /hosts); do \
echo $(etcdctl get $i) $(echo $i | cut -c 8-); \
done >> /etc/hosts; \
done'
ExecStartPost=/usr/bin/etcdctl set /hosts/%H $PRIVATEIP
ExecStopPost=/usr/bin/etcdctl rm /hosts/%H
[X-Fleet]
Global=true
This service uses etcdctl so must start after etcd2
in the /etc/environment file the PRIVATEIP is defined as the address the hostname should be resolved to. This can be set in the cloud-config with:
write_files: - path: /etc/environment content: | PRIVATEIP=$private_ipv4Restart=always to always restart this service
Before the service starts (ExecStartPre=-) it will create the etcd directory hosts (etcdctl mkdir /hosts). Note: the =- means that if the directory creation fails (if it already exists) it will not stop the service
ExecStart= will start an infinite loop (while true) that waits for changes in the hosts folder (etcdctl watch — recursive /hosts). When a change happens it will:
wait (sleep 1) in case there is a burst of changes all at once
overwrite the exists /etc/hosts file with localhost information
loop over all entries in the /hosts directory (for i in $(etcdctl ls /hosts)) and append the value of the entry (etcdctl get $i) and the path (minus the /hosts bit) (echo $i | cut -c 8-) to the /etc/hosts file
After the service has started (ExecStartPost) it will then register its hostname (%H) and address ($PRIVATEIP) in the /hosts directory. This change will be detected by the service, which will rewrite the /etc/hosts file with itself in it.
When the service stops it will remove itself from the /hosts directory, which will cause other machines to update their /etc/hosts file.
Global=true is there so that this service will automatically run on new machines to the cluster. This means that the /etc/hosts file on all machines in the cluster will always be up-to-date.
For AWS users
For AWS users you may want to add the lines:
ExecStartPost=/bin/sh -c "etcdctl set /hosts/$(hostname | awk -F. '{print $1}') $PRIVATEIP"
ExecStopPost=/bin/sh -c "etcdctl rm /hosts/$(hostname | awk -F. '{print $1}') $PRIVATEIP"
This is because calling hostname may return a host looking like node1.us-east-1.compute.internal, and some applications require only node1.