I alluded to the goal in the in the teaser, but I’ll spend a few sentences clearing up the use-case a bit as all of this might be very Famly specific 😉
All of our services can be configured through environment variables and we use
these to configure how the services can find each other - in production we use
Kubernetes’ DNS based service discovery, e.g. we simply set
famlyapi-service and kubernetes takes takes of the rest. During development we
used Dockers DNS resolver to achieve the same thing. Here’s a bit from an
internal issue on our
famlydev repository explaining the need for the proxy.
The original vision was that everything would be running inside of Docker so routing could be handled entirely by Dockers DNS server but in practice this hasn’t resulted in a nice developer experience. Some examples: App’s compilation speed is roughly doubled if you run it inside of a container using Docker for Mac. If we were to run our Scala code in watch mode in docker we would end up compiling everything twice: once in the IDE and once in Docker. So we needed to be able to mix and match what we’re running in Docker, what we’re running locally and what we’re routing to staging. This is why we have a proxy in famlydev.
Alright, so we introduced a proxy that’s running on port
80 which routes to
the right services. However, this came with its own set of issues, namely
that if you wanted to switch out a service you’d had to jump through a few
hoops. Here’s another excerpt from the same issue.
If you want to route traffic to a local version of
famlyapi, for example, you have to update your
environment.envfile, update the
nginx.conffile and restart the relevant containers as updated environment variables aren’t propagated into running containers.
The whole purpose of
famlydev is that everything should just work™️ so forcing
people to remember these kinds of workflows just to route traffic to a locally
running service isn’t good enough.
Our current solution is to use DNS for service discovery together with NGINX upstream definitions to encode a specific precedence when routing traffic to a service. I’ll cover each of these techniques in the next two sections.
Service discovery through DNS
In order to use DNS for service discovery we have to make sure that all of the relevant host names can be resolved on the host as well as inside of Docker; luckily this was pretty straightforward.
On the host
To configure the DNS resolution on the host we simply add a bunch of entires
/etc/hosts file. Here’s what the famly part of my
127.0.0.1 famlyapi.famly.local 127.0.0.1 app.famly.local 127.0.0.1 api.famly.local 127.0.0.1 docs.famly.local 127.0.0.1 demo.famly.local 127.0.0.1 signin.famly.local
In docker we simply use Dockers network alias feature:
version: "2" services: proxy: image: nginx:1.13.3-alpine ports: - "80:80" volumes: - ../etc/proxy.conf:/etc/nginx/nginx.conf networks: default: aliases: - famlyapi.famly.local - api.famly.local - app.famly.local - docs.famly.local - demo.famly.local - signin.famly.local
Alright, so now
famlyapi.famly.local will route to
localhost if resolved
on the host and will route to NGINX if resolved inside of Docker. So now we just
need to make sure that the NGXIN proxy routes the traffic to the correct place.
NGINX upstream definitions
Now that all traffic goes to NGINX we have to make sure it sends the traffic to the right service. To do this we define a server block for each service and use upstream blocks to define a set of ports it should try to connect to a specific service on.
The trick here is that we’ve defined a precedence for each “configuration” of a
service. E.g. running
famlyapi locally on the host should receive traffic
famlyapi instance running inside of Docker (if one is running there
at all) - we use the port number to distinguish between the various
Here’s the full NGINX configuration - I’ve annotated it with plenty of comments to explain some of the oddities of the configuration.
I realize this might be incredibly Famly specific, but I hope you still got something out of it 😉