Make

by Mads Hartmann - 20 Aug 2016

I've used Make in a range of projects – some large, some tiny – and I've been happy with the results. Over time, through colleagues and books, I've picked up a few tricks and found a nice way to think about your Makefile(s). However, it's only about a year ago that I really got Make under my skin and felt I understood the power of Make and what problems it solves nicely. I'm putting down my thoughts here in a way that I believe past-Mads would've found helpful; it might have helped him appreciate Make a little faster.

If you want to get into some deeper aspects of Make after reading this I highly recommend the book Managing Projcts with GNU Make 3rd edition. Furthermore you can always dig into a specific topic by consulting the Make documentation – I will link to the specific sections where relevant but I won't cover all of the features that Make has to offer so feel free to go explore.

1 Why Make is Still Useful Today

Make is a programming language agnostic tool that provides a simple set of abstractions for describing the relationships between the source files in your project and the artifacts you wish to produce. This makes it a very useful tool for automating the development work-flow as well as the deployment pipeline of a given project. Ideally getting started on a project should be a simple matter of executing a single shell command which installs the required tools, configure the project and build the required artifacts; nobody likes having to go through a 14 steps README in order to get a project up and running – the same goes for deploying your code to production.

However, having been created in 1977, Make can at times feel extremely archaic. It's fair to assume that it must have been surpassed by more modern tools. You are probably using a more recent programming language specific build tool such as SBT for Scala, Leinigen for Clojure, Rebar3 for Erlang, Webpack for web assets, etc. These tools solve the task of turning your source files into an artifact that you can run and deploy. For this specific task I don't recommend that you use Make, but rather that you should use Make to automate the use of these tools.

In the next section I'll introduce Make in the way that I like to think of it – as a programming language.

2 Make as a Programming Language

I like to think of Make as a declarative string-oriented functional programming language with very good support for executing shell commands. It's a programming language that happens to be very good at creating build systems 😉

You write your Make program in a file called a Makefile. The purpose of your Makefile is to describe a dependency graph of your build artifacts and source files such that Make knows how and when to build a specific artifact. The dependency graph is specified using rules – this is what I think of as the declarative part of Make. In order to specify these rules concisely Make has a string-oriented functional programming language that is mostly used to compute the value of variables that can then be used when defining rules.

Once you have your Makefile you invoke Make like so

make my_file

where my_file is the name of the file you want Make to produce. Based on your Makefile Make computes and traverses the dependency graph of my_file in order to figure how and if it should produce my_file.

Let's start by having a look at the language used to specify the rules.

2.1 The Rule Language

In your Makefile you declare your rules using the following syntax.

target: prerequisite1 ... prerequisiteN
	command1
	...
	commandN

The target is the name of the file you want to produce. The prerequisites are the files that are required to produce the taget. The commands (also known as the recipe) are the shell commands Make should run in order to produce the target.

Image you have a project where you need to convert a Markdown file to HTML. The following rule would achieve this (assuming you have a program called marked on your system).

index.html: index.md
	marked --input index.md --output index.html

This tells Make that it can produce a file named index.html if a file named index.md exists and that it can produce it using the shell command marked --input index.md --output index.html. Additionally Make knows that if index.md changes it should re-produce the index.html file. An important note is that each command in the recipe (is this case there's just one) should be prefixed with a tab – it's one of the many archaic aspects of Make.

A rule doesn't necessarily have to contain a recipe, the following is a perfectly valid rule.

build: index.html about-me/index.html posts/hello-world/index.html

This simply tells Make that there's a target called build and that it should consider it satisfied if all the prerequisites are satisfied. Targets like these makes it possible to provide a nice interface to your Makefile – you can now invoke make build instead of make index.html about-me/index.html posts/hello-world/index.html.

2.1.1 Implicit Rules

With the rules you've seen so far you can imagine that your Makefile gets a bit long as the number of Markdown files increase as you have to explicitly describe the relationship between each Markdown and HTML file.

build: index.html about-me/index.html posts/hello-world/index.html

index.html: index.md
	marked --input index.md --output index.html

about-me/index.html: about-me/index.md
	marked --input about-me/index.md --output about-me/index.html

posts/hello-world/index.html: posts/hello-world/index.md
	marked --input posts/hello-world/index.md --output posts/hello-world/index.html

For cases like these you can define a pattern rule (one of many types of implicit rule that Make has) instead and get a nice and succinct Makefile.

build: index.html about-me/index.html posts/hello-world/index.html

%.html: %.md
	marked --input $< --output $@

The pattern rule %.html: %.md tells Make that it can generate an .html file from a .md file of the same name using the body of the rule. The symbols $< and $@ are automatic variables that expand to the name of the left-most prerequisite and the name of the target respectively. More on variables later.

2.1.2 Auto-generating prerequisites

There are cases where managing the lists of prerequisites of a target can become cumbersome and error-prone. A solution to this problem is to generate prerequisites automatically for some targets. This can be achieved using the include directive.

The include directive tells Make to read another Makefile – an example would be include foobar.mk. Before reading the Makefile foobar.mk it checks if there exists a rule that can produce foobar.mk. If such a rule exists it will reproduce foobar.mk before including.

This makes it possible to automatically generate a Makefile that declares prerequisites and then import it. A tool that produces such a Makefile has historically been called make-depend.

This is a little out of scope for this blog post so head over to Generating Prerequisites Automatically to read more about this; I only mention it here as it's something you are likely to need in a more complex build systems.

2.2 String-oriented functional programming

So the dependency graph is specified through rules. Let's have a look at the language features that Make has that enables you to write nice concise Makefiles.

2.2.1 Variables

There are two kinds of variables in Make – simple and recursive – they differ in the way they're expanded when referenced. There are four different operators that can be used when assigning values to variables.

:= assignment operator In the example python := python$(PYTHON_VERSION) the := assignment operator will set the value of python to python2.7 given the value of PYTHON_VERSION is set to 2.7. That is, the right-hand side is evaluated during the assignment of the variable.

= assignment operator In the example python = python$(PYTHON_VERSION) the = assignment operator won't evaluate the right-hand side but rather set the value of python to python$(PYTHON_VERSION) and the final expansion of python depends on the value of the variable PYTHON_VERSION when the variable python is referenced. You can think of this as lazy evaluation as you might be used to from using the lazy keyword in Scala (or if you've used Haskell). Some good advice is to only use = if you really have to. Otherwise stick to the other operators.

?= conditional variable assignment operator In the example PYTHON_VERSION ?= 2.7 the conditional assignment operator will set the value of PYTHON_VERSION to 2.7 unless PYTHON_VERSION is already defined as an environment variable.

+= append operator In the example foo += bar baz Make will append bar baz to the current value of foo. For simple variables this is simply a shorthand for foo := $(foo) bar baz but for recursive variables the operator will still do the right thing without crashing in an infinite evaluation of foo.

There's a convention that words in variables should be separated_by_underscores and that variables that a user might want to customize should be written using UPPERCASE letters.

When you reference a variable it is expanded in-place – In that sense a Make variable is similar to variables as you know them from templating languages such as jinja2 or ejs. The syntax for referencing a variable is $(variable_name).

In adittion to the variables you define Make has a set of variables that are called automatic variables. These are variables that change value based on the rule that is currently being executed.

2.2.2 Functions

Make contains a bunch of built-in functions and it's worth reading through them as it will make your Makefile much more concise, especially the File Name Functions.

The syntax for calling a functions is $(function-name arg1, ..., argN).

It is possbile to define your own functions. A user-defined function is really just a variable that is defined in such a way that it's intended to be used with the built-in call function. So when you're calling a user-defined functions the pattern is $(call variable-name, param1, ..., paramN)

The following shows how to define a function. The arguments to the function are available through the variables $1, $2, etc.

# $(call print-rule, variable, extra)
#   Used to decorate the output before printing it to stdout.
define print-rule
	@echo "[$(shell date +%H:%M:%S)] $(strip $1): $(strip $2)"
endef

There's a convention that user-defined functions use lowercase-words separated-by-dashes.

3 Example – A Static Website

Let's see how all of this fits together. Let's create a Makefile for a simple statically generated web-site based on Markdown files. You can find this little example on Github. The project has the following structure.

├── Makefile
└── site
    ├── make.md
    └── tools
        ├── osx
        │   └── homebrew.md
        └── xargs.md

For each Markdown file we want to generate a HTML file with an appropriate path and put it in a folder named _build. For the example above the result of building the site should be the following.

_build
├── make
│   └── index.html
└── tools
    ├── osx
    │   └── homebrew
    │       └── index.html
    └── xargs
        └── index.html

This can be achieved with the following Makefile.

### User-changable variables.

BUILD_DIR := _build

### Variables

markdown_sources := \
	$(shell find site -name "*.md")

html_pages := \
	$(foreach f, $(markdown_sources), \
		$(subst site/,$(BUILD_DIR)/, \
			$(dir $(f)))$(basename $(notdir $(f)))/index.html)

# Alternative way to achieve the same thing as html_pages
html_pages_alternative := \
	$(patsubst site/%.md,$(BUILD_DIR)/%/index.html,$(markdown_sources))

### Targets

build: setup $(html_pages)
setup: node_modules/.installed
clean: ; rm -rf $(BUILD_DIR)
distclean: ; rm -rf $(BUILD_DIR) node_modules
print-%: ; @echo $* is $($*)

### Rules

$(BUILD_DIR)/%/index.html: site/%.md
	@mkdir -p $(dir $@)
	node_modules/.bin/marked --input $< --output $@

node_modules/.installed: requirements.txt
	rm -rf node_modules
	npm install $(shell cat $<)
	touch $@

Let's go through the Makefile step by step. First we define a single variable that the user might be interested in changing (hence the capitalized letters).

### User-changable variables.

BUILD_DIR := _build

We then go on to define a variable that contains the path of all of the Markdown files in site and the sub-folders of site. This is achieved by using the very handy shell function which allows you to execute a shell command and use the result in your Makefile. An alternative way to find all Markdown files could've been to use the wildcard built-in function.

markdown_sources := \
	$(shell find site -name "*.md")

Before we move on let's make sure we got the use of shell right by inspecting the variable using the print-% target – the print-% target is something I copy-paste into all of my Makefiles as it's extremely useful for debugging Makefiles.

make print-markdown_sources
markdown_sources is site/make.md site/tools/osx/homebrew.md site/tools/xargs.md

Great, so we've successfully managed to capture the the filenames of the Markdown files we want to process. Next up we use the markdown_sources variable to create a new variable that contains the filenames of the HTML pages we want to generate.

html_pages := \
	$(foreach f, $(markdown_sources), \
		$(subst site/,$(BUILD_DIR)/, \
			$(dir $(f)))$(basename $(notdir $(f)))/index.html)

Here we're using some of the most commonly used built-in in functions. We're using the text-function subst to replace site/ with _build/. We're using the file name functions dir, notdir and basename to get the directory part of a path, the filename part of a path and finally the filename part of a path without the extension. We're using the the foreach function to perform the transformation on each path separately.

There's another very useful text-function named patsubst which takes a pattern instead of a string to perform substitution. Here's how to use it to provide an alternative implementation of the variable html_pages.

html_pages_alternative := \
	$(patsubst site/%.md,$(BUILD_DIR)/%/index.html,$(markdown_sources))

In this case I think the use of patsubst yields the clearest result but it depends on the situation so it's nice to keep both approaches in your toolbox. Let's have a look at the value of both variables.

make print-html_pages ; make print-html_pages_alternative
html_pages is _build/make/index.html _build/tools/osx/homebrew/index.html _build/tools/xargs/index.html
html_pages_alternative is _build/make/index.html _build/tools/osx/homebrew/index.html _build/tools/xargs/index.html

With the variables in order we go on to define the targets that we intend the user to call; this is the interface of the Makefile.

build: setup $(html_pages)
setup: node_modules/.installed
clean: ; rm -rf $(BUILD_DIR)
distclean: ; rm -rf $(BUILD_DIR) node_modules
print-%: ; @echo $* is $($*)

The clean, distclean and print-% are using a convenient syntax for defining the recipe of a rule on the same line as the rule itself.

Notice that build depends on setup as well as $(html_pages) – this means that if you were to run make build before having run make setup Make will ensure the the setup target is satisfied before attempting to satisfy the targets in the html_pages variable; it's a small convenience this that means you have to remember fewer targets as a developer.

Finally let's have a look at the rules.

$(BUILD_DIR)/%/index.html: site/%.md
	@mkdir -p $(dir $@)
	node_modules/.bin/marked --input $< --output $@

node_modules/.installed: requirements.txt
	rm -rf node_modules
	npm install $(shell cat $<)

The first rule $(BUILD_DIR)/%/index.html: site/%.md describes how to generate a HTML file from a Markdown file using a pattern rule. The recipe of the rule makes sure that the directory exists using mkdir -p before generating the HTML files using the NPM package marked. The prefix @ of @mkdir tells Make not to output the command when executing it. Notice we're using two automatic variables $@ and $<: The first is the name of the target, the second is the name of the left-most prerequisite – in our case there is only one and it's the name of the Markdown file that should be used to generate the HTML file.

The last rule node_modules/.installed: requirements.txt describe how to install the NPM package(s) that we're using. The packages are enumerated in a file named requirements.txt. First we remove the previously installed node_modules, then we install the packages and finally touch the node_modules/.installed file. The rule is using a common pattern where we introduce an empty file (in this case node_modules/.installed) which represents the successful execution of the rules recipe. Here's the Makefile manual section that describes this pattern.

Here's a quick demonstration. We're running make distclean build to first get a completely clean workspace and then build to show how Make installs the NPM package before attempting to generate the output.

make distclean build
rm -rf _build node_modules
rm -rf node_modules
npm install marked
/Users/hartmann/dev/mads-hartmann.github.com/examples/markdown-site
└── marked@0.3.6 

touch node_modules/.installed
node_modules/.bin/marked --input site/make.md --output _build/make/index.html
node_modules/.bin/marked --input site/tools/osx/homebrew.md --output _build/tools/osx/homebrew/index.html
node_modules/.bin/marked --input site/tools/xargs.md --output _build/tools/xargs/index.html

If we were to run make build again without changing any files then Make is smart enough to know that nothing needs to be re-built.

make build
make: Nothing to be done for `build'.

If we change a file Make is smart enough to only re-build that single file.

touch site/make.md ; make build
node_modules/.bin/marked --input site/make.md --output _build/make/index.html

And that's it.