Die Hard Make Habits
 by Adrian Neagu, in Editorials - Sat, Sep 1st 2007 00:00 PDT

The make build tool was (and still is) very influential in the sphere of software development tools. Its influence is so powerful that even bad aspects of its design survive in the next generation of build tools. Many generations of developers grew up in the school of make. Like the frog in the slowly heating bowl, they got used to its quirks to the point of not feeling the pain anymore. But they shouldn't be too quick to conclude that make's way is the one and only way. The punishment is to miss the opportunity for significant improvement.

The confusion between actions and targets

Targets are the entities that are acted upon (updated, deleted, etc.), and actions are the transformation you apply to them (update, delete, etc.). Usually, the targets are files or group of files.

The make tool introduced so-called phony targets. For example, "make clean" or "make touch" don't build a dependency tree, they just perform some action on some files. This is unlike "make my_exe", which first constructs a dependency tree and then performs some action on the nodes in this tree. The make tool puts both the names of the targets to be built and the name of the actions in the same namespace. This is ok for amateur software projects, but it is a serious limitation for real products.

To start with, there is the trivial problem of the name clashes. "clean", "depends", "test", etc. are frequent names for "targets" and also a pain in the neck when you want to make an executable named "clean", "depends", "test", etc. If you think that it's easy to just be careful and not give such names to what you want to build, you are simply wrong. Think about a code base with 10,000 C files and 100 contributors and how you would go about making sure that no file will be named clean.c, test.c, etc. (the "etc." here really means an open-ended set). The opposite has to be watched, too: there can be no phony target with a name that happens to be the name of some file in the dependency tree. This is technically possible, but it is not worth the trouble. It is possible to separate targets and actions by, for example, "make action=clean". It would have been even better if the tool had that already implemented for you, as in "make --clean" (because in the alternative "make action=clean" you still have some careful implementation to make in your makefiles).

Besides the name clashes, there is a more subtle and more important problem with all phony targets. The problem is that it is very difficult to have the action of phony targets performed on the same files as the non-phony targets. Is it important to act on the same files? Well, again, it is not that important in very small projects. Let's consider again the example of "make clean". If this removes all binary files with a file pattern, it is probably good enough. Even removing the entire directory with the build output (with "rm -R") might be ok. There are some (but not many) projects for which building from scratch takes a whole day on a capable computing farm.

More important than wasting build time, there may be serious consistency problems. Assume that, in some location in your code base, you can build two targets that don't have an exact set of files. It may be "make target_A" and "make target_B", or it may be "make target flavor=1st" and "make target flavor=2nd". Now, just what is "make clean" supposed to do? Remove the files associated with target A, or with target B, or with both? Again, in small C/C++ projects, this is no serious problem (no flavors, no targets with a variable number of files, etc.), but the same can't be said of larger projects (perhaps using something completely different from C/C++ compilation). Contrast this with the situation in which you could say "make target_A --clean" and "make target_B --clean". There, the tool can implement laser sharp cleaning. As a developer, you would only have to get the dependency tree right. Once it's correct, it's correct for all actions -- update, clean, etc. That would be the end of files overlooked in "clean".

Despite the fact that these problems have been known since the early 1990s, we keep seeing new build tools with the same confusion between targets and actions on targets. For example, Ant has only phony targets, rake has both file targets and non-file targets, etc.

Commit by default instead of print-out by default

The commandline of make is ill designed. It is probably the worst commandline among all tools that have ever been used by more than one person, yet its style is mimicked today by almost all want-to-be build tools.

A first problem, quite trivial, is the fact that the make command by itself, without any argument, commits something. Moreover, it commits something that is impossible to revert. Why do you care? Because if something unexpected happens, you will want to roll back and debug or just run again in a more verbose mode. There's no such possibility with make, and that's just to save typing 2-3 keys (like " -b").

When run without arguments, the majority of commandline tools in this world tell you something: "What is this?", "How do I use it?", etc. They don't do something irreversible. Does this sound like good design to you? Of course, you can do that with make also (as with "make action=update" and having "make" just print out which final target file will be built), but how many make-based build systems do you know that actually do it?

As a side effect, this "commit by default" scares away whoever may want to try a partial build. You may descend into a lower-level source directory and run "make" there, but if it builds more than you expect, it's too late. This aspect, coupled with some other shortcomings, discourages users of make from "starting small" when trying something new.

Passing parameters by choosing the shell's current directory

The most far-reaching bad habit of the make commandline is the fact that too much information is transmitted to the tool by choosing a directory from which to run it.

First, let's get agreement on the basic facts. You run a commandline in one location in the code base, and it builds something. You run the exact same commandline with the same shell environment in another location, and it builds something else. Some information is passed through the name of the current directory, and it is an important piece of information, not just a marginal one like, for example, the desired verbosity level.

What are the semantics of the current directory?

The first question is also the most difficult to answer: Just what exactly is passed through the name of the current directory? Think about how would you describe this in a limited space, as in a ten-line usage message.

One may say that the answer is easy: "The current directory serves only one purpose, fetching a Makefile, which tells make what else to do." That is so incomplete that we can simply say that it is wrong. Most importantly, the current directory still dictates a lot of things inside that Makefile. The same Makefile in another location may do something else (or just fail to do anything). Secondly, I can use the commandline argument -f <path_to_makefile> to fetch it from elsewhere (and some make-based build systems use that at lower levels if you run "make" at the top of a build tree).

The fact is that the current directory may mean almost all (what to build, from what, and which flavor) and may mean not much (for example, with "what" and "how" taken from the shell environment). That's already bad. It means that my make-based build system may be very different from yours. It also makes this basically impossible to document in a generic way (by the build tool authors, not by the build description authors). Contrast this with a commandline like make --source_root=<dir> --output_root=<dir>. That can be documented easily. source_root and output_root may have default values if you are keen on saving typing (even with the current directory as the default). That syntax would make my build system differ from yours less in total and in less fundamental ways.

Mapping parameters to directories

In most make-based build systems, the current directory tells what to build and where to put it, but some systems take this a bit further and let you choose which flavor you build by firing make in a different subdirectory. Consider as example the following directory layout:

dir_my_exe
	i86_mswin32
		english
			Makefile
		french
			Makefile
		german
			Makefile
		Makefile_level1
	i86_linux2
		english
			Makefile
		french
			Makefile
		german
			Makefile
		Makefile_level1
	inc
	src
	Makefile_common

where the content of the files in the inner directories is the following:

Makefile:
LANGUAGE = ENGLISH
include ../Makefile_level1

Makefile_level1:
PLATFORM = I86_MSWIN32
include ../Makefile_common

This describes, in fact, two variants (let's call them "platform" and "language") with, respectively, two and three allowed values. You can issue "make" in the end leaf locations, and you get a flavor of "my_exe" built. This may look extreme, but many systems are fundamentally doing just this kind of unfolding of parameters, including the good old GNU Build System (the ubiquitous one based on autoconf and GNU make).

There are many issues with this kind of approach. How do you decide what you keep as a commandline argument and what you specify through the current directory? What effort is needed to introduce a new allowed value for the language? Or a new value for the platform? Or a new variant? How would you go about documenting the available variants and the allowed values for each of them? You may place cross-linked Readme files next to the Makefiles, but how many codebases you know that have that?

Contrast this with the commandline "make platform=<plat> language=<lang>". Such a commandline makes it easy to document what variants are there and what values are allowed. It doesn't introduce any arbitrary order among variants (which you will have to learn and remember), and it is probably easier to maintain during the lifetime of the codebase.

Sometimes this possibility is implemented as well, so that the lines:

LANGUAGE ?= ENGLISH
PLATFORM ?= I86_MSWIN32

are put in Makefile_common, along with some tricks so that the build result goes to the same location as it would if you fired "make" in a leaf directory. But why bother when you can just use commandline arguments instead of the current directory information from the beginning of your build description?

Building from the binary directory or from the source directory

The user community quickly learned the benefit of having separate root directories for the sources and for the build output. Let's call them the source directory and the binary directory, respectively.

There are many good and bad reasons to separate. One bad reason, in the case of make-based systems, is the fact that make's timestamp checks are not reliable enough and the clean is not very precise (so a manual complete cleaning is needed at times, and that's most practical with "rm -R" on the binary directory). Whatever the reasons, the fact is that you have a source directory structure and a binary directory structure that are somewhat similar. Usually, the binary directory tree is "broader" in order to be able to store all derived files in several flavors without filename clashes.

Here's the tricky question: Where do you put your makefiles? There are two opposed styles:

Close to sources

Makefiles are placed in the source directory. For any built file, the path into the binary directory is computed somewhere inside the Makefile from the path of the source file.

Close to binaries

Makefiles are placed in the binary directory. The path to the sources is computed somehow from the current directory, perhaps with the help of given arguments.

Mixed solutions are also frequently used. For example, you start "make" somewhere in the binary directory (you specify the output path through the shell's current directory) and you point it to a Makefile "close to the sources" (you specify the location of the sources by explicitly giving the location of the Makefile).

Do we really need all these possibilities? It's true that a good build system must be able to accommodate almost any organization of sources, and it must let you design any sensible binary directory structure. But how exactly does the use of the current directory of the shell help with that? What sensible directory structure is made impossible by a commandline syntax like "make --source_root=<dir> --output_root=<dir>"? The use of the current directory by make is not flexibility from the make tool, it is just useless variability in the many make-based build systems.

One last word about the myth of build descriptions "close to sources". In large real-life C/C++ projects, it is not uncommon to have hundreds of directories containing source files. What does it mean to be "close" to 500 different locations? Are you going to spread your build description over 500 small chunks? That's a possibility, but then each build description will contain very little information, probably just a very few trivial lines (often the real information is then the full path of the directory where the description is placed). Smarter make-based build systems moved away from that model at the same time they moved away from the recursive make model. That means that the entire build description is in one place, probably close to the root source directory tree (and possibly "far" from the deepest directory with sources).

I hope that you are convinced by now that using the current directory instead of commandline arguments is not a good idea. If you're still not convinced, try to count how many commandline tools you know that use the current directory to pass crucial information. Any compilers? Linkers? Debuggers? Others?

Inability to list sources

One of the first actions that one would implement on a dependency tree is the ability to print it out. Several flavors are interesting to print: all genuine input, all build output, all implicit dependencies, etc. Personally, before I would implement "incremental build", "forced build", "clean", etc., I would implement "print", and I would test whether my tree is as expected in less trivial cases.

How many systems have something like "make action=listsources"? Even worse, a depressing number of new build tools are proposed without such a simple, basic feature. What hope then for more elaborate, yet useful actions like "clean all files that are not up-to-date"?

No target platform concept

Make doesn't go a long way toward modelling important concepts like the C/C++ tool chain, build platforms, target platforms, optimization levels, etc. This is presented as flexibility; you are free to model them as you see fit, with any set of shell variables, any piece of shared Makefile, etc. Ok, let's buy it as freedom. But there is one place where this freedom hurts badly: The lack of separation between the build platform and the target platform. Even proposing some weak separation (like a few conventional macros or macro prefixes) would have been better than nothing at all. It may well be that the majority of the compilations in this world are not cross-compilations, but that thought will not alleviate the pain of the embedded software engineer. It will only make him curse harder.

When designing a build system, based on make or not, it is not smart to cut yourself out of the community of embedded porting engineers. The porting engineer is usually more closely involved with the build than the average mainstream platform developer. An important part of the software porting process to embedded platforms is the change/adaptation of the build description, and it had better not be a complete rewrite of the build description (to another build system).

It is sad that most build tools since "make" have shown no courtesy toward embedded software engineers. Moreover, the most recent build tools only make it worse. How many of these recent build tools have an option like "tgtplatform=<plat>" to choose what to build for?

Some recent build tools expect to configure everything on the fly when you start a build, and they are proud to present this as progress. They say, "Look, no configuration needed!" They actually detect installed tool chains by some investigation of the build platform they are running on. While this is a desirable feature when not cross-compiling, it will most likely be a pain for the porting engineer. His tool chain will not be detected. He will have to do some manual configuration (which is acceptable), but he will also have to implement something that allows him to switch tool chains (because any porting involves reference builds for the build platform itself). He now has more work than with his previous, "dumber" build system.

With a commandline like "make --tgtplatform=<plat>", the make tool had the opportunity to introduce the world to the fact that cross-compilation exists, but it missed it. When will the chance come again?

In the end

I've listed only a few shortcomings of "make" that tend to survive longer than "make" itself. There are others, but the point is not to count them, not even to get a "total weight" of them. The point is to get in the habit of questioning whether some way of doing something is still the most appropriate way for the situation at hand, to compare alternatives and not choose by inertia. You may discover a new world that feels even cozier than your previous one.

Author's bio:

Adrian Neagu has spent the recent years as a release engineer with Nuance. Before that, he worked as an R&D engineer with Lernout and Hauspie in Brussels, Belgium. He received a PhD in speech perception and recognition in 1998 in Grenoble, France.

T-Shirts and Fame!

We're eager to find people interested in writing articles on software-related topics. We're flexible on length, style, and topic, so long as you know what you're talking about and back up your opinions with facts. Anyone who writes an article gets a t-shirt from ThinkGeek in addition to 15 minutes of fame. If you think you'd like to try your hand at it, let jeff.covey@freshmeat.net know what you'd like to write about.

[Comments are disabled]

Referenced categories Topic :: Software Development :: Build Tools

Referenced projects Apache Ant - A Java-based build tool. Autoconf - A package of M4 macros to produce scripts to automatically configure sourcecode. GNU make - Controls the generation of executables and other compile-related tasks. Regression mAKEr. - Simple modular application for data investigation.

 Comments

[»] Example
by Jan Engelhardt - Sep 28th 2007 01:40:37

> where the content of the files in the inner directories is the following: [...] This describes, in fact, two variants (let's call them platform" and "language") with, respectively, two and three allowed values. You can issue "make" in the end leaf locations, and you get a flavor of "my_exe" built. This may look extreme, but many systems are fundamentally doing just this kind of unfolding of parameters, including the good old GNU Build System (the ubiquitous one based on autoconf and GNU make).

Here is how that would look in autoconf/automake. Where you have NxM makefiles already, I have one - no includes.
configure.ac: http://pastebin.ca/718290
Makefile.am: http://pastebin.ca/718291
and what you would do on the shell: http://pastebin.ca/718292
I am not trying to convert you, but at least convince you that autotools does the right thing for those who use it. I'd be delighted to discuss and implement with you how your project could/would look like if it were to use autotools. Then you can still decide.

[reply] [top]

[»] Re: Example
by Adrian Neagu - Sep 28th 2007 17:15:14

First, I would like to thank you for your nice
offer, for your patience and for make me discover
http://pastebin.ca (looks like neat tool).

Second, I would like to make clear that my rant is
not primarly against the autotools. It primarly
against the make tool and against some more modern
replacements that (despite the fact that they
were not constrained by any backward compatibility)
missed some oportunities. The autotools
are not in that category (not a replacement
but a patch to make, not free of backward
compatibility...).

Now on to the details:


> Where you have NxM % makefiles already,

> I have one - no includes.

You mean you still have N*M but
you don't feel the pain because they are
all "easy" to generate from some unique genuine
input. You are sadly wrong. The thing you
overlook is the fact that not everybody
is in the use case of "generate once
and forget ever after". To me, this approach
is "been there, done that, went away".
We ended rather quickly with a build to keep
the makefiles up to date and a very tricky one
(difficult to detect automatically
when exactly the makefiles need
to be regenerated).

One way to see that this doesn't fly is to catch
a Symbian developer and ask him how many times
he forgot to run the a.bat (that the top script
that regenerates the makefiles in the build system
in the Symbian SDK).



> configure.ac: http://pastebin.ca/718290

> Makefile.am: http://pastebin.ca/718291

> and what you would do on the shell:

> http://pastebin.ca/718292

Thank you for spending the time to set up
the example. Reading it reinforces my older
conviction that autoconf is rather OK (despite
its shellish, old-looking syntax) and automake is
much less OK. Not because of the syntax but because of
the same "soup of global variables" as the bare make.
No structure, no attempt to separate
things that should be separate. AM_CFLAGS
can be abused exactly how CFLAGS can be abused.



> at least convince you that autotools does

> the right thing for those who use it.

Right. I'm convinced. The key wording here is
"for those who use it". Meaning that people
are generally smart and they choose and stick
with the tool that maches their needs.
The autotools made a quatum leap
for software distribution as sources on Unix
systems. We should be grateful.



> I'd be delighted to discuss and

> implement with you how your project

> could/would look like if it were to use

> autotools.

Thank you. You know, my company (Nuance Communications)
is hiring right now in my division :-). But you should
be available to relocate to Germany or Belgium...



> I am not trying to convert you,

> ...

> Then you can still decide.

I do appreciate that you don't try to convert me.
There are to many pasionate flames around build tools.
I will give you a bit of a background.
You'll see why I'm difficult to convert and also
what kind of needs I have from a build system.

My company is all about closed source products.
I work in embedded and we see a dozen new target
platforms every year (basically all hardware
manufacturers come with their home-baked GCC
toolchain or their Win CE favor). The build machines are
90% of the time Windows XP (and we don't really
have any choice in that). As far as the build
is concerned, the 13 years history of our products
(speech processing engines) goes like this:

1. A few years of manually maintained makefiles
and Microsoft project files (dsp/dsw)

2. Quite some years of generated makefiles and dsp files

3. Finally 3 years on SCons and continuing

The last solution before SCons was not autotools
or similar (my preferred in that
category is CMAke). No source package and no
executable thing in it. Yet, in our home-grown solution
the makefiles were also generated from higher-level
build descriptions. Python scripts were generating
both the makefiles and the dsp files from XML files.
We never distributed our builds except
to outsourcing partners. But what really killed
the autotools for us was the fact that they use GNU make.
GNU make is a catastrophy when you use a version-control
system with dynamic views (sources can change to older in
time or newer but still older than your compiled objects).

The solution currently in use is based on SCons.
More than one part of our company migrated
independently to SCons. The big argument is
the fact that the command line is part of
the MD5 signature of the built files.
Very reliable builds are the immediate consequence.
Another argument was that it's a radical
solution to the "out-of-date makefile" accidents.
There are also disadvantages to SCons
(see my article "Make alternatives"
or the story of KDE trying SCons and abandoning it).

All in all, the lesson to take away is that
the active development of C code is a usage scenario
very different in needs from distribution
of software as sources.



> each `make` invocation. "--build" is

> even a standardized option. While

> "--enable-debug" is not, it is still

> much more common ...

Yes, --build is a very good thing. Although
the target platform as concept is a bit more than just
selecting the backend of the C compiler.

--enable-debug is poor. This is what could be named
"build optimization" variant/flavor. Any serious
development needs more than just dbg/rls.
Take a look at the Jam build tool setup.
Those people did a fair job for analyzing
the interesting variants for the C/C++ builds.
SCons is as barebone as make in this respect.



> (you would have to add it to all

> flavors of Make; GNU, Solaris, BSD... -

> tough luck).

Aah, good that you remind it! Although in the end
we didn't shoot anymore to support more than 2 make tools
(Opus make and GNU make), that was still such a pain
in the neck (mainly because different syntax for 'if').
Since Python was already present on our
build machines (for generating the makefiles, a.o.),
when we moved to SCons we just eliminated one tool
from the requirements (actually several, all make clones
in one shoot). Everybody felt this as a relief...



> 'make' should do "is this target newer? rebuild",

If this is the core business of make, then this
summarizes well why make is bad. Because
it is so poor at answering reliably the question
"is this target out of date?". I'll stop here,
many classic articles are available on this subject.
SCons is one of the best for this. Both in what
it offers out of the box and in how it allows
you to program your own "is it out-of-date?".




[reply] [top]

---

[»] alternatives already exist
by PerlChild - Sep 2nd 2007 07:36:39

Oddly enough, rake, from ruby, comes from a language with no compilation needs, but does quite a bit many of the things you describe, I'd suggest you take a look. I've not seen "platforms" yet, but if you set a flag inside the Rakefile, call it platform, and use it inside your targets

The dependency parser came about, as a I understand, part of a desire to reimplement make in ruby, and part because of the needs of projects, like ruby on rails, where one has to upload/synchronise in order, to make automatic script generation easier, etc...

Seems compilation kept the dependency tool to itself for too long, now others are claiming it back.

[reply] [top]

[»] Re: alternatives already exist
by Adrian Neagu - Sep 27th 2007 15:57:17


> rake, ... I'd suggest you take a look.

I did try out rake. And I prefer SCons. Mainly because more things are available out of the box
for almost similar expression power in the build scripts. But I will not be able to build a very strong case against rake.
(see also my article "Make alternatives").

[reply] [top]

---

[»] Make has problems for sure...
by David F. Skoll - Sep 1st 2007 19:29:56

There are many problems with "make". The SCons folks (http://www.scons.org/) and Cons author (http://www.dsmit.com/cons/) have enumerated many of them.

However, make is unlikely to be replaced any time soon for the same reason that vi is unlikely to go away: Although vi is a crappy idiosyncratic editor that violates just about every user-interface principle known to man, it is ubiquitous and "good enough". Make is similarly ubiquitous and "good enough", so there's no real incentive to replace it. Replacing make means having to ship your build system with your source, and that's really annoying.

[reply] [top]

[»] Re: Make has problems for sure...
by Adrian Neagu - Sep 27th 2007 16:26:30



> The SCons folks... enumerated many of them.

Sure. Everybody, lists them.
By the way, my preferred build tool is SCons.
(please see also my articles on Freshmeat
"What's wrong with make" and "Make alternatives").



> However, make is unlikely to be replaced

> any time soon

Very much true (and not that sad, IMO).
I have other examples like this: the C language,
the Bourne shell, etc.
But you have to agree with me that this is not
a reason to not try to do better. The same way
that vi didn't prevent Eclipse to happen.

"good enough" is a relative notion.
Good enough for that project (say, 20 source files,
1 target platform) may not be good enough for my project
(say, 2000 source files, 15 target platforms).
And the pain may be enough to grant the time
to investigate other tools
(may be not to write a new tool, since the choice
is already so large).



> Replacing make

> means having to ship your build system

> with your source, and that's really

> annoying.

"ship your build system",
that's one assumption too much.
Replacing make (with something
not yet very popular) means shipping something
executable with it but not necessarly all
the build tool. It may be some shell script
(depending what the receiving end will
do with the sources, may be "build once and forget").

This problem you mention is called
bootstrapping. There are many different possible
approaches to bootstrasping. One is to make
some tool ubiquetous enough (that's "make" approach,
Perl, Python, Ruby come close as well).
Others are to ship a shell script with very low
requirements (that's autotools approach).
And there are others.

Also note that bootstrapping is a concern only
for the use case of distributing software as sources,
which is not the only use case of a build tool!
There are still developers not working on open source
software to run on Unix machines.


[reply] [top]

---

[»] Re: Make has problems for sure...
by X-Nc - Oct 21st 2007 19:18:05


> However, make is unlikely to be replaced

> any time soon for the same reason that

> vi is unlikely to go away: Although vi

> is a crappy idiosyncratic editor that

> violates just about every user-interface

> principle known to man, it is ubiquitous

> and "good enough".


Fei, I say. Fei to you and your horse. vi(m) is as user friendly as $PICK_YOUR_FAVORITE_EDITOR, even on a Mac. Idiosyncratic? Yes. Difficult to use? No more so than MS-Windows or KDE or GNOME. Apple used to publish a book "Macintosh Human Interface Guidelines" that really did a great job of breaking down the computer/user interface. Everything else is just as idiosyncratic as vi(m).

And, yes, I am a vim aficionado.

--
If I actually _could_ spell I'd have spelled it right in the first place.

[reply] [top]

---

[»] Thank you
by David Necas (Yeti) - Sep 1st 2007 09:07:21

for a good laugh.

Every bloody Unix command does something when run -- or complains you did not give it the things to act upon. If some crazy program asks first, people make aliases that enable to just run it (and conversely, if you want it to ask, you are free to make an alias that does so). If you run random programs (e.g. `halt') without arguments to see what they do, you deserve the burns.

The GNU build system (autoconf, automake and other stuff) enables building software on all queer platforms with their b0rken implementations of everything (including make) without the need to install a build system. It has issues and people question this goal too, but you have to reconsider who to blame for the lowest common denominator being so low.

To make --source_root=<dir> --output_root=<dir> work, every single command used in the rules must be made to support such mode of operation (unless it's a simple alias to cd to one of the directories). For any custom rule the burden is on the users.

And finally, make is a general dependency processor. People use it for all kinds of tasks, even to resolve the order of system service startups. How all the binary directory and target platform stuff and --clean target_B maps to this? Sometimes all targets are phony and it is a good thing.

The point about name clash between files and phony targets is valid (though not the `solution' which just recreates the same clash between command line options and phony targets), but generally, although make can be blamed of various things, the fact you'd be better served by something more narrow-minded does not belong among them.

[reply] [top]

[»] Re: Thank you
by Jan Engelhardt - Sep 17th 2007 06:59:05


> To make --source_root=<dir> --output_root=<dir> work, every single command used in the rules must be made to support such mode of operation (unless it's a simple alias to cd to one of the directories). For any custom rule the burden is on the users.

In fact, autotools shine here. You name the rules, the build system does the rest(*). All that is needed to sort the wheat from the chaff is to (mkdir obj; cd obj; ../configure;) and voilą, all the non-source parts get into their own obj directory. Which of course, you can also pre-poulate (see MXS case below) if you have to guts to deal with the timestamping.

(*)It really helps: I would rather want to type the 9K of important rules/CFLAGS rather than 130K of multi-platform workarounds.
$ l obj/GNUmakefile
-rw-r--r-- 1 jengelh users 139266 Sep 14 22:47 obj/GNUmakefile
$ l GNUmakefile.am
-rw-r--r-- 1 jengelh users 9485 Sep 14 22:47 GNUmakefile.am

[reply] [top]

---

[»] Re: Thank you
by Adrian Neagu - Sep 27th 2007 18:05:02


> All that is needed to sort the

> wheat from the chaff is to (mkdir obj;

> cd obj; ../configure;) and voilą, all

> the non-source parts get into their own

> obj directory.

You see, this (mkdir obj; cd obj; ../configure;) is
in my eyes a problem with the design of the tool.
You say it is a solution and I have to admit that.
Indeed, it is a solution to
the problem "How do I separate?".

But the point I try to make in this article is that
no thinking went into "How should I separate?",
"What are the interesting variants?", "What are the most
useful ways to separate?". How did it come to mind to call
that dir "obj". Why not "output" or something else?
And I can create next to it "obj2" and "obj3"
and my colleague developer "obj4" and "obj4/second_try"
and at the end of the day we will have a hard time
"to sort the wheat from the chaff".

Sure it is easy to do
(mkdir dir; cd dir; back_to_root_of_sources/configure;).
But what does it buy me this flexibility?
It is misplaced. It would have been more interesting
to have (make --platform=m64 --type=dbg)
to decide for a location of the output.
Of course, I can still do that with classic make like this
(make PLATFORM=m64 TYPE=dbg) but you will name those
make macros differently than mine. And then we have
variability for no good reason.

By contrast (make --platform=m64 --type=dbg) could
mean a $PLATFORM and $BUILDTYPE pre-defined by make,
the same for the vast majority of Makefiles.
And without blocking in any way somebody with special needs
to invent $PLATFORM2.

Do you see any shortcoming of that compared to the total
chaos (sorry, freedom) of today? Regards.


[reply] [top]

---

[»] Re: Thank you
by Jan Engelhardt - Sep 28th 2007 01:12:46


> How did it come to mind to call that dir "obj". Why not "output" or something else? And I can create next to it "obj2" and "obj3" and my colleague developer "obj4" and "obj4/second_try" and at the end of the day we will have a hard time "to sort the wheat from the chaff". It would have been more interesting to have (make --platform=m64 --type=dbg)

Yes, you can have multiple obj dirs in various locations, owned by different people, with a name you choose, with the flags you like. `cd ~/proj; /usr/src/proj/configure --build=x86_64-unknown-linux --enable-debug` for example, whose flags you only need to enter once instead of each `make` invocation. "--build" is even a standardized option. While "--enable-debug" is not, it is still much more common than adding "--platform" or thelike to one flavor of 'make' (you would have to add it to all flavors of Make; GNU, Solaris, BSD... - tough luck). At the same, adding options like these to 'make' goes gainst the unix principle. 'make' should do "is this target newer? rebuild", and not fiddle with platform details.

[reply] [top]

---

[»] Re: Thank you
by Adrian Neagu - Sep 27th 2007 18:59:12


> Thank you for a good laugh.

You are welcome. Although I have a difficult time
to understand what can I learn from you laughing
at my needs, even assuming that my needs
are narrow minded.



> although make can be blamed of various

> things, the fact you'd be better served

> by something more narrow-minded does not

> belong among them.

Granted. I would have been better served
by a more narrow-minded tool (a long time ago,
before I moved to other build systems).
A lot of my criticism only applies to the tool
if it is used to build some files (I still believe
that's the most frequent use case) and there
was no word of caution in the article
that it doesn't apply to other uses.



> If you run random programs (e.g.

> `halt') without arguments to see what

> they do, you deserve the burns.

You need to work a bit on your attitude.
People do make mistakes. Place yourself for
a second in the place of a developer having
to go to his boss to admit a mistake.
Would you like your boss to think
on the line "you deserve the burns"?

Imagine yourself for a moment in a position
where you have to support 30 developers
building your product. Some are MS Windows
developers not even familiar with a command
line. Some are external customer that are
paying you. How will this "you deserve the burns"
attitude will help them?


> The GNU build system (autoconf, automake

> and other stuff) enables building

> software on all queer platforms with

> their b0rken implementations of

> everything (including make) without the

> need to install a build system. It has

> issues and people question this goal

> too, but you have to reconsider who to

> blame for the lowest common denominator

> being so low.

Really? It seems you need to open again
the autoconf book and read in the 1st chapter
what the developers of autoconf have to say
about make in general and gmake in particular.

One thing to remember is that
the GNU build system does not address
at all a few categories of limitations
of make. It addresses, as you mention,
portability. It also addresses
easy development of the build description.
It doesn't address build consistency,
build speed, build debugging and some
others which are also important
performance criteria for builds.
For example, make tool timestamp
heurestic is hurting badly the build
(except for the "build once forget ever
after" kind of casual builder).
You may get a larger view
on various requirements from a build
system reading here



> And finally, make is a general

> dependency processor. People use it for

> all kinds of tasks, even to resolve the

> order of system service startups.

> ...

> Sometimes all targets are

> phony and it is a good thing.
That is true. I saw experts systems
(for pattern recognition) implemented
with Makefile for GNU make (as I already
mentioned in other article on Freshmeat).
Although I would rather use Prolog
for implementing rules for an expert system,
I don't see anything bad with that use of make.



> How all the binary directory and target

> platform stuff and --clean target_B maps

> to this?
What's the problem? Suppose that make would
have an argument --source_dir= translated
into a pre-defined macro $SOURCE_DIR, how
would that prevent other uses of make?
Why cannot a Makefile ignore it, if
it doesn't have a use for it?



[reply] [top]

---

[»] make is the assembler of build tools
by Gianni Mariani - Sep 1st 2007 07:57:09

I agree with almost everything you say except that I tend to think of make as a build tool assembler. The MakeXS project (which kinda merged with Austria C++ but is very easily separable) does quite alot with GNU make as the basis.

Your first point is that targets and actions are the same. While you're right, make does not make much of a distinction between a target and an action, you're welcome to make some convention to reduce the issue. Actions are somewhat identifiable by the use of the .PHONY: target if you inspect the Makefile. After using them for many years, it appears the conventional targets "all", "install", "clean", "distclean" etc don't really pose a significant risk of confusion.

As for large projects, I have used MakeXS on moderate projects and it has a "pre-build" binary feature. MakeXS will allow you to build and then create a compressed binary of the result. On a regular "make all", if the pre built binary tar exists, it will simply extract the file avoiding long build time. If you want to build from scratch, simply clean the directory, remove the pre-built binary and make all.

As for the issue you raise regarding "make clean" and multiple targets, it depends very much on your source tree. Using MakeXS I advocate strongly on organizing the source tree in a very flat manner where each library or major target has it's own directory and performing a "make clean" in that directory will clean all intermediate files for that particular target. While MakeXS does not do this right now, I suspect that it would not be a big deal to provide a way to clean dependant targets as well.

The "commit by default" issue is I think yet another issue that can be "fixed" by the Makefile. In MakeXS, there are a large number of make variables that can be queried. An example, "make env/MXS_TARGETS" will cause the "all" targets to be printed (at least the all targets that MakeXS knows about). Also "make -n" does give you an idea of what 'gmake' is going to do without actually doing it.

As for scaring anyone away with a "partial build", I think this is a source tree organization issue, and not so much a fault of make. If you use the MakeXS preferred source tree organization, you're not going to have any issues with building more than you want to build.

The "current directory" issue is also one that MakeXS "fixed". MakeXS mandates that there be 2 files in every MakeXS folder. The first is the regular "Makefile" which must include the second, "Makefile.xsi" which are all identical. Makefile.xsi is used to locate the "root" directory and so you have a consistant "environment" across the entire source tree. The local Makefile is usually empty or contains "current directory" specific parameters.

Regardless, running "make" will read the current Makefile and update the first target in the Makefile. That's what it does. While MakeXS still uses the convention to place "all" as the first target, you're more than welcome to make the all target in the root directory.

I won't defend Autoconf. I've fixed many a build error with autoconf projects that was made very difficult by the nature of how autoconf works. It fails the "result of least surprise" rule too often for me to be "comfortable".

As for variants, MakeXS supports "debug/release", "m32/m64" and can be easily extended to support "architectures". A single instance of a source tree can accomodate multiple builds. Again, this is not an issue with make as the "assembler of build tools" but with the Makefiles that provide the policy.

I don't think I can agree that the build tool must accomodate any source tree organization. I think it's important to reduce the number of variables in source tree organization so that conventions can reduce the learning curve.

Resursive builds are not ideal and I think it is hard to create a make based system that does anything but a recusrive build. So this would be a limitation of make. However, as for build descriptions being close to the sources vs close to the root, I'll have to go with close to the sources. One of the most important features of a source tree is having a "separable" one where I can add or remove chunks of my source tree and all the "things" that go with the "chunks" including the build preferences go with them. This "feature" would mandate that the build preferences go with the chunk (directory).

As for listing sources, I think again this would be quite easy with a minor modification to MakeXS and hence not a make-as-an-assembler issue.

If I directed my engineers to assembler to write code today, I would be have some very strange looks. I don't know if it is appropriate for make to be the underlying build system, however it does have the advantage of being available on alot of platforms (gmake in particular which MakeXS depends on). Having said that, I think there is plenty of room for improvement and one of my back-of-the-mind projects is to clean up MakeXS and get away from some of the limitations of make, in the mean-time however, it builds just fine.

[reply] [top]

[»] Re: make is the assembler of build tools
by Adrian Neagu - Sep 27th 2007 17:29:57

I do understand that you gained a lot of experience
with gmake while making MakeXS, yet you missed some
background on gmake itself and also
you are a bit too confined to make world.

Please read the articles of John Graham Cumming
in Dr. Dobbs Journal. In one of these, January 2005,
he shows precisely how to easily implement
non-recursive builds with gmake.
I would also humbly recommend my
article
although it doesn't mention MakeXS.

You make a strong point that many issues
can be compensated in Makefiles.
That is correct. But the issue is not
if they can or can not.
The issue is that they MUST be compensated.
I must have a naming convention, I must
implement "print sources" and similar things.

I have to agree that this is less of an issue
if you have a tool "on top" of make
like MakeXS, CMake, QMake, autotools, etc.
But we have some many of these and so incompatible
between them because the make tool was
so poor to start with.
The very fact that you have MakeXS demonstrates
my need (and the fact that make failed to that
need in the first place): having more functionality
from the tool, only implement things that
are specific to my project, not dump info
kind of functionality.


> I don't think I can agree that the build

> tool must accomodate any source tree

> organization. I think it's important to

> reduce the number of variables in source

> tree organization so that conventions

> can reduce the learning curve.

Here you have a point and I tend
to agree. We can see the success that Maven has
in the Java community and it is rather
strict on the directory structure of the code base
(it has a "project template").
All for the good cause, the Java developers say.
But the opposite can be claimed as well. For example
if you are a tool provider and you don't want
to cut yourself from the market, then it's important
that your build tool supports whatever
organization of the code base.


> However, as for

> build descriptions being close to the

> sources vs close to the root, I'll have

> to go with close to the sources. One of

> the most important features of a source

> tree is having a "separable" one ...

Again, you have a good point.
It is especially true for version control
systems that label (or other manipulate)
one directory and everything below in one shot.
But I still dislike having a Makefile somewhere
just for the sake a getting its path
(why not directly the path of the source file
that I want to compile?).

Thank you for your time.


[reply] [top]

---