Mk the better Make

Back in the days when I was playing with MetaPRL, the project was built using OMake, a Make like build system dictated to MetaPRL project. OMake has very advanced recipe language that supports functional style programming, and novel features like able to handle complex dependency relation, custom defined action when build fails, subdirectory management, and using hashing function instead of time stamp to test if a file is changed. Although OMake needs OCaml tool chain to build and cannot be installed standalone, and sometimes it bugged because been installed multiple versions at same time, it is still a decent general purpose building system that I would like to have.

That was all before I actually learned and used Unix make, and later I discovered it is almost impossible to define anything close to OMake's abstraction using Make or GNUMake without littering the file system with touched empty files to track dependency. I struggled, but finally I gave up.

That was the situation until I meet Mk that is being used on Plan 9. Unlike OMake which has all sorts of fancy features and even maintains a database at the project root, Mk still relies on simple features like file time stamps, although this time you also have an attribute to specify custom file testing method.

The first interesting thing is Mk evaluates the mkfile in order, and create the dependency graph in the memory after expended variables, the recipes are however uninterpreted and will be directly passed to shell verbatim and using inherited shell variables to obtain information such like file names. During this process a shell of your choice would be used to interpret the things for meta programming, and you can switch them in between.

Unlike Make which failed at giving a reasonable semantic to a rule having multiple targets, Mk can capture the notion that a recipe could produce multiple targets that needs to be tracked very well. A rule without recipe only adds additional dependency relation, to complement meta rules. A rule without dependency describes one or multiple targets that would produce by running the recipe, and the dependency of each individual target can be added as additional rules.

The OMake hashing feature can be mimic by the YACC example in Mk man page, you still need to store an extra copy of the file, or at least the checksum, however that's also what OMake need to maintain. In case you don't understand why need hashing in addition to time stamp, that's because a file could be regenerated as a side effect, but the content could be unchanged, and detecting that can save building time.

The regexp metarule is such an obscure feature that no one has yet mentioned somehow the Unix port of Mk need the prerequisites to use double backslashes instead of one, due to different escaping rules in shell. The original Mk paper Hume did use single quote like '\1/\2.c' to prevent shell interpretation of backslashes. Here is another example.

  |$DESTDIR/([^/.]+)\.(png|mp3|svg|jpg):R: media/\\1.\\2
  |  cp $prereq $target

In my opinion this a useful feature for checking dependencies that spanning multiple directories.

Having multiple targets in one rule and able to declare them as virtual enables parallelized execution of "PHONY" targets, which would be a useful thing for automating testing. Here is an example that I use to validating DocBook XML.

  |%-valid:VQ: %.dbk
  |  echo "  JING" $stem
  |  $JING $SCHEMA $stem.dbk
  | 
  |validate:V: ${DOCBOOKS:%.dbk=%-valid}

When the times stamp of the target is identical to the prerequisites', Mk would try to update the target. This could unfortunately been the case if a fast command like copy has been used to update the target and the file system does not use high enough resolution for time stamps. If that happens to you, try to add a sleep command to extend the time. The Plan 9 sleep command also cannot do milliseconds level sleep and a float argument would be rounded so remember to use something like u sleep 0.1 to call the Unix version instead if you want finer control of it.

The custom compare command set by P can be a local shell script file, and it allows additional switches to be supplied. The target would be then appended to the command and the rest would be the dependencies. Mk would also require the dependencies to be present even though the compare script can ignore the dependencies. Also it is implied P must be the last of the attributes.