---
title: "QMake hackery: Dependencies & external preprocessing"
date: "2011-11-13"
author: Chris Hodapp
tags:
- Project
- Technobabble
---
* TODO: Put the code here into a Gist?

[Qt Creator](http://qt-project.org/wiki/Category:Tools::QtCreator) is
a favorite IDE of mine for when I have to deal with miserably large
C++ projects. At my job I ported a build in Visual Studio of one such
large project over to Qt Creator so that builds and development could
be done on OS X and Linux, and in the process, learned a good deal
about [QMake](http://doc.qt.nokia.com/latest/qmake-manual.html) and
how to make it do some unexpected things.

While I find Qt Creator to be a vastly cleaner, lighter IDE than
Visual Studio, and find QMake to be a far more straightforward build
system for the majority of things than Visual Studio's build system,
some things the build needed were very tricky to set up in QMake. The
two main shortcomings I ran into were:

* Managing dependencies between projects, as building the application
  in question involved building 40-50 separate subprojects as
  libraries, many of which depended on each other.
* Having external build events, as the application also had to call an
  external tool (no, not `moc`, this is different) to generate some
  source files and headers from a series of templates.

QMake, as it happens, has some commands that actually make the project
files Turing-complete, albeit in a rather ugly way. The `eval`
command is the main source of this, and I made heavy use of it.

First is the dependency management system. It's a little large, but I'm including it inline here.

```bash
# This file is meant to be included in from other project files, but it needs
# a particular context:
# (1) Make sure that the variable TEMPLATE is set to: subdirs, lib, or app.
#     Your project file really should be doing this anyway.
# (2) Set DEPENDS to a list of dependencies that must be linked in.
# (3) Set DEPENDS_NOLINK to a list of dependencies from which headers are
#     needed, but which are not linked in. (Doesn't matter for 'subdirs'
#     template)
# (4) Make sure BASEDIR is set.
#
# This script may modify SUBDIRS, INCLUDEPATH, and LIBS. It should always add,
# not replace.
# It will halt execution if BASEDIR or TEMPLATE are not set, or if DEPENDS or
# DEPENDS_NOLINK reference something not defined in the table.
#
# Order does matter in DEPENDS for the "subdirs" template. Items which come
# first should satisfy dependencies for items that come later.
# You'll often see:
# include ($$(BASEDIR)/qmakeDefault.pri)
# which includes this file automatically.
#
# -CMH 2011-06

# ----------------------------------------------------------------------------
# Messages and sanity checks
# ----------------------------------------------------------------------------
message("Included Dependencies.pro!")
message("Dependencies: " $$DEPENDS)
message("Dependencies (INCLUDEPATH only): " $$DEPENDS_NOLINK)
#message("TEMPLATE is: " $$TEMPLATE)

isEmpty(BASEDIR) {
    error("BASEDIR variable is empty here. Make sure it is set!")
}
isEmpty(TEMPLATE) {
    error("TEMPLATE variable is empty here. Make sure it is set!")
}

# ----------------------------------------------------------------------------
# Table of project locations
# ----------------------------------------------------------------------------

# Some common locations, here only to shorten descriptions in the _PROJ table.
_PROJECT1   = $$BASEDIR/SomeProject
_PROJECT2   = $$BASEDIR/SomeOtherProject
_DEPENDENCY = $$BASEDIR/SomeDependency

# Table of project file locations
# (Include paths are also generated based off of these)
_PROJ.FooLib               = $$_PROJECT1/Libs/FooLib
_PROJ.BarLib               = $$_PROJECT1/Libs/BarLib
_PROJ.OtherStuff           = $$_PROJECT2/Libs/BarLib
_PROJ.MoreStuff            = $$_PROJECT2/Libs/BarLib
_PROJ.ExternalLib          = $$BASEDIR/SomeLibrary

# ----------------------------------------------------------------------------
# Iterate over dependencies and update variables, as appropriate for the given
# template type
# ----------------------------------------------------------------------------

# _valid is a flag telling whether TEMPLATE has matched anything yet
_valid = false

contains(TEMPLATE, "subdirs") {
    for(dependency, DEPENDS) {
        # Look for an item like: _PROJ.(dependency)

        # Disclaimer: I wrote this and it works. I have no idea why precisely
        # why it works. However, I repeat the pattern several times.
        eval(_dep = $$"_PROJ.$${dependency}")
        isEmpty(_dep) {
            error("Unknown dependency " $${dependency} "!")
        }

        # If that looks okay, then update SUBDIRS.
        eval(SUBDIRS += $$"_PROJ.$${dependency}")
    }
    message("Setting SUBDIRS=" $$SUBDIRS)
    _valid = true
}

contains(TEMPLATE, "app") | contains(TEMPLATE, "lib") {
    # Loop over every dependency listed in DEPENDS.
    for(dependency, DEPENDS) {
        # Look for an item like: _PROJ.(dependency)
        eval(_dep = $$"_PROJ.$${dependency}")
        isEmpty(_dep) {
            error("Unknown dependency " $${dependency} "!")
        }

        # If that looks okay, then update both INCLUDEPATH and LIBS.
        eval(INCLUDEPATH += $$"_PROJ.$${dependency}"/include)
        eval(LIBS += -l$${dependency}$${LIBSUFFIX})
    }
    for(dependency, DEPENDS_NOLINK) {
        # Look for an item like: _PROJ.(dependency)
        eval(_dep = $$"_PROJ.$${dependency}")
        isEmpty(_dep) {
            error("Unknown dependency " $${dependency} "!")
        }

        # If that looks okay, then update INCLUDEPATH.
        eval(INCLUDEPATH += $$"_PROJ.$${dependency}"/include)
    }
    #message("Setting INCLUDEPATH=" $$INCLUDEPATH)
    #message("Setting LIBS=" $$LIBS)
    _valid = true
}

# If no template type has matched, throw an error.
contains(_valid, "false") {
    error("Don't recognize template type: " $${TEMPLATE})
}
```

It's been sanitized heavily to remove all sorts of details from the
huge project it was taken from. Mostly, you need to add your dependent
projects into the "Table of Project Locations" section, and perhaps
make another file that set up the necessary variables mentioned at the
top. Then set the `DEPENDS` variable to a list of project names, and
then include this QMake file from all of your individual projects (it
may be necessary to include it pretty close to the top of the file).

In general, in this large application, each sub-project had two
project files:

* One with `TEMPLATE = lib` (a few were `app` instead as well). This
  is the project file that is included in as a dependency from any
  project that has `TEMPLATE = subdirs`, and this project file makes
  use of the QMake monstrosity above to set up the include and library
  paths for any dependencies.
* One with `TEMPLATE = subdirs`. The same QMake monstrosity is used
  here to include in the project files (of the sort in #1) of
  dependencies so that they are built in the first place, and permit
  you to build the sub-project standalone if needed.

...and both are needed if you want to be able to build sub-project
independently and without making to take care of dependencies
individually.

The next project down below sort of shows the use of that QMake
monstrosity above, though in a semi-useless sanitized form. Its
purpose is to show another system, but I'll explain that below it.

```bash
QT -= gui
QT -= core
TEMPLATE = lib

## Include our qmake defaults
DEPENDS = FooLib BarLib
include ($$(BASEDIR)/qmakeDefault.pri)

TARGET = Project$${LIBSUFFIX}
LIBS += -llua5.1 -lrt -lLua$${LIBSUFFIX}
DEFINES += PROJECT_EXPORTS

INCLUDEPATH += /usr/include/lua5.1 
    ./include

HEADERS += include/SomeHeader.h 
    include/SomeOtherHeader.h

SOURCES += source/SomeClass.cpp 
    source/SomeOtherClass.cpp

# The rest of this is done with custom build steps:
GENERATOR_INPUTS = templates/TemplateFile.ext 
    templates/OtherTemplate.ext

gen.input = GENERATOR_INPUTS
gen.commands = $${DESTDIR}/generator -i $${QMAKE_FILE_IN}
# -s source$(InputName).cpp -h include$(InputName).h

# Set the destination of the source and header files.
SOURCE_DIR = "source/"
HEADER_DIR = "include/"
# What prefix and suffix to replace with paths and .h.cpp, respectively.
TEMPLATE_PREFIX = "external/"
TEMPLATE_EXTN = ".ext"

#
# Warning: Here be black magic.
#
# We need to use QMAKE_EXTRA_COMPILERS but its functionality does not give us
# an easy way to explicitly specify the names of multiple output files with a
# single QMAKE_EXTRA_COMPILERS entry. So, we get around this by making one
# entry for each input template (the .ext files).
# The part where this gets tricky is that each entry requires a unique
# variable name, so we must create these variables dynamically, which would
# be impossible in QMake ordinarily since it does only a single eval pass.
# Luckily, QMake has an eval(...) command which explicitly performs an eval
# pass on a string. We repeatedly use constructs like this:
#    $$CONTENTS = "Some string data"
#    $$VARNAME = "STRING"
#    eval($$VARNAME = $$CONTENTS)
# These let us dynamically define variables. For sanity, I've tried to use a
# suffix of _VARNAME on any variable which contains the name of another
# variable.
#

# Iterate over every filename in GENERATOR_INPUTS
for(templatefile, GENERATOR_INPUTS) {
    # Generate the name of the header file.
    H1 = $$replace(templatefile, $$TEMPLATE_PREFIX, $$HEADER_DIR)
    HEADER = $$replace(H1, $$TEMPLATE_EXTN, ".h")
    # Generate the name of the source file.
    S1 = $$replace(templatefile, $TEMPLATE_PREFIX, $$SOURCE_DIR)
    SOURCE = $$replace(S1, $$TEMPLATE_EXTN, ".cpp")
    # Generate unique variable name to populate &amp; pass to QMAKE_EXTRA_COMPILERS
    QEC_VARNAME = $$replace(templatefile, ".", "")
    QEC_VARNAME = $$replace(QEC_VARNAME, "/", "")
    VARNAME = $$replace(QEC_VARNAME, "\", "")
    # Append _INPUT to generate another variable name for the input filename
    INPUT_VARNAME = $${QEC_VARNAME}_INPUT
    eval($${INPUT_VARNAME} = $$templatefile)

    # Now generate an entry to pass to QMAKE_EXTRA_COMPILERS.
    eval($${VARNAME}.commands = $${DESTDIR}/generator -i ${QMAKE_FILE_IN} -s ${QMAKE_FILE_OUT} -h $${HEADER})
    eval($${VARNAME}.name = $$VARNAME)
    # ACHTUNG! The 'input' field is the _variable name_ which contains the
    # input filename, not the filename itself. If you put in a filename or
    # either of those variables don't exist, this will fail, silently, and
    # all attempts at diagnosis will lead you nowhere.
    eval($${VARNAME}.input = $${INPUT_VARNAME})
    eval($${VARNAME}.output = $${SOURCE})
    eval($${VARNAME}.variable_out = SOURCES)

    # Now tell QMake to actually do this step we meticulously built.
    eval(QMAKE_EXTRA_COMPILERS += $$VARNAME)
    # Also add our header files. I doubt it's really necessary, but here it is.
    HEADERS += $${HEADER}
}
```

This one uses a bit more black magic. The entire `GENERATOR_INPUTS`
list is a set of files that are inputs to an external program that is
called to generate some code, which then must be built with the rest
of the project. This uses undocumented QMake features, and a couple
kludges to generate some things dynamically (i.e. the filenames of the
generated code) from a variable-length list. I highly recommend
avoiding it. However, it does work.

These two links proved indispensable in the creation of this:

[QMake Variable Reference](http://qt-project.org/doc/qt-4.8/qmake-variable-reference.html)

[Undocumented qmake](http://www.qtcentre.org/wiki/index.php?title=Undocumented_qmake)
