Multiple Mesos build configurations using Docker

I recently worked on a Mesos issue to improve build time. The idea was to use a tool that will help us remove unnecessary header inclusions.
One of the only tools available for this is Include-What-You-Use, it’s not a proclaimed 100% accurate tool but we had to give it a try since doing the same task manually can be a little repetitive.

Making a llvm clang plug-in work with Mesos was not obvious at first that’s why I had to try in an out of tree builds for Include-What-You-Use and multiple clang versions before finding the right combination.
To make this easier I begun creating a Mesos Docker image for developers that I recently pushed to their registry , there are Mesos Dockerfiles that can already be found in the registry but it was something new to learn, since I always built my Docker containers from command line with only base images and also there were any IWYU image available.

After several tests and managing to make work both in and out of tree builds, I chose to keep the out of tree clang-3.4 configuration because the image is much smaller, so here is the final Dockerfile for the base include what you use Docker image:

#llvm clang 3.4 + IWYU image

FROM ubuntu:14.04
MAINTAINER Isabel Jimenez <[email protected]>

RUN apt-get update -q

#Install Dependencies
RUN apt-get -qy install         \
    build-essential             \
    clang                       \
    cmake                       \
    libclang-dev                \
    libncurses-dev              \
    linux-libc-dev              \
    llvm-dev                    \
    make                        \
    subversion                  \
    --no-install-recommends

#Checkout IWYU and switch the banch clang_3.4
RUN mkdir include-what-you-use && svn co http://include-what-you-use.googlecode.com/svn/trunk/ source && cd source && svn switch ^/branches/clang_3.4

ENV CC clang
ENV CXX clang++
ENV CMAKE_C_COMPILER clang
ENV CMAKE_CXX_COMPILER clang++

#Configure IWYU
RUN mkdir build && cd build && cmake -G "Unix Makefiles" -DLLVM_PATH=/usr/lib/llvm-3.4 ../source

#Compile and install IWYU
RUN cd build && make && make install

And here is the Mesos Dockerfile to build an image based on the IWYU one (just above):

#Mesos image from clang 3.4

FROM jimenez/iwyu:clang_3.4
MAINTAINER Isabel Jimenez <[email protected]>

RUN apt-get update -q

#Install Dependencies
RUN apt-get -qy install         \
    autoconf                    \
    automake                    \
    ca-certificates             \
    git-core                    \
    libcurl4-nss-dev            \
    libsasl2-dev                \
    libtool                     \
    --no-install-recommends

#Clone mesos repository
RUN git clone http://git-wip-us.apache.org/repos/asf/mesos.git

#Bootstrap
RUN cd mesos && ./bootstrap

#Configure
RUN mkdir /mesos/build && cd mesos/build && ../configure --disable-java --disable-python --disable-optimize --without-included-zookeeper

#Make
RUN cd /mesos/build && make -j 16

#Master and slave ports 
EXPOSE 5050 5051

WORKDIR /mesos

Note: If you are interested on the in tree Dockerfile see my github.

Now I use this Mesos image for my builds and developments. This allows me to develop in parallel for different branches in my machine, while not interrupting the compilation of one another. Just keep in mind that Cgroups are used in different ways in Docker than in Mesos so when launching tests for Mesos you will have to disable some securities in Docker.

So I launch with interactive (-i) mode, having a proper terminal in the container (-t) and with all privileges so as cgroups tests can be done if needed:

docker run -it --privileged --name <branch> ijimenez/mesos-dev bash

This starts a bash process inside the container so I can interactively work inside of it.

If I need to work on another branch, I dettach from this container using <ctrl>-p <ctrl>-q

And attach to another container running the Mesos branch I want by doing:

docker attach <branch> 

I do all my changes inside the containers, this not only allows me to stop worrying about messing between git branches but also if needed to just ship my container to another machine as is.

Side notes: To run tests think about disabling some Mesos tests not compatible with libcontainer (cf setns test), maybe one day the Docker team will be adding this support.
If you are using clang trunk you may need to disable debug in boost that produces errors (cf CXXFLAGS=-DBOOST_CB_DISABLE_DEBUG)