Multiple Mesos build configurations using Docker
I recently worked on a Mesos issue to improve build time. The idea was to use a tool that will help us remove unnecessary header inclusions.
One of the only tools available for this is Include-What-You-Use, it’s not a proclaimed 100% accurate tool but we had to give it a try since doing the same task manually can be a little repetitive.
Making a llvm clang plug-in work with Mesos was not obvious at first that’s why I had to try in an out of tree builds for Include-What-You-Use and multiple clang versions before finding the right combination.
To make this easier I begun creating a Mesos Docker image for developers that I recently pushed to their registry , there are Mesos Dockerfiles that can already be found in the registry but it was something new to learn, since I always built my Docker containers from command line with only base images and also there were any IWYU image available.
After several tests and managing to make work both in and out of tree builds, I chose to keep the out of tree clang-3.4 configuration because the image is much smaller, so here is the final Dockerfile for the base include what you use Docker image:
#llvm clang 3.4 + IWYU image FROM ubuntu:14.04 MAINTAINER Isabel Jimenez <[email protected]> RUN apt-get update -q #Install Dependencies RUN apt-get -qy install \ build-essential \ clang \ cmake \ libclang-dev \ libncurses-dev \ linux-libc-dev \ llvm-dev \ make \ subversion \ --no-install-recommends #Checkout IWYU and switch the banch clang_3.4 RUN mkdir include-what-you-use && svn co http://include-what-you-use.googlecode.com/svn/trunk/ source && cd source && svn switch ^/branches/clang_3.4 ENV CC clang ENV CXX clang++ ENV CMAKE_C_COMPILER clang ENV CMAKE_CXX_COMPILER clang++ #Configure IWYU RUN mkdir build && cd build && cmake -G "Unix Makefiles" -DLLVM_PATH=/usr/lib/llvm-3.4 ../source #Compile and install IWYU RUN cd build && make && make install
And here is the Mesos Dockerfile to build an image based on the IWYU one (just above):
#Mesos image from clang 3.4 FROM jimenez/iwyu:clang_3.4 MAINTAINER Isabel Jimenez <[email protected]> RUN apt-get update -q #Install Dependencies RUN apt-get -qy install \ autoconf \ automake \ ca-certificates \ git-core \ libcurl4-nss-dev \ libsasl2-dev \ libtool \ --no-install-recommends #Clone mesos repository RUN git clone http://git-wip-us.apache.org/repos/asf/mesos.git #Bootstrap RUN cd mesos && ./bootstrap #Configure RUN mkdir /mesos/build && cd mesos/build && ../configure --disable-java --disable-python --disable-optimize --without-included-zookeeper #Make RUN cd /mesos/build && make -j 16 #Master and slave ports EXPOSE 5050 5051 WORKDIR /mesos
Note: If you are interested on the in tree Dockerfile see my github.
Now I use this Mesos image for my builds and developments. This allows me to develop in parallel for different branches in my machine, while not interrupting the compilation of one another. Just keep in mind that Cgroups are used in different ways in Docker than in Mesos so when launching tests for Mesos you will have to disable some securities in Docker.
So I launch with interactive (-i) mode, having a proper terminal in the container (-t) and with all privileges so as cgroups tests can be done if needed:
docker run -it --privileged --name <branch> ijimenez/mesos-dev bash
This starts a bash process inside the container so I can interactively work inside of it.
If I need to work on another branch, I dettach from this container using
And attach to another container running the Mesos branch I want by doing:
docker attach <branch>
I do all my changes inside the containers, this not only allows me to stop worrying about messing between git branches but also if needed to just ship my container to another machine as is.
Side notes: To run tests think about disabling some Mesos tests not compatible with libcontainer (cf setns test), maybe one day the Docker team will be adding this support.
If you are using clang trunk you may need to disable debug in boost that produces errors (cf