While contributing to the esp8266/Arduino project, Ivan posted a link to a test build using Appveyor. After a bit of research, I learned that there is a whole slew of companies offering cloud-based build servers in this space called continuous integration. More impressive is that it is free for open-source projects with most companies in this space.
When I first started writing software it was in basic and assembler on a Commodore 64. When writing small programs on fixed-configuration systems like that, the development cycle was reasonably quick, with even "large" assembler programs taking seconds to build. Deployment testing was simple as well; if it worked on my C64 it would work on everyone else's. As computers got bigger and more complex, so did the development cycle. While working on large projects at places like Nortel, full system builds could take several hours or even days. Being able get quick feedback on small code changes is very important to software development productivity. The availability of low-cost, on-demand services like Amazon Web Services has enabled companies in the CI space to offer build services with minimal infrastructure investment.
Who's Who
The esp8266/Arduino project uses Appveyor for Windows builds and Travis for Linux builds. Other CI companies offering Linux-based build services include Drone.io, Snap CI, and my favorite, Codeship. All of these companies offer some level of service for free to open-source developers, so I decided to try all four of the Linux-based CI services.
For my work with embedded systems, I have been writing build scripts for avr-gcc, which I intend to extend to building a gcc cross-compiler for the xtensa lx106 CPU on the esp8266. Full builds of binutils, avr-gcc, and avr-libc take a few hours on an Intel Core i5, so getting a working build was a slow process. Having a large build like this also turned out to be helpful differentiating between the different CI services.
One thing all the CI services have in common is they make it easy to set up an account and try their service if you already have a github account. With Google Code shutting down, everyone in open-source development should have a github account already. While the CI services support a number of different languages, I was only concerned with C++ using the gcc compiler. All the servers had gcc >= 4.4 and typical tools like autoconf, flex, and bison.
Travis
Travis was the first CI service I tried, and it turned out to be one of the more complicated. In order to get Travis to set up for your build (set environment variables, download dependencies), you need to make a .travis.yml file in the root of your repository. The format is similar to a shell script, so it wasn't too hard to figure out. After a bit of experimenting I was able to get a build started.
From some of the posts I read online, I was concerned whether the build would complete in the allowed time of 50 minutes. The problem I ended up having was not build time but build output. After 4MB of log output, Travis terminates the build. If my build failed I wanted to see where in the build process the problem occurred. So I turned off minor log output from things like tar extracting dependency libraries, but I still hit the 4MB limit.
Another problem you might have with large amounts of log output relates to how your browser handles it. Firefox started freezing on me when I tried to view a 4MB log file, but Chrome was OK.
Drone.io
Drone's service was easier to setup, allowing a shell script to be written in their web interface, which would be run to start your build. Drone has a limit of only 15 minutes on free builds, which turned out to be the showstopper with their service.
Codeship
I almost missed the boat on Codeship since they don't even list C/C++ in their supported languages. I guess a gcc installation is taken for granted for Linux-based CI. Codeship, like Drone, allows you to write a build script in their web interface. Unlike Travis and Drone, no sudo access is availabe on the build servers, so installing updated packages is not possible. Since the servers have a reasonably recent version of gcc (4.8.2-19ubuntu1) and gnu tools, this was not a problem for me. Their build servers (running on AWS) are nice and fast, with a full build, using make -j4, taking about 13 minutes.
Codeship doesn't seem to have any limit on build time, though they do have a 10MB log output limit. Fortunately that is just a limit of what is displayed in the web inteface, and the build does not stop. The most impressive thing about Codeship's service is that they give you ssh access to a build server instance for debugging! Clicking on Open SSH Debug Session gives you the IP and port to ssh into, assuming you've already updated your account with your ssh public key.
On the debug server, your code is already copied to the "clone" directory. The servers are running Ubuntu 14.04.2 LTS, The servers seem to have un-throttled GigE ports, as a download of gcc 4.9.2 clocks in at 24MB/s or 200mbps. With the debug server I was able to manually run my build, review config.log files, and copy files using scp to my computer for later review.
Snap CI
One way Snap CI is different than the other services, is that their servers run CentOS instead of Ubuntu. I started using RedHat before Debian and Ubuntu existed, and never had a good reason to leave rpm-based distributions, so I like the CentOS support. The version of gcc on their servers (4.4.7) is rather old, but they enable sudo so you can upgrade that with a newer RPM. They also have a limited shell interface called snap-shell. It's not full ssh access like Codeship, but it does make it easy to check the environment by running things like "gcc --version".
Snap CI also uses AWS servers, and build times were very similar to Codeship. If your builds require downloading a lot of prerequisite files, Snap may take a bit longer than Codeship though, as gcc 4.9.2 took about twice as long to download on Snap.
Conclusion
CI services eliminate the time and cost of setting up and maintaining build servers. They simplify software testing by having a clean server instantiated for each build. No more broken or incompatible builds because someone installed a custom library version on the build server that normal users don't have on their machine. I closed my Drone.io account, and will probably close my Travis account too. I'll keep using both Codeship and Snap, to be sure the software I'm working on can build on both Ubunto and Centos. If I continue to support programs like picoboot-avrdude that builds under Linux and MinGW, I'll also try out Appveyor.
No comments:
Post a Comment