I am constantly astounded at the number of shops out there that do not have source control not to mention even the most rudimentary build server or continuous integration environment. Or worse, there are developers that scoff at the idea of being ‘held back’ by things like this. More commonly management considers this a waste of time.

My Personal Experience

Fresh out of college I worked for a company that had a full-fledged continuous integration environment. In other words, whenever anyone checked in any project code ro artifacts for any project, a process somewhere would be monitoring source control and a build would get kicked off. That project, plus any dependent projects, will get checked out and built. If and only if it built without any errors, that build would then have a series of unit tests run against it. If any of the unit tests did not pass then the build was considered to be broken.

As I first joined the company, I was part of the team that took this one step further. Each night we would automatically take the latest successful build and install it in a clean environment, then load a bunch of test data in to it, and run a series of integrations tests against. These test were mostly automated UI tests, and generally verified use cases or user scenarios could be accomplished. Also, there was a series of tests performed against the database.

The results of these tests were automatically compiled together and displayed a SharePoint site, and were broken down into various graphs and what not, and visible to all the development teams, management, and product/project management to see.

The advantage of a system like this are pretty obvious.

  • Developers would know the minute the broke something, or forgot to check something in. As a result of the fast feedback loop, the developer at fault would generally know exactly what he did wrong because he was just working on it. Plus no one else ever wasted time trying to figure out why what the code they just pulled down out of source control would not compile.

  • Not wanting to break the build, developers would run all the unit tests prior to checking in, catching many bugs before it even reached source control.

  • Writing unit tests along with your code is very beneficial to distinguish what exactly it is you are trying to get your code to do. So if I made changes, then ran all the unit tests and found I broke one, reading the unit test that broke would more quickly reveal the intent of the code that was broken and to know when it was fixed.

  • Best of all the progress on the project was completely transparent. Any kind of problems were obviously from the beginning and could be addressed early and dealt with efficiently. This might be having someone else, who has experience with the problem take a look at it, or the project scope/time might need to be changed.

As this was my first job out of college, I was extremely naïve and thought that this was the norm for software development shops. Now that I have been in a dozen or more software development environments and found none having anything like this, most not even having a build server, and some do not even have source control, I can say without a doubt that continuous integration buys you so much in terms of time, money and quality.

##Objections

Let go over each objection to using it.

  • It takes more time to write, develop, and maintain unit tests. We have limited time on this project so we can worry about them later.

What no one considers is that when I write new code, and most people I know do the same, I write some sort of test harness anyway (an executable, a web page, a win32 app, something), so I can call the code I am writing over and over again while I am writing it. In this way I can have some level of assurance that the interface I am writing for is correct.

So, while it is true that it does take time, in many ways it is just a standardization and capturing of the code that is already being written and thrown away anyway.

  • It will take time to set up and maintain the build server and the machines will cost additional resources we do not have.

Assuming that you are using source control anyway, it is not complicated to set up a build server. I have used cruise control .net and another product called team city. Cruise control is free, team city is commercial product but is free for up to a certain number of developers. I’ve set both of these up to do a simple build on extremely underpowered machines and they run fine. Depending on the complexity of your build it should take a competent person less than a day to get initially up and running for a single build. With team city using their web interface, if you are using svn or TFS you can just point to your source control machine, give it credentials then it is literally just a drop down menu for a visual studio sln build, you can point to it and about as complicated as it gets.

If your build is very complex you will most likely need to write a build script. If you are on the Microsoft stack then you will most likely want to use MSbuild. Again, this is probably a good exercise to do anyway. Not only will it save tons of time every time a developer comes on the project, but it will bring to light a bunch of issues and dependencies that your project has and you need to know about and document anyway… right?

Once, there was a bug that one of my colleagues was working on. After about a half a day he had made little progress and ask me to help. I looked at it for another 3 or 4 hours and made some progress but the bug was not resolved. Finally another developer looked at it for another few hours before fixing it. So 1 bug took 3 developers a total of 8 or 9 hours to fix. What does 1 hour of developer time cost a company, maybe 100$? So 9 hours times 100$ is 900$ to fix that one bug. This is an extreme example, but each bug effectively can cost an organization several hundred dollars apiece to fix, and that is only development time. Do you have QA and deployment process as well? This will cost. Consider, it might cost you say 2 or 3 thousand dollars to set up a continuous integration environment in terms of time and hardware. And developers might send an extra 10% of their time writing and maintaining tests. If even 25% of bugs are caught up front then that organization has already saved a ton of money for very little overhead.

Even consider the savings simply setting up a simple build server. At one company I was at, I probably wasted 10 to 15 hours a month dealing with and tracking down, why the source I just pulled down did not compile. Again, my salary + benefits + the building I was working in + HR costs + opportunity cost of not working on new features + etc. etc. = 1000$ or 1500$ in wasted time. Having even one developer spending one day could have saved so much wasted time, energy, and money.

  • This is complicated and will confuse people.

Okay, after setting up a continous integration environment in my own spare time to show a manger what kind of benefits it could bring, this was his objection. My response to this is: what kind of people have you hired that they cannot deal with something like this? Developers and generally pretty smart I am sure they can handle it. This is not a new technology, it has been around for years and it benefits have been proven. In his defense he let me go ahead and set it up but I could only build once a night. Of course he was never on board with any organizational buy-in or integrating it into the development process (which, even if he was, was beyond his control).

Here are some objections that a developer might have.

  • I don’t want to be blamed or ridiculed if I break the build. We have a complex product and this is easy to do.

If everyone in your organization is an adult (hope most organizations can say this is mostly true) they should be able to handle it. The effect over time of everyone on the team being aware of this should cause more focus to be placed on improving the problem that exist that are causing this. This will in turn reduce the amount of time spent wading through the complexity and allow individual developers to focus on the problems at hand.

Software development is complex by nature. It is far better to realize there is a problem now when you break the build or a test, than six months from now when it is deployed into a customer’s environment and is causing major chaos. Everyone makes mistakes: this is human. This process is simply a recognition of that fact and an attempt to control for it.

  • Our product is an enterprise app and I couldn’t possible deploy and test this thing realistically.

Study up on testing. In particular mocking. You can write you code in such a way using something like dependency injection and the mock test pattern to create a mock or a fake interface such that when you run your tests, they are testing your code and your code alone.

I’ve seen continuous integration environments that use the API for VM ware to spin up multiple new VMs that then have various products that talk to each other all up and running so that you can run actual integration testing automatically in your environment.

Check out Microsoft’s Team Foundation Lab Management. It basically has solved this problem for you.

Benefits in Terms of Quality

The software industry in my options is facing a crisis of quality. No one trusts their software, and with good reason. Most software that is built, even very expensive enterprise level software is riddled with bugs and security problems.

Management could make the argument and many times does, that it isn’t worth the extra cost to implement this because they never intended on fixing those bugs anyway. This could make sense from a business perspective. If one bug is effecting one customer sometimes, then it may not be worth the 1000$ it might take to fix it, go through QA and deploy.

There is two point I would make against this.

  • What is your reputation worth?

What is this bug really costing you? 1000$? Over time, if your customer, be it an outside company, or another group within your organization, loses confidence in the quality of your software then they will advise others not to buy your software, move on to a competitor, or complain loudly enough to management that you are replaced.

  • You don’t know what you don’t know.

If you had a clear picture of the bugs that you had, you could make a better informed business decision on whether or not to go ahead and fix it. This leads to my next point, technical debt.

Technical Debt

I highly recommend you reading Steven McConnell’s article on technical debt.

Essentially he argues that it is a legitimate business decision to do bad software engineering now to take advantage of a business opportunity or market conditions now, knowing that it will cost you later to maintain and enhance those systems later. However, the huge caveat, is that management is very rarely aware of the technical debt that they are accruing, and therefor they cannot make informed business decisions.

I argue that continuous integration’s relatively small up-front cost and maintenance buy you huge advantage in terms of productivity, plus it makes you aware of the technical debt whether you choose to address it or not, at least you can make better informed decisions.

Did I mention you should read the article? It is absolutely brilliant, I cannot recommend it enough, I cannot do it justice here. Read it!

Conclusion

Continuous integration like many things, has a diminishing rate of return. Setting up a build server has a huge and immediate benefit. Setting up unit tests to run automatically will give a huge benefit, but the effect may not be felt immediately. As you approach 100% code coverage the quality of your application might be stellar, but developers might be spending an unacceptable amount of time on the overhead, depending on the type of your application.

Once you have a continuous integration environment set up, you can go down the path of really interesting things like test driven development, or behavior driven development, or things like dependency injection with mocking like I had mentioned earlier… if you would like to. These things will really push the limit on you in terms of technical prowess, and truly understanding the product that you are working with. But without it, the possibility doesn’t exist.