Revisiting "Behind the Scenes: How I Build Software"

Back in February I had a post that introduced how I build software around here. I thought I would update you on what has been happening in the last couple of months as we get closer to our launch with our latest stuff.

More importantly, I wanted to blog about our final decisions with the source control, installer and automated testing tools we are using.

Firstly, the source control. I was getting frustrated with Subversion as I couldn't understand their methods in tags and branches. This was because I got ahead of myself eons ago when I first set up the repository and DIDN'T follow their suggestion of having a product/trunk, product/branches and product/tags heirachy. When we upgraded our servers this spring and started using two factor authentication for everything, I revisited this and rebuilt the repo from scratch. We lost all the history, but that was a small price to pay for a heirachy that makes TOTAL sense now... and makes tags and branches easy to understand. I also moved to using Apache/SSL for the delivery of the master sources instead of the svnserve daemon. The biggest reason was individual user authentication and control within the repo. We have much more fine grained control over the different areas in the repo now... QA have their own hive as do DEV. And our two factor authentication tokens have a built in module for Apache that works rather nicely. So Subversion and TortoiseSVN are here to stay.

Now for the installer. If you recall, I was NOT happy with our installer, which was a product called InstallAware. Well, more to the point I was vexed at how I was treated by the developer after I had to debug so much NONWORKING code in the cmd line build util (miabuild) and other pieces. The problem stemmed from the fact that I was taking a HUGE risk on a unknown product written by a single guy with no business aptitude. I had no idea if he would even be around in 6 months. On top of that he was overseas where my legal recourse was not well established if he wouldn't come through with delivery of his promises to me. The reality was that the developer was VERY quick to resolve most bugs (except for a huge memory leak and a problem with the compression library that got fixed in the more recent versions) and I REALY liked the product. It's hard when you find the technology good, but the company ran rather poorly. And to boot, less than 3 months after I got most of my stuff finally resolved he released a new version that fixed some of my outstanding issues, and he wouldn't even give me a free or reduced rate upgrade! I couldn't believe how I was being treated.

Then I remembered that as a developer myself, I have been in that same boat over a decade ago. When you get a developer thinking like a technician, acting like a technician and BEING the technician, the sacrifice is that of the business mind. In other words, its hard to balance the entrepreneurial aspects of a business with that of a developer who is "married" to the widget that he or she provides. The reality that without the customer you are NOTHING doesn't sink in until you have the war wounds to see that almost anyone can make software... but very few can make a successful software BUSINESS.

I decided to stop being angry about the way I was being treated and deal with it head on. I decided to weigh the product on its technical merits against the competitors and then weigh it on the need for stability of the business. In the midst of this evaluation I had Wise (our old installer company) and InstallShield both approach me with insanely discounted pricing. At that point, it didn't become an issue about money any more... it came to technical merit... and actual support. And here was where my mind cleared up.

After the dust settled, I stuck with InstallAware and paid for the upgrade to InstallAware Studio 2005. Its just too good a product to let go. Its got everything I want in an installer, at a rather attractive price to boot (without heavy discounting by aggresive sales people). The developer offers GREAT technical support (note I said technical support, not customer service), and is more helpful in his forums than Wise ever was. The product builds great MSI packages and has an easy UI that supports visual layout AND a full scripting mode. The cmd line build tool called "miabuild" works great in our automated build environment, called directly through nant. And finally, the product has started to gain more traction since I first tested it. Besides the fact its getting more testing, the developer is trying to get serious about his company and its offerings... which is starting down the path to be a sustainable business. And in the end, if he fails and goes away... the current product is stable and works rather well. I reduced the risk to MY business by ensuring that the MSI output can be imported to other installers. If I have to every migrate away to a competing product... I have comfort knowing both Wise and Installshield can import my MSI and get me to a pretty stable project within an hour. But I doubt I will ever have to do that... InstallAware is THAT good. I am even comfortable enough now to recommended the product to fellow developers. InstallAware is great product that you need to try out for yourself. You can download the trial here.

Finally I want to talk about our automated testing. Since I last discussed this I have done a 180 on our testing approach. Although I find unit testing to be great developer's asset, I think an even better approach is to use a testing framework that offers unit, regression and integration testing in a single suite. That's where our decision to use AutomatedQA's TestComplete has been a god-send. I now have a dedicated Quality Assurance Test Specialist whose only job is to build automated tests for every single piece of shippable code using TestComplete. As our test library continues to grow, we continue to refine the testing mechanisms in an effort to maintain stable, shippable code at all times. We also added some new systematic processes in both our dev and QA workflow:

• Any new bug reported in FogBugz immediately gets assigned to QA. QA then builds an automated test script to reproduce the bug, attaching it to the Case and reassigning it to the developer who needs to work on it. This test is then used by the developer to not only repo the issue, but ensure the fix passes the test. Only after it passes the test can the code be checked into the source control server, and the Case be resolved
  
• The said test above is immediately added to our automated testing framework once the Case is resolved. In this way... this bug should NEVER reappear in the future. If it does, we will immediately catch it before we ship it out
  
• All new features must have a set of tests completed before it can be shipped. Kind of a moot point in our current dev cycle.. since most things are new, and we are still building tests for things... but its a good process to have in place.
  
• Some of these tests are what we call "public facing" tests. In other words, we can ship an executable test harness to customers in the future to run specific tests on their own systems. This will allow people to not only evaluate the product, but expose problems that may exist on the specific platform being tested. This will let our Customer Service reps get an immediate indication of what is going on without burdening the end user with tonnes of questions.

Out of all the tools I have purchased in the last year, this is the one that excites me the most. As the number of USEFUL tests increase, so does the assurance level of the quality of product that we ship. For a small dev shop like ours, thats great news. I would bet that our QA process rivals that of many larger software dev firms that use manual testing, SPECIFICALLY because we removed the wasted burden and poor investment in people to do trival things over, and over and over and over again. By automating so much of this, we can zero in on the real issues quickly and efficiently, and leave people to work on things that really make a difference. And THAT'S a great investment in the business, our products and our customer's success with our products.

Wow, long post. Never realized I had so much to say on all of this. But alas, I do hope it was useful to you. The use of Subversion, InstallAware and TestComplete have been great assets to our software development lifecycle, and I am sure our customers will be pleased with the results. I know I am. And so is my team.