July 05, 2005

Revisiting "Behind the Scenes: How I Build Software"

Back in February I had a post that introduced how I build software around here. I thought I would update you on what has been happening in the last couple of months as we get closer to our launch with our latest stuff.

More importantly, I wanted to blog about our final decisions with the source control, installer and automated testing tools we are using.

Firstly, the source control. I was getting frustrated with Subversion as I couldn't understand their methods in tags and branches. This was because I got ahead of myself eons ago when I first set up the repository and DIDN'T follow their suggestion of having a product/trunk, product/branches and product/tags heirachy. When we upgraded our servers this spring and started using two factor authentication for everything, I revisited this and rebuilt the repo from scratch. We lost all the history, but that was a small price to pay for a heirachy that makes TOTAL sense now... and makes tags and branches easy to understand. I also moved to using Apache/SSL for the delivery of the master sources instead of the svnserve daemon. The biggest reason was individual user authentication and control within the repo. We have much more fine grained control over the different areas in the repo now... QA have their own hive as do DEV. And our two factor authentication tokens have a built in module for Apache that works rather nicely. So Subversion and TortoiseSVN are here to stay.

Now for the installer. If you recall, I was NOT happy with our installer, which was a product called InstallAware. Well, more to the point I was vexed at how I was treated by the developer after I had to debug so much NONWORKING code in the cmd line build util (miabuild) and other pieces. The problem stemmed from the fact that I was taking a HUGE risk on a unknown product written by a single guy with no business aptitude. I had no idea if he would even be around in 6 months. On top of that he was overseas where my legal recourse was not well established if he wouldn't come through with delivery of his promises to me. The reality was that the developer was VERY quick to resolve most bugs (except for a huge memory leak and a problem with the compression library that got fixed in the more recent versions) and I REALY liked the product. It's hard when you find the technology good, but the company ran rather poorly. And to boot, less than 3 months after I got most of my stuff finally resolved he released a new version that fixed some of my outstanding issues, and he wouldn't even give me a free or reduced rate upgrade! I couldn't believe how I was being treated.

Then I remembered that as a developer myself, I have been in that same boat over a decade ago. When you get a developer thinking like a technician, acting like a technician and BEING the technician, the sacrifice is that of the business mind. In other words, its hard to balance the entrepreneurial aspects of a business with that of a developer who is "married" to the widget that he or she provides. The reality that without the customer you are NOTHING doesn't sink in until you have the war wounds to see that almost anyone can make software... but very few can make a successful software BUSINESS.

I decided to stop being angry about the way I was being treated and deal with it head on. I decided to weigh the product on its technical merits against the competitors and then weigh it on the need for stability of the business. In the midst of this evaluation I had Wise (our old installer company) and InstallShield both approach me with insanely discounted pricing. At that point, it didn't become an issue about money any more... it came to technical merit... and actual support. And here was where my mind cleared up.

After the dust settled, I stuck with InstallAware and paid for the upgrade to InstallAware Studio 2005. Its just too good a product to let go. Its got everything I want in an installer, at a rather attractive price to boot (without heavy discounting by aggresive sales people). The developer offers GREAT technical support (note I said technical support, not customer service), and is more helpful in his forums than Wise ever was. The product builds great MSI packages and has an easy UI that supports visual layout AND a full scripting mode. The cmd line build tool called "miabuild" works great in our automated build environment, called directly through nant. And finally, the product has started to gain more traction since I first tested it. Besides the fact its getting more testing, the developer is trying to get serious about his company and its offerings... which is starting down the path to be a sustainable business. And in the end, if he fails and goes away... the current product is stable and works rather well. I reduced the risk to MY business by ensuring that the MSI output can be imported to other installers. If I have to every migrate away to a competing product... I have comfort knowing both Wise and Installshield can import my MSI and get me to a pretty stable project within an hour. But I doubt I will ever have to do that... InstallAware is THAT good. I am even comfortable enough now to recommended the product to fellow developers. InstallAware is great product that you need to try out for yourself. You can download the trial here.

Finally I want to talk about our automated testing. Since I last discussed this I have done a 180 on our testing approach. Although I find unit testing to be great developer's asset, I think an even better approach is to use a testing framework that offers unit, regression and integration testing in a single suite. That's where our decision to use AutomatedQA's TestComplete has been a god-send. I now have a dedicated Quality Assurance Test Specialist whose only job is to build automated tests for every single piece of shippable code using TestComplete. As our test library continues to grow, we continue to refine the testing mechanisms in an effort to maintain stable, shippable code at all times. We also added some new systematic processes in both our dev and QA workflow:

  • Any new bug reported in FogBugz immediately gets assigned to QA. QA then builds an automated test script to reproduce the bug, attaching it to the Case and reassigning it to the developer who needs to work on it. This test is then used by the developer to not only repo the issue, but ensure the fix passes the test. Only after it passes the test can the code be checked into the source control server, and the Case be resolved
  • The said test above is immediately added to our automated testing framework once the Case is resolved. In this way... this bug should NEVER reappear in the future. If it does, we will immediately catch it before we ship it out
  • All new features must have a set of tests completed before it can be shipped. Kind of a moot point in our current dev cycle.. since most things are new, and we are still building tests for things... but its a good process to have in place.
  • Some of these tests are what we call "public facing" tests. In other words, we can ship an executable test harness to customers in the future to run specific tests on their own systems. This will allow people to not only evaluate the product, but expose problems that may exist on the specific platform being tested. This will let our Customer Service reps get an immediate indication of what is going on without burdening the end user with tonnes of questions.

Out of all the tools I have purchased in the last year, this is the one that excites me the most. As the number of USEFUL tests increase, so does the assurance level of the quality of product that we ship. For a small dev shop like ours, thats great news. I would bet that our QA process rivals that of many larger software dev firms that use manual testing, SPECIFICALLY because we removed the wasted burden and poor investment in people to do trival things over, and over and over and over again. By automating so much of this, we can zero in on the real issues quickly and efficiently, and leave people to work on things that really make a difference. And THAT'S a great investment in the business, our products and our customer's success with our products.

Wow, long post. Never realized I had so much to say on all of this. But alas, I do hope it was useful to you. The use of Subversion, InstallAware and TestComplete have been great assets to our software development lifecycle, and I am sure our customers will be pleased with the results. I know I am. And so is my team.

Posted by SilverStr at July 5, 2005 09:56 AM | TrackBack
Comments

Wow! Excellent post on how you do everything! I quickly looked over the TestComplete page and it looks very useful and may be something I spend some time looking in to. Automated testing is something we don't have here, instead we have a person that does the testing with documented test scripts, but it looks as though TestComplete may be able to do what she does, and we can put her to use in other areas of QA.

I also definitely like how you've integrated the test scripst with FogBugz! It would be nice to get a case that has the script attached to it instead of the standard "It's broken" text.

Posted by: scoobyd at July 5, 2005 11:55 AM

But do you really need visual tools for MSI? I've moved all company setups from InstallShield/Wise to WiX (wix.sourceforge.net) because of ugly IS/Wise MSI result and really great WiX support comunity (btw DIFxApp have WiX library). So I'll never return back to visual tools ;-)

Posted by: Sergey Simakov at July 5, 2005 12:27 PM

I enjoyed that a lot! Question: How does TestComplete help with the typical responsibilities of a unit test. Looking at the link it strikes more as a UI and frontend testing tool whereas NUnit and whatnot are used to test specific classes, methods etc. Can TestComplete do the same or does it always need a UI (winform, webform app) to run against?

Posted by: Thomas Wagner at July 6, 2005 12:26 PM

Hey Sergey,

I tried WiX out. Although it was interesting, I found it cumbersome when doing more complex things... mostly because I don't have time for the learning curve of learning the syntax for the XML format of the tool.

InstallAware let me have the flexibility I needed without having to use Orca to modify the MSI.

A good combination of both visual and scripting power.

Posted by: SilverStr at July 6, 2005 02:37 PM

Hey Thomas,

Being that we are a .NET shop here for the UI (all frontend code is written in standalone Winforms in C#), TestComplete has the ability to inspect and execute our functions from within external scripts and the unit test harness of the application. To see how TestComplete does this... check out their tutorial on the subject at: http://www.automatedqa.com/products/testcomplete/tc_tutorial_advanced.asp

Posted by: SilverStr at July 6, 2005 02:44 PM

If only TestComplete worked under non-admin account...

Posted by: Sergey Vlasov at July 6, 2005 07:17 PM

Sergey,

Ya I had the same complaint. I have discussed this with their dev team, and they assure me that the next version will make this a bit easier. For now there is a thread I started in their newsgroup back in May that discussed what perms are needed where:

http://www.automatedqa.com/forums/d.cgi?cmd=xover&group=automatedqa.public.testcomplete&related=7529&utag=

I wasn't to keen on opening entire registry hives to the application. What we do is run TestComplete and TestExecute in a VMWare image. This reduces the associated risks with admin accounts, and allows us to still fully use the product.

Posted by: SilverStr at July 7, 2005 07:47 AM