Dana, I think you should give a look here : http://www.automatedqa.com/ this guys ahve developed several products, one of which has the majority of the features you'v ebeen askign for (TestComplete).

Hope this helps :)

MASAN

About 18 months ago I looked into some functional testing software. There were a couple of ones that look really appealing, did anything related to Win32, GUI, Web Services, etc.

I called up Mercury Interactive to get some information on WinRunner, but it's $7000 USD per developer. When they heard I was only looking for a couple of licenses and not 10+, they started giving me the runaround and stopped calling me.

Anyways, that was my experience :-)

Hi Dana,

I'm a one-man QA show at my company, and I have some experience with QuickTest as it's our primary testing tool. I very much desire the same sort of features that you have, and QuickTest is _sometimes_ good at them.

Going point-by-point:

- QuickTest's IDE is fine.
- No experience with Winforms (we're a web app shop)
- Automated runs are possible using the Windows Script Host. One of the first things you will want to do is dump the crappy batch job runner that comes with it. I have 150 lines of JScript that are WAY more reliable.
- OS compatibility is sticky. Mercury uses an Activation-based approach for their software, so you'll have to activate each install. To get it to run on a Windows Server-family system, you'll need to purchase the concurrent licensing scheme (we were quoted $10,000 CDN per user for that... single user was $6,000 CDN per seat). We tried to use our seat license over remote desktop (only one user) but the Mercury licensing prevented this entirely (and we had to call in for a reactivation).
- QuickTest tries to be all things to all people. It's definitely not "lightweight".
- Custom actions are in VBScript. This sounds ok and is easy to start non-programmers on, but becomes really, really annoying once the users learn how to code.
- Documentation is OK. The Mercury forums and telephone support are really hit-and-miss.
- No built in automated notifications. They want you to buy Test Director (~$10,000) to do this. We're writing a Web Service based custom reporting system that's substantially cheaper (and also not dependent on Mercury software).
- I noted pricing above. It was $6,000 CDN with just the default plug-ins for our single seat.

A few other things you should know about:
- Working in Subversion is difficult. QuickTest scripts are actually entire directories full of files. Some are text (like the VBScript commands), but most are binary. The text is useless without the corresponding binaries.
- QuickTest puts a mapping layer between your app and your test scripts. This is convenient when the name of a control changes... you simply update the mapping and all is well. Unfortunately, the mapping is stored in a proprietary binary format. Furthermore, the tool for managing it has no way to add controls after the first recording session. This leads to the entire mapping being dumped and re-recorded frequently, which breaks things regularly. We dumped the mapping component entirely and wrote code generators to look at HTML and generate interface classes (that talk to QuickTest). I have seen other shops write a custom mapping layer, or simply do without and hard-code control names and properties.
- Recorded QuickTest scripts are really brittle. Small changes to the application frequently breaks them in ways that require re-recording.
- Recording is a terrible way to QA, in my opinion. The app must be in a known-good state before it can be recorded. This prevents writing the test scripts before the app is ready. Worse, bugs that go unnoticed during recording become part of the script and get "passed" by the automated tests in future test cycles.
- QuickTest stores each "action" as a separate set of files in its own directory, and has reimplemented parameter passing between them. This is a nightmare to debug.

Overall, I'm not enormously impressed by QuickTest. Its core feature, interacting with the Application Under Test, works very well, but it's a "high maintenance" solution. In the long run, we will probably be moving to a script-first (ie: non-recorded) open source tool like WATIR or Selenium as these mature. Already WATIR outperforms QuickTest in many areas.

Your mileage may vary. I'd be happy to discuss further with you if you like - please contact me through my email / blog.

Thanks for the input guys. Loved the deep insight Alf.

It looks like I am going to try TestComplete from from Automated QA. It is very affordable at $795 per user and seems to have no addition fees for the standalone 'runner' through something called TestExecute. This means I can run TestExecute on the test boxes and only have to buy a copy of TestComplete for each QA guy. Whats also nice is apparently you can ship TestExecute to a client site. That way... if there is a problem you COULD ship a test suite to run it on the production system and see what is failing and why.

I sent an email to confirm the pricing... but if its correct I think this might just be a winner. Will keep you posted once I start more seriously looking into testing it.