July 14, 2005

How a single software bug froze SBS, and cost Trend $8 million

Back in April, Trend accidentally shipped a dat file that drew a weekend of hell for a LOT of SBS administrators. Susan even warned about it when the roll out of dat file 594 caused enough problems that most SBS admins were rolling back to previous versions (including her); others dropped Trend on the spot.

Well, the cost of the software failure didn't end there. Techworld reports that Tokyo-based anti-virus software vendor Trend Micro said the bug affected thousands of customers and has cost the company $8 million USD. The issue has also forced it to lower its revenue and profit forecasts for the April to June quarter. Thats right... this one small mistake cost them over $8 million dollars directly to the company (mostly in call centre costs).

Makes you wonder.

How can a company this large, with a sales revenue of $150 million USD in the same quarter, let something like this happen. It would be a fair bet that they have a large testing team. A strong QA department. A budget to be envied by many software development firms. Yet a single small flaw impacted them enough that they have to restate their revenue guidance... it cost them over $8 million USD to deal with it.

Wow. There are lessons to be learned from this. This could happen to any software company. I would have loved to be a fly on the wall when Trend went over its QA process and refined their business systems to ensure this never happens again.

Posted by SilverStr at July 14, 2005 08:16 AM | TrackBack
Comments

Well, it can't happen to any software company, just those who automatically update their customers. That's a growing number, but many customers still have their own testing process for vendor patches. Only "database" updates, like virus dat files or spam blacklists, happen outside of that approval process. The lessen learned is for those companies without a patch approval process to realize the risk they place in trusting a vendor. From your previous posts, I know you understand evaluating risk. Any time you trust a vendor, you introduce risk. A patch management process can mitigate that risk to a certain degree.

Posted by: Xavier Ashe at July 14, 2005 10:55 AM

Actually, it doesn't surprise me. While we didn't have this particular issue with Trend, we had a lot of other beef with their support structure in the last year.
Trend wasn't very helpful until we threatened to terminate the contract and take a hard look at other vendors.
Now all of a sudden they're trying very hard. If this is what they understand under IT Service Management then I can only support anyone moving away from their products.

Posted by: Axel at July 14, 2005 01:03 PM

Trend aren't the only ones.

Another AV vendor (headquartered in Oxford, UK) released a signature file this week which managed to identify modem drivers amongst other files as viruses. On Monday morning, this was looking like a serious outbreak (and all the potential ramifications of such an occurence such as business downtime, clean-up costs, restore from backups, compromised data etc.)

Apparently, it took two hours to contact their support desk about this and for them to finally admit that there was a problem. The information never made it to their web site (and is still not being reported.)

You would think for a company with a £55,000,000+ turnover (2004) that they'd be able to test signature files prior to releasing them (and it's not as if they're first to market with signature files for newly discovered viruses usually anyway.)

I shudder to think of the consequences of this 'error' if an eagle eyed admin had not spotted the false positives.

And this is also the same vendor that removed all adware detection from its flagship AV product last year without telling anyone and have as yet still offered no plausible or creditable explanation for this decision. Apparently it is to make a reappearance in version 6 but why was it removed in the first place?

No wonder we are thinking of moving vendors and perhaps like Axel's company, we need to tell them this!

Posted by: Fatmanmp at July 20, 2005 01:48 PM