ConsumerReports beats up anti-virus, then gets beat up

ConsumerReports just completed a study which tested anti-virus softwares for their effectiveness. But instead of just using the known threats and existing signatures, they created thousands of virus variants of their own to see if protective measures did any good. Of course, you have to be a subscriber to their magazine to get the results, so I’d love to hear from someone as to who won the battle, but nevertheless I thought it was a great idea.

Not everyone did.

The watchdog group is now being slammed for their approach, and I say this should serve as a warning to everyone who trusts their boxed anti-virus kit. Graham Cluley of Sophos noted:

“When I read about what ConsumerReports has done I want to bash my head against a brick wall. With over 185,000 viruses in existence was it really necessary for this magazine to create 5,000 more? It’s irresponsible behavior, and will be frowned upon by the antivirus industry. Leave antivirus testing to the independent testing bodies with expertise in the field”

Alarms aren’t designed to set themselves and subsequently go off only on designated burglar days, but anti-virus is certainly designed to trigger against known threats. That is what ConsumerReports was trying to get at – could anti-virus protect against previously unknown viruses. They even used existing signatures, varying them just slightly (like malcreants do). And I found no mention of ConsumerReports releasing them into the open, as the quote infers.

Maybe Graham wants to bash his head against the wall because his product doesn’t really protect like it should, and now he and his entire industry have been called out?


No argument from Slashdotters.


Matt Sergeant says:

Not all anti-virus works like that. Though all the commercial ones do which is why we wrote our own. You’re spot on though – Graham doesn’t want the cat let out of the bag that most AV software only detects viruses they’ve already seen before. Not much use in an outbreak really.

Not much use in an outbreak…”very”

Thanks for the point, Matt. I think all readers (plus me) would love some additional explanation on what you guys do differently. May be set an example for the commercial folks (proprietary methods exclusive, of course)?

Matt Sergeant says:

So the big problem with commercial AV is that it’s designed for the desktop. Even when it’s deployed to stop viruses in email it is looking at attachments because that’s what they know – files. We saw that there were specific techniques used in email that made viruses easier to detect and implemented them.

Also there’s ways we can examine executables that the AV vendors just can’t do (because of the desktop thing). A good way of thinking about this is that on the typical desktop there’s millions of files and executables, but for the most part nobody emails them to eachother (you don’t email someone notepad.exe). So a false positive on a desktop scanner can be really critical, but when you’re just scanning email you know you can be a bit more aggressive.

Finally, all viruses are basically evolutions of the one that came before it. Virus writers don’t tend to write a brand new virus – they take their previous code and modify it slightly, tweaking it until they find it gets through the version of McAffee or Sophos they have installed. Then they send it out. So we create heuristics that can detect variations on the themes.

I make it sound simpler than it is really – but we saw this flaw in the usual AV techniques and took the opportunity to fix it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.