Much has been said in recent weeks about the state of AV technology. To add facts to the debate, Lastline Labs malware researchers studied hundreds of thousands of pieces of malware they detected for 365 days from May 2013 to May 2014, testing new malware against the 47 vendors featured in VirusTotal to determine which caught the malware samples, and how quickly.
The focus of this test is to determine how fast the anti-virus scanners catch up with new malware.
Note that the configuration of the various AV scanners used by VirusTotal is not necessarily optimal, and it is always possible that a better detection rate could be achieved by relying on external signals or using more “aggressive” configurations.
On any given day, according to Lastline Labs’ analysis, much of the newly detected malware went undetected by as much as half of the AV vendors. Even after 2 months, one third of the AV scanners failed to detect many of the malware samples. By averaging the daily detection rates, we are able to plot the pace at which the AV scanners catch up with the malware. The least-detected malware - that is the malware in the 1-percentile “least likely to be detected” category - went undetected by the majority of AV scanners for months, and in some cases was never detected at all.
Some other interesting findings of this Lastline Labs research:
- On Day 0, only 51% of AV scanners detected new malware samples
- When none of the AV scanners detected a malware sample on the first day, it took an average of two days for at least one AV scanner to detect it
- After two weeks, there was a notable bump in detection rates (up to 61%), indicating a common lag time for AV vendors
- Over the course of 365 days, no single AV scanner had a perfect day - a day in which it caught every new malware sample
- After a year, there are samples that 10% of the scanners still do not detect
Top 1% of malware evolved against AV patterns
As you can see in grey lines in the chart above, there is a steady growth curve in the detection rates from Day 0 to Day 365 of the average malware. This pattern mostly mirrors that in the 1-percentile malware trajectory (percentiles based on least detected) which are likely more sophisticated or unique. The 1% of malware that most effectively evaded detection in this dataset is likely to represent the kind of advanced malware created and exploited by cyber-criminals who are persistently and directly targeting and infiltrating organizations, as opposed to more opportunistic malware distributors.
AV alone is not enough
For us, this preliminary dataset leaves us with as many questions as answers. This analysis does not single out any AV vendor, and provides only insights based on VirusTotal data (with the caveats expressed at the beginning). We think that “traditional” AV technology is not dead, but needs to be complemented with other approaches (e.g., based on dynamic analysis of samples, network anomaly detection) that provide additional signals for detection.
In future analyses, we will be looking for patterns in the least-detected malware that may indicate common trends or behaviors that could help all network security - including AV scanners - improve malware detection effectiveness and speed. This data definitely points to the conclusion that AV alone is not enough.
More research required
We plan to test further and compare the effectiveness of traditional sandboxing with next-generation sandboxing. Our hypothesis is that the least detectable malware is designed to both evade detection and fingerprint the analysis environment. From what we have seen so far, no commercially available signature-based security system appears to be able to get ahead of advanced malware on its own.