Interesting overview by the WSJ on this one aspect of Russian trollage.
Update: SentinelOne responds in the comments. Additionally, they claim (and I have no reason to doubt this claim) that the report I referenced was an older version that had incorrect information on the part of Tevanos.
As a follow-on to my recent post about endpoint security (see “A bomb just dropped in endpoint security…“), I thought I’d share some additional thoughts, conclusions and opinions.
It really comes down to beer. But more on that later.
First off, Joe Menn at Reuters wrote a story this morning. It’s a good story, fair and balanced. Worth reading. There will likely be more stories as well.
Beating up vendors isn’t really my thing
In the recent blog post, there were a lot of slings and arrows thrown at a few endpoint security players. My blog got trolled quite a bit. I cleaned it up.
Cylance certainly came under heavy attack by some commenters, and I removed those comments.
I’m not interested in beating the crap out of some company; I’m really just interested in writing about stuff that I find is interesting.
There was a possible confusion that got propagated that Cylance was using VirusTotal directly in their product. I now have information that this may be incorrect.
Cylance was using VirusTotal, as they said in the Reuters article. It’s possible they were using the service to download samples to train their engine, not directly from inside their product. It’s also possible that they used VirusTotal to help detect malware.
I don’t know for sure, and that’s why I expect to be talking to them in the next several days.
Note that Cylance is not a bad group of people. There are many very good people working there, and they run a good business. I’ve challenged them to get more public tests, and I hope they do so. So far, there have been two tests that I know of — a condoned test by Av Test, and another test, where AV Comparatives/Effitas had to basically break the rules to try and get a copy of Cylance (one can’t just download Cylance and test it, as Cylance keeps its trials very closely monitored).
So testing more, and being more public, would be a good thing.
At any rate, peace, people.
Other endpoint players
I don’t know about other endpoint players. I know SentinelOne, for example, has been open about using VirusTotal.
SentinelOne ensures that it is always up to date, checking file hashes against reputable sources such as Virus Total. Using this method, SentinelOne’s platform does not suffer the traditional time lapse in needing to push out new definitions…
[Edit – As mentioned earlier in this blog, SentinelOne tells me this quote is from a PDF of an older document that incorrectly noted VirusTotal as a source and has since been corrected.]
Regarding Palo Alto Networks and CrowdStrike, I really don’t know the involvement of these players. But as the Reuters article mentions, they have been users of VirusTotal (and have not been participating with the community). That’s not to say they’re bad people or have bad products (CrowdStrike and Palo Alto both make excellent products). It’s just something that has been addressed.
My knowledge of the endpoint security market comes from the fact that I’ve been there in the trenches.
I’ve presented at conferences like VirusBulletin and submitted papers, etc., but that’s the fun stuff. Actually being in the day-to-day sweating bullets to try to keep your customers protected is a very difficult task.
You see, we created a full-stack antivirus product at my last company (Sunbelt Software’s VIPRE) and I personally ran the antivirus lab for some time.
When we released the product, it was probably mediocre, and we avoided tests. It got better when we started opening up to tests, and in fact, it got quite good (and then I sold the company and, well, I don’t recommend the product anymore).
We were on VirusTotal, and it was painful to see us miss detections. But at least we saw ourselves for what we were, and made our product better as a result.
Is there a “next-generation?”
The truth is there are only so many ways to skin a cat. The former Eastern Bloc countries were famous for producing some of the world’s most brilliant mathematicians and now we have companies like Kaspersky and BitDefender with just those same type of people. Yet even they have a tough time of it (no matter what they say publicly). It’s not an easy business, trust me.
I don’t entirely buy a lot of the “next-gen” security arguments. I think that there is room for innovation, but I’ve seen companies like Malwarebytes (of which I’m a board member and incredibly biased toward) and (recently) Symantec do some very impressive work in detections, without resorting to anything of the next-generation type. It really comes down to a lot of hard work, block-and-tackling type of stuff.
So, it’s no surprise that people do whatever it takes to get the best result possible. If this means using VirusTotal to do a hash lookup (which IMHO is fairly silly, since polymorphism makes hash lookups far less useful than people might think), or good old fashioned PR to paint lipstick on a pig, well, so be it.
The key is openness. If you have something special, open it up to the world for them to look at, to test, to validate. Be a part of the community and give back to it. It will make your product much better.
Which comes to the beer
Information sharing in security happens around conferences and beer. (When I brought in a new lab manager for my research team years ago, I urged him to spend as much time as possible going to conferences and drinking beer with other experts. He didn’t object).
Perhaps that is a bit tongue-in-cheek, and it’s not that we’re a bunch of alcoholics (well, mostly not), it’s simply that a lot of information sharing happens at conferences, when experts talk to each other freely. The swords of competition are put down briefly, people open up, and you hear a lot of interesting things.
And what I hear around the tables is what is reflected in some of my blog posts. There is good data, but it can’t be substantiated and it won’t ever be confirmed. So, I’m sorry I can’t be specific, despite many of you emailing me for much more detail than I am prepared to give. Trust is everything in security.
But beer? Yes, we can all share that freely. So, here’s to beer (in my case, the ever excellent Buckler Non-Alcoholic).
You can see who VirusTotal credits here.
Update: Reuters now has the story.
Update 2: I’ve updated this post with additional information, here.
VirusTotal just dropped a major bomb, and only people deep in the endpoint security ecosystem understand the ramifications of this announcement.
If you’re involved in endpoint security to any degree – as a customer or an industry person – you need to understand what just happened. It’s really, really big.
A bit of background.
VirusTotal is a multi-engine virus scanner. You upload a file, and it passes the file to a large number of commercial antivirus products, and it tells you which engines detected the file as malicious.
While there are other tools available, and some have come and gone, VirusTotal is the big dog in the space. It’s owned by Google, has massive computing and resource power and everyone in the security industry uses it.
VirusTotal shares the results with subscribers. So, you can pay to get extensive and detailed information on what has been detected at any moment of the day, and who detected it.
How antivirus companies use VirusTotal to make better detections.
It’s common practice of antivirus companies to use VirusTotal as a tool to make better signatures.
For example, if a researcher finds that two high quality antivirus engines detect a file as malicious, he/she has a high confidence that it’s actually malicious without further analysis. As an antivirus researcher, it saves an enormous amount of time.
Now, there’s absolutely nothing wrong with using VirusTotal results in research, and many antivirus companies use VirusTotal to supplement their own labs. They get samples from VirusTotal, and along with the samples, what engines detected them. If they find that a couple of high quality engines are detecting a file, they can easily add the detection to their own signatures without much further thought.
Now, there’s a next step. You could set up an an API integration with your product. If you scan a user’s machine and find an unknown file, you could upload it through an API to VirusTotal and get a disposition on the file –who detects it. From this data, you can flag a file as malicious.
In other words, you can use VirusTotal to create your own antivirus program. Easily.
It’s fine to use other engines. If you’re also contributing.
Using other engines to improve your detection rate is completely fine. If you’re also contributing back to the community yourself. In other words, if your antivirus product is also one of the participating antivirus engines.
The dirty little secret
And here’s the dirty little secret that very few people know. There are a number of endpoint products that use VirusTotal to determine if a file is malicious. Without any contribution to the community. Without giving anything in return.
They simply pay VirusTotal a subscription fee, and receive the information.
And some of these companies have been getting a lot of attention for their supposed prowess. But for some mysterious reason, they refuse to put their own engines on VirusTotal. Could it be because they don’t want to contribute back? Maybe. Or it could be that they just don’t want everyone else to see how poorly their products actually perform.
Using VirusTotal information without any contribution back to the community is patently unfair. The people who are actually writing detections are sharing their results with the rest of the community, while a small group of endpoint products have been boasting of their extraordinary abilities, while working off the backs of other researchers.
So as a customer, perhaps you can ask the next endpoint security vendor if they’re on VirusTotal. If they are, they’re contributing to the antivirus community. If they’re not, they’re not. Whatever their PR story, that’s the simple truth.
Well, the world just got a bit brighter for the many endpoint security companies that actually contribute to VirusTotal: Because VirusTotal just announced that they are requiring that all scanning companies that use their service must integrate their engines into VirusTotal. Furthermore, “…new scanners joining the community will need to prove a certification and/or independent reviews from security testers according to best practices of Anti-Malware Testing Standards Organization (AMTSO).”*
It’s big news. It levels the playing field. No longer will antivirus companies see their hard work taken by some sexy startup that’s raised millions of dollars on the false promise of “next generation” endpoint or other such nonsense, while bashing the very companies that they’re effectively stealing the intellectual property of. And perhaps, we’ll see what their products are really made of. Because without VirusTotal as a crutch, companies that rely on it are going to see their detection rates take a hit.
Poetic justice, indeed.
What does this mean for the IT manager?
If you’re an IT manager who has been duped by sparkly marketing materials to buy-in to one of these “next-generation” endpoint products, take a hard look at their actual detection capabilities. If they’ve been using VirusTotal results but not contributing back, their ability to detect malware just took a potentially serious hit. This is serious.
You don’t have to believe the marketing hype. Setup a virtual machine that’s separated from your corporate network, and go to a site like MDM to find all kinds of nasty stuff. In the words of Ronald Reagan, “trust, but verify”. One nasty piece of malicious software (especially ransomware) can have serious consequences.
My compliments to the VirusTotal team for seeing this disparity and unfairness and taking such swift action. A class act, indeed.
And now, perhaps, we can all finally see what is really behind the curtain.
* Disclaimer: I am a board member of Malwarebytes (a contributing member to the VirusTotal community), and an advisory board member to AMTSO. The opinions in this blog post are my own and are not connected to these two organizations.
Yesterday, Kaspersky announced that some of its internal systems had been breached. While this may have created a sense of Schadenfreude in some parts of the security community, Kaspersky has handled the situation quite well. Instead of other companies that have suffered a breach, Kaspersky worked straight from the crisis management playbook — full disclosure, plenty of information and a plan. Kudos.
As Graham Clulely says: “In short, it handled what could have been a corporate crisis well – and reassured customers and partners that their data was safe, and the integrity of its security products had not been compromised.”
(Although one can’t help but wonder at the timing. According to the press release, the malware was found in “early spring 2015”, but the announcement is coming on June 10th — just a few weeks away from the official start of summer…)
Kaspersky is being up-front, but they are also spinning this as a research item. And that’s okay, because it is some fascinating research. This is a very interesting new malware variant, and quite sophisticated, quite likely tied to state-sponsored activities.
But they just can’t help being Kasperskyish:
From a threat actor point of view, the decision to target a world-class security company must be quite difficult. On one hand, it almost surely means the attack will be exposed – it’s very unlikely that the attack will go unnoticed. So the targeting of security companies indicates that either they are very confident they won’t get caught, or perhaps they don’t care much if they are discovered and exposed. By targeting Kaspersky Lab, the Duqu attackers probably took a huge bet hoping they’d remain undiscovered; and lost.
(Yeah, that explains the Schadenfreude part.)
Anyway, the bigger story is the state of the antivirus detections on the day following Kaspersky’s announcement.
Considering that most antivirus vendors practice what I call “hash-whoring”, where hash detections from VirusTotal or internal scans are dumped wholesale into their databases (explaining the massive size of today’s antivirus engine), the poor detection state of this variant is surprising. (Incidentally, I’m not condemning this practice — it’s a very useful stop-gap until a detection team can make a good detection — nevertheless, it’s abused way too much, especially by poorer quality, also-ran engines.)
And so on.
So, just for fun, I’ve published the hashes below, hyperlinked to VirusTotal. You can click on them as the week progresses to see the state of detection of your antivirus product.
(And, also remember to blacklist the C&C IPs: 220.127.116.11 and 18.104.22.168.)
Is this blog post another tired rant against antivirus? Absolutely not. AV isn’t dead. It’s part of a valid belt-and-suspenders approach to security.
But, a day later, and we still see poor detection? Yeah, that part sucks.
If you’re heavily stressed as a business leader, the business is running you — not the other way around. Chances are you’re not prioritizing correctly, and you’re not delegating.
I’ve worked with CEOs who put in an insane amount of hours and don’t do any better than CEOs who work a fairly normal schedule (granted, usually 50–60 hours a week).
One could describe a leader as someone who establishes and communicates clear goals, gets the right people in place, gets everyone working toward these goals and focuses on what’s important.
Culture is an additional ability of leadership. Culture is less important, actually, than fanatical execution on a clear set of goals. Ping pong tables, beautiful offices — nice — but not vital.
The core is figuring out where you’re going, getting the right people going in the same direction, and focusing on what’s important.
Sounds easy, but it’s an art. It’s why great CEOs are paid a lot of money and are in high demand, because it doesn’t come intuitively or naturally to a lot of people. However, it can be learned.
Teaching leadership skills, however, isn’t the purpose of this blog post. I’m just going to tell you what’s important.
There are just a few things that you have to do really, really well in this business. If you do those well, everything else follows.
Many years ago, one of my early mentors told me, “if you just focus on creating a great product, support it well and do a good job on PR, you should do just fine.”
Not bad advice. I’ll expand on it with a bit of my own experience.
Here is the scale of importances in running a product or services business.
1. The product or service.
2. The quality of the product or service.
3. Support/customer service
4. PR and marketing
Assign KPIs to each area (you can’t manage what you can’t measure…). At the beginning of every week, go through each of these areas by yourself. And then go through these with your senior staff at your Monday morning staff meeting.
The funny thing is that as an executive, you may find yourself spending a tremendous amount of time keeping people focused on doing the important things. And, you may find yourself burdened down with things that aren’t that important. People add complexity to everything they do. It’s a natural tendency, but it generally means that they are not confronting what really needs to get done (either because they don’t know, or because they don’t understand something).
If you establish an organization with this set of importances, you’ll increase your chances of doing well.
The mistakes I’ve made are when I’ve reversed the priority — too much emphasis on finance, or sales, etc. The product (or service) is the most important thing to focus on (read my other post, The Product is All). Give the accountants the problem of worrying how to book the revenue. Give the sales and marketing guys the problem of actually getting the revenue. And get the product guys firmly lined up with what’s needed and wanted from the market, and delivering it.
And lead a less stressful life.