A preliminary look into who was hacked in the Sunburst attack

At Prevasio, we started to narrow down those potentially affected by the Solarwinds hack as the Sunburst used a DGA (Domain Generation Algorithm) that gives us a glimpse into who may have been infected.

The list (with disclaimers) follows:

Decoded DomainMapping (Could Be Inaccurate)
hgvc.comHilton Grand Vacations
AmerisafAMERISAFE, Inc.
kcpl.comKansas City Power and Light Company
SFBALLETSan Francisco Ballet
scif.comState Compensation Insurance Fund
LOGOSTECLogostec Ventilação Industrial
bmrn.comBioMarin Pharmaceutical Inc.
AHCCCS.SArizona Health Care Cost Containment System
nnge.orgNext Generation Global Education
cree.comCree, Inc (semiconductor products)
calsb.orgThe State Bar of California Public Schools
cisco.comCisco Systems
pcsco.comProfessional Computer Systems
barrie.caCity of Barrie
ripta.comRhode Island Public Transit Authority
uncity.dkUN City (Building in Denmark)
bisco.intBoambee Industrial Supplies (Bisco)
haifa.eduUniversity of Haifa
smsnet.plSMSNET, Poland
fcmat.orgFiscal Crisis and Management Assistance Team
wiley.comWiley (publishing)
ciena.comCiena (networking systems)
belkin.comBelkin Public Schools
pqcorp.comPQ Corporation
ftfcu.corpFirst Tech Federal Credit Union Bank of Punjab
insead.orgINSEAD (non-profit, private university)
usd373.orgNewton Public Schools
agloan.adsAmerican AgCredit
pageaz.govCity of Page
jarvis.labErich Jarvis Lab
ch2news.tvChannel 2 (Israeli TV channel)
bgeltd.comBradford / Hammacher Remote Support Software Department of State Hospitals
dotcomm.orgDouglas Omaha Technology Commission
sc.pima.govArizona Superior Court in Pima County Prevention Society (IPS)
moncton.locCity of Moncton
acmedctr.adAlameda Health System
csci-va.comComputer Systems Center Incorporated
Redacted(law firm – redacted)
keyano.localKeyano College
uis.kent.eduKent State University
alm.brand.dkSydbank Group (Banking, Denmark)
ironform.comIronform (metal fabrication)
corp.ncr.comNCR Corporation
ap.serco.comSerco Asia Pacific
mmhs-fla.orgCleveland Clinic Martin Health
nswhealth.netNSW Health
mixonhill.comMixon Hill (intelligent transportation systems) de Formosa, City in California
siskiyous.eduCollege of the Siskiyous
weioffice.comWalton Family Foundation
ecobank.groupEcobank Group (Africa)
corp.sana.comSana Biotechnology
med.ds.osd.miUS Gov Information System
wz.hasbro.comHasbro (Toy company)
its.iastate.edIowa State University
cds.capilanou.Capilano University
e-idsolutions.IDSolutions (video conferencing)
helixwater.orgHelix Water District
detmir-group.rDetsky Mir (Russian children’s retailer)
int.lukoil-intLUKOIL (Oil and gas company, Russia)
ad.azarthritisArizona Arthritis and Rheumatology Associates
net.vestfor.dkVestforbrænding (Cloud based services, Israel)
central.pima.gPima County Government
city.kingston.Kingston City, Australia
staff.technionTechnion – Israel Institute of Technology
airquality.orgSacramento Metropolitan Air Quality Management District
phabahamas.orgPublic Hospitals Authority, Caribbean
parametrix.comParametrix (Engineering)
ad.checkpoint.Check Point
corp.riotinto.Rio Tinto (Mining company, Australia)
us.rwbaird.comRobert W. Baird & Co. (Financial services)
ville.terrebonnVille de Terrebonne
woodruff-sawyerWoodruff-Sawyer & Co., Inc.
fisherbartonincFisher Barton Group
banccentral.comBancCentral Financial Services Corp.
taylorfarms.comTaylor Fresh Foods
neophotonics.coNeoPhotonics (optoelectronic devices)
gloucesterva.neGloucester County
magnoliaisd.locMagnolia Independent School District
zippertubing.coZippertubing (Manufacturing)
milledgeville.lMilledgeville (City in Georgia)
digitalreachincDigital Reach, Inc.
thoughtspot.intThoughtSpot (Business intelligence)
lufkintexas.netLufkin (City in Texas)
digitalsense.coDigital Sense (Cloud Services)
wrbaustralia.adW. R. Berkley Insurance Australia
christieclinic.Christie Clinic Telehealth
signaturebank.lSignature Bank
dufferincounty.Dufferin County
mountsinai.hospMount Sinai Hospital
securview.localSecurview Victory (Video Interface technology)
weber-kunststofWeber Kunststoftechniek
parentpay.localParentPay (Cashless Payments)
europapier.inteEuropapier International AG
molsoncoors.comMolson Coors Beverage Company
fujitsugeneral.Fujitsu General
cityofsacramentoCity of Sacramento
ninewellshospitaNinewells Hospital
fortsmithlibraryFort Smith Public Library
dokkenengineerinDokken Engineering
vantagedatacenteVantage Data Centers
friendshipstatebFriendship State Bank
clinicasierravisClinica Sierra Vista
ftsillapachecasiApache Casino Hotel
voceracommunicatVocera (clinical communications)
mutualofomahabanMutual of Omaha Bank

† In this case, the company in question has reached out to me directly and asked that they not be listed. The company had performed a forensic review and believes they are not affected. In the interest of transparency, I can provide more details if contacted directly.


The Sh*tstorm of the Solardwinds hack

Pretty simple hack in concept – alleged Russian actors hacked an update package for Solarwinds Orion software (a sophisticated software for enterprise and institutional managing network resources).

This is an extremely crafty hack. An update package from Solarwinds Orion is uploaded onto the Solarwinds site. It’s even digitally signed.

According to Fireeye’s excellent writeup:

The trojanized update file is a standard Windows Installer Patch file that includes compressed resources associated with the update, including the trojanized SolarWinds.Orion.Core.BusinessLayer.dll component. Once the update is installed, the malicious DLL will be loaded by the legitimate SolarWinds.BusinessLayerHost.exe or SolarWinds.BusinessLayerHostx64.exe (depending on system configuration). After a dormant period of up to two weeks, the malware will attempt to resolve a subdomain of avsvmcloud[.]com. The DNS response will return a CNAME record that points to a Command and Control (C2) domain. The C2 traffic to the malicious domains is designed to mimic normal SolarWinds API communications. The list of known malicious infrastructure is available on FireEye’s GitHub page.

And now, it looks like the US Government may have been seriously compromised.

This is kind of a big deal…

Malwarebytes also has some good coverage as well, as does Prevasio


Is Docker secure? We executed all the containers in the Docker Hub to find out. What we found was troubling.

Docker – a way of building applications using containers (micro-apps that are used in Lego-block fashion to build larger programs) – is taking the enterprise world by storm. With customers that count among the top technology-forward companies in the world (Netflix, AT&T, PayPal, Snowflake, Verizon, Target, and many others), it’s become the new standard in deploying highly robust and distributed applications. At the core is Docker Hub, the main universe of Docker containers, with over 4 million container images*.

So with so many enterprises and institutions running Docker, security is a concern, and we have seen the growth of security vendors like StackRox, Aqua and Snyk to answer the call. And I’m sure plenty of these vendors have done analysis of containers, but in the end, these analyses will only be a static analysis, without taking into account all of the complexities and actions that occur when one actually runs a container.

So I’m happy that today, Prevasio, a company that I’m advising, has launched the results of the first and only analysis of the entire Docker Hub. What Prevasio did is new and important: They actually went through the entire Docker Hub and ran each container to see what happens.

This is not a small task; the company spent tens of thousands of dollars in computing power to perform this task. And the way the Prevasio Analyzer works is different than any other solution on the market: At its core is a sandbox which “detonates” (executes) a container to see what actually occurs during the runtime process. This sandboxing action allows us to get a unique view into what actually occurs when one executes a container. (This is the difference between behavior analysis and static analysis – a static analysis will use a signature to determine maliciousness but can never tell what an application will actually do. For a real look, one needs to analyze behavior.)

Some of the results are troubling:

  • 51 percent of all containers had “critical” vulnerabilities, while 13 percent were classified as “high” and four percent as “moderate” vulnerabilities.
  • Six thousand containers were riddled with cryptominers, hacking tools/pen testing frameworks, and backdoor trojans. While many cryptominers and hacking tools may not be malicious per se, they present a potentially unwanted issue to an enterprise.
  • Over 400 examples (with over 500,000 pulls) of weaponized Windows malware crossing over into the world of Linux. This crossover is directly due to the proliferation of cross-platform code (e.g. GoLang, .NET Core and PowerShell Core).

The full whitepaper is here, and Prevasio’s blog post is here.

But if you want to have some fun, Prevasio has all the data publicly available live on their site, at Feel free to look at the results for yourself.

*A container image is the template that creates the actual Docker containers.


How Russian Trolls Took Over Americans’ Instagram Accounts

Interesting overview by the WSJ on this one aspect of Russian trollage. 


It really comes down to beer (endpoint security redux)

572px-Dutch_beersUpdate: SentinelOne responds in the comments. Additionally, they claim (and I have no reason to doubt this claim) that the report I referenced was an older version that had incorrect information on the part of Tevanos.

As a follow-on to my recent post about endpoint security (see “A bomb just dropped in endpoint security…“), I thought I’d share some additional thoughts, conclusions and opinions.

It really comes down to beer. But more on that later.

First off, Joe Menn at Reuters wrote a story this morning. It’s a good story, fair and balanced.  Worth reading. There will likely be more stories as well.

Beating up vendors isn’t really my thing
In the recent blog post, there were a lot of slings and arrows thrown at a few endpoint security players. My blog got trolled quite a bit. I cleaned it up.

Cylance certainly came under heavy attack by some commenters, and I removed those comments.

I’m not interested in beating the crap out of some company; I’m really just interested in writing about stuff that I find is interesting.

On Cylance
There was a possible confusion that got propagated that Cylance was using VirusTotal directly in their product. I now have information that this may be incorrect.

Cylance was using VirusTotal, as they said in the Reuters article. It’s possible they were using the service to download samples to train their engine, not directly from inside their product. It’s also possible that they used VirusTotal to help detect malware.

I don’t know for sure, and that’s why I expect to be talking to them in the next several days.

Note that Cylance is not a bad group of people. There are many very good people working there, and they run a good business. I’ve challenged them to get more public tests, and I hope they do so. So far, there have been two tests that I know of — a condoned test by Av Test, and another test, where AV Comparatives/Effitas had to basically break the rules to try and get a copy of Cylance (one can’t just download Cylance and test it, as Cylance keeps its trials very closely monitored).

So testing more, and being more public, would be a good thing.

At any rate, peace, people.

Other endpoint players
I don’t know about other endpoint players. I know SentinelOne, for example, has been open about using VirusTotal.

Specifically, this report (PDF) on SentinelOne’s capabilities in healthcare, highlight this point:

SentinelOne ensures that it is always up to date, checking file hashes against reputable sources such as Virus Total. Using this method, SentinelOne’s platform does not suffer the traditional time lapse in needing to push out new definitions…

[Edit – As mentioned earlier in this blog, SentinelOne tells me this quote is from a PDF of an older document that incorrectly noted VirusTotal as a source and has since been corrected.]

Regarding Palo Alto Networks and CrowdStrike, I really don’t know the involvement of these players. But as the Reuters article mentions, they have been users of VirusTotal (and have not been participating with the community). That’s not to say they’re bad people or have bad products (CrowdStrike and Palo Alto both make excellent products). It’s just something that has been addressed.

Endpoint security
My knowledge of the endpoint security market comes from the fact that I’ve been there in the trenches.

I’ve presented at conferences like VirusBulletin and submitted papers, etc., but that’s the fun stuff. Actually being in the day-to-day sweating bullets to try to keep your customers protected is a very difficult task.

You see, we created a full-stack antivirus product at my last company (Sunbelt Software’s VIPRE) and I personally ran the antivirus lab for some time.

When we released the product, it was probably mediocre, and we avoided tests. It got better when we started opening up to tests, and in fact, it got quite good (and then I sold the company and, well, I don’t recommend the product anymore).

We were on VirusTotal, and it was painful to see us miss detections. But at least we saw ourselves for what we were, and made our product better as a result.

Is there a “next-generation?”
The truth is there are only so many ways to skin a cat. The former Eastern Bloc countries were famous for producing some of the world’s most brilliant mathematicians and now we have companies like Kaspersky and BitDefender with just those same type of people. Yet even they have a tough time of it (no matter what they say publicly). It’s not an easy business, trust me.

I don’t entirely buy a lot of the “next-gen” security arguments. I think that there is room for innovation, but I’ve seen companies like Malwarebytes (of which I’m a board member and incredibly biased toward) and (recently) Symantec do some very impressive work in detections, without resorting to anything of the next-generation type. It really comes down to a lot of hard work, block-and-tackling type of stuff.

So, it’s no surprise that people do whatever it takes to get the best result possible. If this means using VirusTotal to do a hash lookup (which IMHO is fairly silly, since polymorphism makes hash lookups far less useful than people might think), or good old fashioned PR to paint lipstick on a pig, well, so be it.

The key is openness. If you have something special, open it up to the world for them to look at, to test, to validate. Be a part of the community and give back to it. It will make your product much better.

Which comes to the beer
Information sharing in security happens around conferences and beer. (When I brought in a new lab manager for my research team years ago, I urged him to spend as much time as possible going to conferences and drinking beer with other experts. He didn’t object).

Perhaps that is a bit tongue-in-cheek, and it’s not that we’re a bunch of alcoholics (well, mostly not), it’s simply that a lot of information sharing happens at conferences, when experts talk to each other freely. The swords of competition are put down briefly, people open up, and you hear a lot of interesting things.

And what I hear around the tables is what is reflected in some of my blog posts. There is good data, but it can’t be substantiated and it won’t ever be confirmed. So, I’m sorry I can’t be specific, despite many of you emailing me for much more detail than I am prepared to give. Trust is everything in security.

But beer? Yes, we can all share that freely. So, here’s to beer (in my case, the ever excellent Buckler Non-Alcoholic).

You can see who VirusTotal credits here. 


A bomb just dropped in endpoint security… and I’m not sure anyone noticed

Pay no attention to the man behind the curtain…

Update: Reuters now has the story

Update 2: I’ve updated this post with additional information, here. 

VirusTotal just dropped a major bomb, and only people deep in the endpoint security ecosystem understand the ramifications of this announcement.

If you’re involved in endpoint security to any degree – as a customer or an industry person – you need to understand what just happened. It’s really, really big.

A bit of background.
VirusTotal is a multi-engine virus scanner. You upload a file, and it passes the file to a large number of commercial antivirus products, and it tells you which engines detected the file as malicious.

While there are other tools available, and some have come and gone, VirusTotal is the big dog in the space. It’s owned by Google, has massive computing and resource power and everyone in the security industry uses it.

VirusTotal shares the results with subscribers. So, you can pay to get extensive and detailed information on what has been detected at any moment of the day, and who detected it. 

How antivirus companies use VirusTotal to make better detections.
It’s common practice of antivirus companies to use VirusTotal as a tool to make better signatures.

For example, if a researcher finds that two high quality antivirus engines detect a file as malicious, he/she has a high confidence that it’s actually malicious without further analysis. As an antivirus researcher, it saves an enormous amount of time.

Now, there’s absolutely nothing wrong with using VirusTotal results in research, and many antivirus companies use VirusTotal to supplement their own labs. They get samples from VirusTotal, and along with the samples, what engines detected them. If they find that a couple of high quality engines are detecting a file, they can easily add the detection to their own signatures without much further thought.

Now, there’s a next step. You could set up an an API integration with your product. If you scan a user’s machine and find an unknown file, you could upload it through an API to VirusTotal and get a disposition on the file –who detects it. From this data, you can flag a file as malicious.

In other words, you can use VirusTotal to create your own antivirus program. Easily. 

Until now. 

It’s fine to use other engines. If you’re also contributing.
Using other engines to improve your detection rate is completely fine. If you’re also contributing back to the community yourself. In other words, if your antivirus product is also one of the participating antivirus engines.

The dirty little secret
And here’s the dirty little secret that very few people know. There are a number of endpoint products that use VirusTotal to determine if a file is malicious. Without any contribution to the communityWithout giving anything in return. 

They simply pay VirusTotal a subscription fee, and receive the information.

And some of these companies have been getting a lot of attention for their supposed prowess. But for some mysterious reason, they refuse to put their own engines on VirusTotal. Could it be because they don’t want to contribute back? Maybe. Or it could be that they just don’t want everyone else to see how poorly their products actually perform.

Unfair? Yes.
Using VirusTotal information without any contribution back to the community is patently unfair. The people who are actually writing detections are sharing their results with the rest of the community, while a small group of endpoint products have been boasting of their extraordinary abilities, while working off the backs of other researchers. 

So as a customer, perhaps you can ask the next endpoint security vendor if they’re on VirusTotal. If they are, they’re contributing to the antivirus community. If they’re not, they’re not. Whatever their PR story, that’s the simple truth.

Until now.
Well, the world just got a bit brighter for the many endpoint security companies that actually contribute to VirusTotal: Because VirusTotal just announced that they are requiring that all scanning companies that use their service must integrate their engines into VirusTotal. Furthermore, “…new scanners joining the community will need to prove a certification and/or independent reviews from security testers according to best practices of Anti-Malware Testing Standards Organization (AMTSO).”*

It’s big news. It levels the playing field. No longer will antivirus companies see their hard work taken by some sexy startup that’s raised millions of dollars on the false promise of “next generation” endpoint or other such nonsense, while bashing the very companies that they’re effectively stealing the intellectual property of. And perhaps, we’ll see what their products are really made of. Because without VirusTotal as a crutch, companies that rely on it are going to see their detection rates take a hit.

Poetic justice, indeed.

What does this mean for the IT manager?
If you’re an IT manager who has been duped by sparkly marketing materials to buy-in to one of these “next-generation” endpoint products, take a hard look at their actual detection capabilities. If they’ve been using VirusTotal results but not contributing back, their ability to detect malware just took a potentially serious hit. This is serious.

You don’t have to believe the marketing hype. Setup a virtual machine that’s separated from your corporate network, and go to a site like MDM to find all kinds of nasty stuff. In the words of Ronald Reagan, “trust, but verify”. One nasty piece of malicious software (especially ransomware) can have serious consequences.

In closing
My compliments to the VirusTotal team for seeing this disparity and unfairness and taking such swift action. A class act, indeed.

And now, perhaps, we can all finally see what is really behind the curtain.

* Disclaimer: I am a board member of Malwarebytes (a contributing member to the VirusTotal community), and an advisory board member to AMTSO.  The opinions in this blog post are my own and are not connected to these two organizations.


What the Kaspersky breach tells us about the state of antivirus

Yesterday, Kaspersky announced that some of its internal systems had been breached. While this may have created a sense of Schadenfreude in some parts of the security community, Kaspersky has handled the situation quite well.  Instead of other companies that have suffered a breach, Kaspersky worked straight from the crisis management playbook — full disclosure, plenty of information and a plan. Kudos.

As Graham Clulely says: “In short, it handled what could have been a corporate crisis well – and reassured customers and partners that their data was safe, and the integrity of its security products had not been compromised.”

(Although one can’t help but wonder at the timing. According to the press release, the malware was found in “early spring 2015”, but the announcement is coming on June 10th — just a few weeks away from the official start of summer…)

Kaspersky is being up-front, but they are also spinning this as a research item. And that’s okay, because it is some fascinating research. This is a very interesting new malware variant, and quite sophisticated, quite likely tied to state-sponsored activities.

But they just can’t help being Kasperskyish:

From a threat actor point of view, the decision to target a world-class security company must be quite difficult. On one hand, it almost surely means the attack will be exposed – it’s very unlikely that the attack will go unnoticed. So the targeting of security companies indicates that either they are very confident they won’t get caught, or perhaps they don’t care much if they are discovered and exposed. By targeting Kaspersky Lab, the Duqu attackers probably took a huge bet hoping they’d remain undiscovered; and lost.

(Yeah, that explains the Schadenfreude part.)

Anyway, the bigger story is the state of the antivirus detections on the day following Kaspersky’s announcement.

Considering that most antivirus vendors practice what I call “hash-whoring”, where hash detections from VirusTotal or internal scans are dumped wholesale into their databases (explaining the massive size of today’s antivirus engine), the poor detection state of this variant is surprising. (Incidentally, I’m not condemning this practice — it’s a very useful stop-gap until a detection team can make a good detection — nevertheless, it’s abused way too much, especially by poorer quality, also-ran engines.)

We know what this piece of malware looks like, because Kaspersk published the complete Indicators of Compromise (IOCs). So, we can just go to VirusTotal and check the detection status:






And so on.

So, just for fun, I’ve published the hashes below, hyperlinked to VirusTotal.  You can click on them as the week progresses to see the state of detection of your antivirus product.

Action loaders:




(And, also remember to blacklist the C&C IPs: and

Is this blog post another tired rant against antivirus? Absolutely not. AV isn’t dead. It’s part of a valid belt-and-suspenders approach to security.

But, a day later, and we still see poor detection? Yeah, that part sucks.

business advice Humor security

Focus on the importances. Empower others. Be happy.

UntitledIf you’re heavily stressed as a business leader, the business is running you — not the other way around. Chances are you’re not prioritizing correctly, and you’re not delegating.

I’ve worked with CEOs who put in an insane amount of hours and don’t do any better than CEOs who work a fairly normal schedule (granted, usually 50–60 hours a week).

One could describe a leader as someone who establishes and communicates clear goals, gets the right people in place, gets everyone working toward these goals and focuses on what’s important.

Culture is an additional ability of leadership. Culture is less important, actually, than fanatical execution on a clear set of goals. Ping pong tables, beautiful offices — nice — but not vital.

The core is figuring out where you’re going, getting the right people going in the same direction, and focusing on what’s important.

Sounds easy, but it’s an art.  It’s why great CEOs are paid a lot of money and are in high demand, because it doesn’t come intuitively or naturally to a lot of people.  However, it can be learned.

Teaching leadership skills, however, isn’t the purpose of this blog post.  I’m just going to tell you what’s important.

There are just a few things that you have to do really, really well in this business.  If you do those well, everything else follows.

Many years ago, one of my early mentors told me, “if you just focus on creating a great product, support it well and do a good job on PR, you should do just fine.”

Not bad advice.  I’ll expand on it with a bit of my own experience.

Here is the scale of importances in running a product or services business.

1. The product or service.
2. The quality of the product or service.
3. Support/customer service
4. PR and marketing
5. Sales
6. HR
7. Finance/administration/legal

Assign KPIs to each area (you can’t manage what you can’t measure…). At the beginning of every week, go through each of these areas by yourself. And then go through these with your senior staff at your Monday morning staff meeting.

The funny thing is that as an executive, you may find yourself spending a tremendous amount of time keeping people focused on doing the important things. And, you may find yourself burdened down with things that aren’t that important. People add complexity to everything they do.  It’s a natural tendency, but it generally means that they are not confronting what really needs to get done (either because they don’t know, or because they don’t understand something).

If you establish an organization with this set of importances, you’ll increase your chances of doing well.

The mistakes I’ve made are when I’ve reversed the priority — too much emphasis on finance, or sales, etc. The product (or service) is the most important thing to focus on (read my other post, The Product is All). Give the accountants the problem of worrying how to book the revenue. Give the sales and marketing guys the problem of actually getting the revenue.  And get the product guys firmly lined up with what’s needed and wanted from the market, and delivering it.

And lead a less stressful life.

facebook security

Facebook hoaxes

Incredibly, I still see people post these silly disclaimers that all of their information on Facebook is private.It’s meaningless. However, you can take some good steps to insure your privacy by reading the tips in this good overview of Facebook privacy by Ronnie Charrier.


Fatal error: Uncaught wfWAFStorageFileException: Unable to save temporary file for atomic writing. in /home/eckelberry1966/public_html/blog/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/storage/file.php:34 Stack trace: #0 /home/eckelberry1966/public_html/blog/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/storage/file.php(658): wfWAFStorageFile::atomicFilePutContents('/home/eckelberr...', '<?php exit('Acc...') #1 [internal function]: wfWAFStorageFile->saveConfig('livewaf') #2 {main} thrown in /home/eckelberry1966/public_html/blog/wp-content/plugins/wordfence/vendor/wordfence/wf-waf/src/lib/storage/file.php on line 34