I have read about ransomware attacks, how destructive they could be, how some organizations recovered by paying the ransom and others by restoring from backups. I’ve also read that some organizations were unable to recover from a ransomware attack and had to close their doors. But these were always stories “out there” and happening to somebody else.
But for the last few weeks, I am in the midst of assisting a client that is recovering from a ransomware attack. Fortunately, this attack will not result in the organization closing its doors as the IT staff here have worked long hours to ensure that is not the case. However, the ongoing road to full recovery has been a long, maddening, tough, hard, and exhausting road. In all honesty, at times I’ve thought the better alternative would be to quit IT altogether in order to pursue a new career as a curator at the National Air and Space Museum. Sure I have absolutely no experience but I do enjoy aviation history and I would work hard to keep the entire collection spotless! Alas, the Smithsonian hasn’t called and so the fight to recover from the ransomware attack continues.
First, Are You Your Own Worst Enemy?
Though the ransomware attack was launched in full measure in early February, the groundwork ensuring its effectiveness had been laid for years. This organization started making their goods about 60 years ago and as such, they could make their products without the everyday use of computers. Thus, computers didn’t matter much. And as you may imagine, the first executives and employees didn’t have desktops, laptops, thin clients, or tablets. They didn’t have email, electronic spreadsheets, or databases but nonetheless, they could produce their goods and produce them well. But as these executives navigated the organization into the digital age, it wasn’t lovingly embraced but reluctantly fought against as a matter of pride. If you walk past some of the older offices and meeting rooms, you can almost hear them say, “I used to build XYZ by hand after walking to work uphill both ways for 18 miles and never had need for a computer (said with a measure of disgust).” You may be able fight against some waves for a long time but others can only be held at bay for so long. As is the case for virtually all organizations throughout the 80s-90s, computers became commonplace in this organization and even required to do business, yet the early bosses maintained the following view regarding the ever increasing use of computers, “Since I ‘have to’ do this, buy your fancy computers but do so as cheaply as possible.” And as the years passed, and though technology systems were implemented that were instantly vital to ensure the continued production of XYZs, the organization from the executive level, continued to spend as little as possible on IT despite the pleas of those supporting these systems to move away from this mindset.
As digital age threats such as viruses, malware, spyware, and ransomware increased in their complexity and securing your IT assets became more and more important, the thoughts of executives, as exhibited by their actions, perceived IT security as something of an optional nice-to-have luxury item….our IT assets are secured by a perimeter firewall right? Besides, we just make XYZ widgets in small town USA, who’s going to take the time to target us?
This is really an extremely obvious statement but I share it nonetheless….every person and organization is a potential ransomware target as ransomware attacks are not limited in scope to large organizations or places like the Pentagon or credit card vendors. Unlike people, ransomware does not discriminate based on company size, name, or income but will happily be used by attackers to demand a ransom payment from whomever or wherever they can.
Perhaps it was inevitable, but the bill for providing IT as cheaply as possible came due one early February morning.
The Importance of Having a Plan…I beseech you to consider your response to a hypothetical ransomware attack prior to experiencing an actual ransomware attack!
I may be giving XYZ Widget Maker too hard a time for the lax manner in which their leadership had historically run their IT department. I suppose even the most controlled and secured environments, which are providing services and resources to some user population (internal or external), are susceptible to ransomware. So what would you do if a ransom note popped up on the screen of your VM as you were troubleshooting an perceived application issue around midnight Sunday/early AM Monday?
Perhaps ask what would you be allowed to do if a ransom note popped up on your screen in the middle of the night and when you’re completely alone…just you and the ransom note. At that point, you must make a decision and the speed at which you make it AND its validity will have a substantial impact on the breath and scale of, and the recovery from, the virus/malware/ransomware attack. And this is really the point of this post. I’m not going to use this post to go into the weeds and details of everything we have been done and experienced the last few weeks. Granted, I may post more as we improve, debrief, and move forward as a team and as I’m able to, individually, step back from this experience, putting away the immediate anger and really ponder what worked and/or what didn’t. But right now, my earnest desire is to strongly encourage you to spend time considering how you and your organization would respond in the event of a ransomware attack.
What I will say is this….the ransom note popped up on a computer screen around midnight after a member of the IT team had spent the day troubleshooting a seemingly unrelated application issue. Little did anyone know, that the application issue this person was troubleshooting was likely the launch of the ransomware attack. At midnight, this person was tired and alone and faced with a decision that has/had huge ramifications because there was no plan in place….calamity strikes at the least opportune time. I recently came across a blog post entitled “Making the Right Decision When Tired Is Like Choosing an Option at Random” and I couldn’t agree more. The author isn’t saying that you will always make poor decisions when tired but simply points out that when you’re tired, you’re not as sharp as you would otherwise be and that’s a dangerous place to be when deciding what actions to take when confronted with a ransom note in the middle of the night.
My recommendation is that you create an “emergency response” team and work with one another, as well as your leadership team, to formulate an emergency response plan. And with the plan developed, create hard copies, distribute to and train all necessary personnel, and then put one in a central “in case of emergency break glass” location. This way, at least in some measure, you won’t have to make a decision because it’s already been made; you (or someone else) just need to execute the plan. Certainly the questions below are not all inclusive but I just add them as food for thought on determining your initial response to an emergency situation:
- Do you instantly pull the plug on all network switches, firewalls, and servers?
- At some point however, you’ve got to plug them back in to triage the system. When and how do you do that and who is involved?
- Who do you contact first, second, and third? And how?
- Once contact is made, who MUST come to the office (or wherever) confront the emergency?
- Where do you go? Where do you setup your base of operations?
- Do you have ransomware insurance coverage? If so, do you have the means to contact a security specialist firm 24/7? This would be good information to know.
- How do you manage the response?
- How often do you hold status meetings?
- How do you communicate with the organization at large to inform employees when they may be able to work?
Final Thoughts
Here are some final points to consider:
- Ransomware has become a billion-dollar business and that means every person and organization is a potential ransomware target. Beware.
- Are you and/or your organization really doing its utmost to ensure the security of your systems?
- I was reading the Veeam Community Forums Digest and saw this “Another company had to learn that storage-based replication is not a backup, the hard way”.
- I can vouch for the validity of that statement in that storage-based replication was used to move backup data offsite and as you may guess, the backup data that was corrupted in the primary site was corrupted in the secondary site. So what do you do if you have no backups on which to draw? For example, are you using Veeam backup copy jobs to move data, and not just replicate it, offsite?
- While discussing a recent ransomware attack against his company, Ian Oxman, the CMO of Apex Human Capital Management states the following:
- “We paid the ransom, and it sucked.” –Yep! It sucks!
- “When they encrypt the data, that happens really fast,” he said. “When they gave us the keys to decrypt it, things didn’t go quite as cleanly.” –Also yes! This sucks too!
- I made a “break glass in case of emergency” comment earlier, but don’t create a response plan, check it off your to-do list, and never look at it again. Periodically evaluate and change it if necessary.
I don’t claim to be a ransomware response expert. But based on recent experience, I’m just trying to help prepare for the worst by giving you something to chew on, to think about, and consider….hopefully for your own good. Trust me, when a ransom note appears on your screen at midnight, and you’re alone, you’re tired, and your stress level instantly skyrockets, you’ll be thankful that you don’t have to make an immensely important decision, that may result in the closing of your organization, at the most inopportune time because you spent time developing an emergency response plan.
