SQL Server Blog

“Ex-Admin Deletes All Customer Data and Wipes Servers”

My headline is borrowed from a headline on bleepingcomputer.com. This just happened to a Dutch hosting provider in 2017. You can read the article. Actually, you must read the article. If you didn’t here’s the basic gist:

  • An administrator quits or was fired or whatever.
  • He or she didn’t get their access terminated right away.
  • He or she carried some sort of a grudge, they connected in and deleted all customer data. They wiped all servers.
  • This host had a horribly scary and sad status page. It’s story is “We likely lost your data, completely, we’ll try and get it back but we aren’t hopeful. We’re sorry. If you still want to work with us, we’ll talk about compensation, we’re in the process of trying to get back what we can.
What are you doing to prevent this? What would you do if you got this page on your host or provider?
This is their current homepage status message. What are you doing to prevent this? What would you do if you got this page on your host or provider?

This Is About YOU

This post isn’t about Verelox. They have a disastrous situation and I literally just prayed for their entire staff and management (and for that of their customers) – they’ll be on my mind these next few days and in my thoughts and prayers. It’s a crappy place to be. They owned their really bad oops. They’ve been incredibly transparent, something I’m fond of – I spoke about that in a blog post many years ago about why Bill Clinton was impeached (it’s a technical blog post).

No. This blog post is about YOU, thoughAnd don’t think you get off the hook because you aren’t a host provider. And don’t now think you are off the hook because you are “just” a host provider. There are some takeways that you both need to have right now. That we all need to have.

When someone passes too early (what is too early.. That’s always a strange phrase), you’ll see someone say something like “hug your kids, and spend time with your family. Well here – If you do nothing else – Go check your backups on Monday. Hug your backups. Check your servers. Test a restore.

I have a question for you. This is sort of my “are you alive?” question. “Just hearing this story, how do you feel?”  There are three feelings you could have, probably more, but this is my post:

  • At Peace and Confident – This could be good, if that confidence is well placed. Read on to see if it is.
  • Nervous, A bit queasy – This is fine. This is good. Before starting a SQL Server consultancy and managing a team, building services and providing all the services we provide, I was a DBA for many years. many. Paranoia is a healthy attitude if your role is responsible for this stuff. That’s good. Read on to see how you can turn this into action.
  • Indifferent – Well if you are a painter and you don’t use a hosted provider for anything (not even your e-mail or pictures and backups of documents) then that’s fine. If you are  CIO, CTO, CEO, Manager, Director, DBA, Backup Admin, MSP, Host, etc – that attitude scares the living daylights out of me.  You probably shouldn’t read on, because it’s stuff that just doesn’t matter from your perspective. Just copy and paste that message I highlighted above. You may need it someday..

I actually expect most folks to be in that second camp. Even the people who know that they know and trust that they are knowing correctly, probably still get a little faint when reading about these things. That’s good. That’s healthy.  So this post is for you. And truth be told, it’s for me and my team and my clients, too. And it’s important. Your company’s future could be on the line. There will undoubtedly be businesses which cease operations because of this event. They will NOT recover. Their customers will not forgive them. They will be closed down. Maybe sued. Maybe bankrupt.

This is not stuff to play around with. So my main point here is two fold – 1.) To scare you into caring and worrying. 2.) To give you some proactive steps to direct this fear towards so it stays healthy and productive. Please don’t be like Johnny as played by Stephen Stucker in Airplane!. . .

Enough Preamble.

Here’s a list of some things you can do. This isn’t exhaustive. But start thinking about more what if’s and what about’s and you’ll add to it.

If You are a Hosting Customer

So this is really all of us. Your host could be your own data center and your own team. It could be Azure. It could be AWS. It could be Rackspace or a host like Verelox.

  • Realize you are your own advocate – So yes you are paying some dollars per month for this great service. But you are one customer. Of many. You are your own advocate. You have to worry about yourself. Be loud. Be worried. Be on top of details. Double check everything.
  • Plan for them to go away right now – If your host had an angry ex-employee, a hacker, a multiple data center fire – what would happen to your data? When was the last time you confirmed you had an off site, off location backup someplace other than their environment? When was the last time you tested or initiated a restore test? When was the last time you performed a DR failover. I have some customers who use AWS S3 storage as their primary storage for certain web tasks and files. They have a process to backup to Azure Blob storage and keep them in sync. Paranoid? Over the top? Maybe. But when the S3 East outage happened a month ago they were enjoying a normal day. Not all customers in S3 East had a normal day that day. Don’t you like normal days? I do.  Simply having off-site backups isn’t enough to get back online, but it’s a heck of a lot better than telling your killer pet cafe CRM SaaS customers, “sorry. we’re going to be offline for awhile, oh yeah and we lost your data, forever, kthxbai”.. It would be far better to say “so we have your data, you lost a day (or less), we’re going to be down while we get our images online or go through our runbooks to bring your site up, but we’ll be up in 2 days, and your data is safe.”
  • Look at the policies – ask about them – What happens when an employee leaves? Show me the policy. Show me an audit proving you follow your policy. If you can’t, why am I giving you all this money?
  • What about your system administrators – A lot of clients in some hosted environment also have domain and local administrators. Who is in there? What happens when they leave?
  • How much money will it cost you to lose all your data? So. Why aren’t your angel investors and founders willing to spend 1/30th (or even 1/10) of that cost to prevent that from happening? It’s insurance, I get it, we all pay it, we all hope to never use. But man, when that car accident comes out of nowhere, it’s nice to have it. No one ever complains about how much they spent on health insurance when they get something that costs so much. No one ever complains that the life insurance premiums were way too high when the family can go on with life…
  • Think about what can go wrong – Even just thinking about what can go wrong is a big deal.
  • Trust and Verify – Have you ever seen your backups? have you participated in DR drills? How do you know they are really doing what they say? Remember it’s not just angry employees who quit you have to worry about. It’s incompetent ones who stay, too.

For The Hosts

And that’s for the folks who self host also.

  • Fix your “goodbye” policies – Once someone leaves? The moment they leave. Their accounts have to go with them. These things don’t take much time. An angry or bitter employee can do a lot of damage in minutes with scripts. A crazy employee can have those scripts ready long before they quit.
  • Have Layers – Why do you have to have every employee in the sysadmin role have permissions to all aspects? What if you had a process to grant elevated permissions and needed permission from another group. Sure there is always a next level up problem – but some layers of security, process and documentation can go a long way.
  • Offsite backups – Do you do offsite backups? Do you do them to some stable media that is on a disconnected network or a system that doesn’t have the same exact access policies? Yes someone creative can still wreak havoc, we’ll never stop every bad disaster – our goal is to prevent most and make it really hard. Someone can easily break into just about any house out there – we do locks and alarms to help deter.
  • Read the stuff above – All those tips help.
  • Get audits and health checks – This isn’t a shameless plug for our SQL Server Health check – there are a myriad of companies that give great health checks. But get someone outside to look in. It’s not that you don’t trust your team, it’s that you want to verify, you want to get an outside eye. The cost of this is even less than the insurance of offsite backups and DR plans I ranted about above. Get that outside review.

So – it’s good they were transparent. It’s good they are trying. It’s sad it came to this. It could have been prevented most likely. These customers could have had their own backups someplace else – and I hope many did, but I bet many don’t. I hate learning opportunities like this, but it’s what it is. Use that feeling to take some action.

Mike Walsh
Article by Mike Walsh
Mike loves mentoring clients on the right Systems or High Availability architectures because he enjoys those lightbulb moments and loves watching the right design and setup come together for a client. He started Straight Path in 2010 when he decided that after over a decade working with SQL Server in various roles, it was time to try and take his experience, passion, and knowledge to help clients of all shapes and sizes. Mike is a husband, father to four great children, and a Christian. He’s a volunteer Firefighter and EMT in his small town in New Hampshire, and when he isn’t playing with his family, solving SQL Server issues, or talking shop, it seems like he has plenty to do with his family running a small farm in NH raising Beef Cattle, Chickens, Pigs, Sheep, Goats, Honeybees and who knows what other animals have been added!

Subscribe for Updates

Name

8 thoughts on ““Ex-Admin Deletes All Customer Data and Wipes Servers””

  1. From the first time I learned how to do a SQL backup to data backup, I’ve been preaching this topic to no end. Most of the time I feel that I am just spitting words into the wind as most companies don’t get it. Or maybe they do get it and they just want to ignore it. Disaster recovery is a big ticket item that companies should not overlook.

    Now, about those policies. I wish companies would really stick to the policies that are in place and create policies that are not present for this type of situation. Too many times have I seen someone leave a company I worked for, come back and talk to me a week later saying they are still getting emails, notifications and other items that should have been shut off. “Oh, look! I still have access to the remote server that I was working on. Think they will notice if I remove my software?”

    I don’t know what is worse. A company with policies who do not enforce them or people who have no morals or better judgment.

    Reply
    • That’s a great philosophical question, Ryan. I don’t know what’s worse either. I actually think maybe the former. Because they know and should know and still got it wrong.

      Reply
  2. I thought this article is going to teach us strategies of recovering the deleted data. Anyways, prevention is always better than cure.

    Thanks!!

    Reply
    • Hey Vikas,

      Thank you so much for the comment. Sure I’d love to write a post about how to get data back from your failed hosting provider, alas that’s a tall order with a lot of detours into “it depends” areas. Bottomline, prevention is far better than the cure here – and for some folks there may be no cure.

      Reply
  3. How about anyone addressing whatever BS did this employee had to suffer through while being employed in this company that made him so frustrated in the first place?

    Reply
    • Hey Tom,

      Thanks for the comment. I’m not fully sure how to respond. For sure employers need to treat their employees in an equitable way and create non-hostile working environments. But this is a technical blog. About SQL Server and backups and all. Some professional development but more for the techies. Absolutely agree – it stinks when an employer can install so much frustration in an employee – but it is never okay. It is never good. It is never normal for an employee to take such actions. Quit. And take your great mind someplace else and punish them by not having access to you. Trashing all of your employers data and their backups? That’s not justified. And this post was really a reminder to companies who rely on hosts and assume all is good. You bring up an excellent point and there is probably much there. Or maybe not. Hard to say not knowing any of the players involved… Plus this company is in a nation with fairly favorable to the employee employment laws.

      Reply

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share This