My headline is borrowed from a headline on bleepingcomputer.com. This just happened to a Dutch hosting provider in 2017. You can read the article. Actually, you must read the article. If you didn’t here’s the basic gist:
- An administrator quits or was fired or whatever.
- He or she didn’t get their access terminated right away.
- He or she carried some sort of a grudge, they connected in and deleted all customer data. They wiped all servers.
- This host had a horribly scary and sad status page. It’s story is “We likely lost your data, completely, we’ll try and get it back but we aren’t hopeful. We’re sorry. If you still want to work with us, we’ll talk about compensation, we’re in the process of trying to get back what we can.
This Is About YOU
This post isn’t about Verelox. They have a disastrous situation and I literally just prayed for their entire staff and management (and for that of their customers) – they’ll be on my mind these next few days and in my thoughts and prayers. It’s a crappy place to be. They owned their really bad oops. They’ve been incredibly transparent, something I’m fond of – I spoke about that in a blog post many years ago about why Bill Clinton was impeached (it’s a technical blog post).
No. This blog post is about YOU, though. And don’t think you get off the hook because you aren’t a host provider. And don’t now think you are off the hook because you are “just” a host provider. There are some takeways that you both need to have right now. That we all need to have.
When someone passes too early (what is too early.. That’s always a strange phrase), you’ll see someone say something like “hug your kids, and spend time with your family. Well here – If you do nothing else – Go check your backups on Monday. Hug your backups. Check your servers. Test a restore.
I have a question for you. This is sort of my “are you alive?” question. “Just hearing this story, how do you feel?” There are three feelings you could have, probably more, but this is my post:
- At Peace and Confident – This could be good, if that confidence is well placed. Read on to see if it is.
- Nervous, A bit queasy – This is fine. This is good. Before starting a SQL Server consultancy and managing a team, building services and providing all the services we provide, I was a DBA for many years. many. Paranoia is a healthy attitude if your role is responsible for this stuff. That’s good. Read on to see how you can turn this into action.
- Indifferent – Well if you are a painter and you don’t use a hosted provider for anything (not even your e-mail or pictures and backups of documents) then that’s fine. If you are CIO, CTO, CEO, Manager, Director, DBA, Backup Admin, MSP, Host, etc – that attitude scares the living daylights out of me. You probably shouldn’t read on, because it’s stuff that just doesn’t matter from your perspective. Just copy and paste that message I highlighted above. You may need it someday..
I actually expect most folks to be in that second camp. Even the people who know that they know and trust that they are knowing correctly, probably still get a little faint when reading about these things. That’s good. That’s healthy. So this post is for you. And truth be told, it’s for me and my team and my clients, too. And it’s important. Your company’s future could be on the line. There will undoubtedly be businesses which cease operations because of this event. They will NOT recover. Their customers will not forgive them. They will be closed down. Maybe sued. Maybe bankrupt.
This is not stuff to play around with. So my main point here is two fold – 1.) To scare you into caring and worrying. 2.) To give you some proactive steps to direct this fear towards so it stays healthy and productive. Please don’t be like Johnny as played by Stephen Stucker in Airplane!. . .
Enough Preamble.
Here’s a list of some things you can do. This isn’t exhaustive. But start thinking about more what if’s and what about’s and you’ll add to it.
If You are a Hosting Customer
So this is really all of us. Your host could be your own data center and your own team. It could be Azure. It could be AWS. It could be Rackspace or a host like Verelox.
- Realize you are your own advocate – So yes you are paying some dollars per month for this great service. But you are one customer. Of many. You are your own advocate. You have to worry about yourself. Be loud. Be worried. Be on top of details. Double check everything.
- Plan for them to go away right now – If your host had an angry ex-employee, a hacker, a multiple data center fire – what would happen to your data? When was the last time you confirmed you had an off site, off location backup someplace other than their environment? When was the last time you tested or initiated a restore test? When was the last time you performed a DR failover. I have some customers who use AWS S3 storage as their primary storage for certain web tasks and files. They have a process to backup to Azure Blob storage and keep them in sync. Paranoid? Over the top? Maybe. But when the S3 East outage happened a month ago they were enjoying a normal day. Not all customers in S3 East had a normal day that day. Don’t you like normal days? I do. Simply having off-site backups isn’t enough to get back online, but it’s a heck of a lot better than telling your killer pet cafe CRM SaaS customers, “sorry. we’re going to be offline for awhile, oh yeah and we lost your data, forever, kthxbai”.. It would be far better to say “so we have your data, you lost a day (or less), we’re going to be down while we get our images online or go through our runbooks to bring your site up, but we’ll be up in 2 days, and your data is safe.”
- Look at the policies – ask about them – What happens when an employee leaves? Show me the policy. Show me an audit proving you follow your policy. If you can’t, why am I giving you all this money?
- What about your system administrators – A lot of clients in some hosted environment also have domain and local administrators. Who is in there? What happens when they leave?
- How much money will it cost you to lose all your data? So. Why aren’t your angel investors and founders willing to spend 1/30th (or even 1/10) of that cost to prevent that from happening? It’s insurance, I get it, we all pay it, we all hope to never use. But man, when that car accident comes out of nowhere, it’s nice to have it. No one ever complains about how much they spent on health insurance when they get something that costs so much. No one ever complains that the life insurance premiums were way too high when the family can go on with life…
- Think about what can go wrong – Even just thinking about what can go wrong is a big deal.
- Trust and Verify – Have you ever seen your backups? have you participated in DR drills? How do you know they are really doing what they say? Remember it’s not just angry employees who quit you have to worry about. It’s incompetent ones who stay, too.
For The Hosts
And that’s for the folks who self host also.
- Fix your “goodbye” policies – Once someone leaves? The moment they leave. Their accounts have to go with them. These things don’t take much time. An angry or bitter employee can do a lot of damage in minutes with scripts. A crazy employee can have those scripts ready long before they quit.
- Have Layers – Why do you have to have every employee in the sysadmin role have permissions to all aspects? What if you had a process to grant elevated permissions and needed permission from another group. Sure there is always a next level up problem – but some layers of security, process and documentation can go a long way.
- Offsite backups – Do you do offsite backups? Do you do them to some stable media that is on a disconnected network or a system that doesn’t have the same exact access policies? Yes someone creative can still wreak havoc, we’ll never stop every bad disaster – our goal is to prevent most and make it really hard. Someone can easily break into just about any house out there – we do locks and alarms to help deter.
- Read the stuff above – All those tips help.
- Get audits and health checks – This isn’t a shameless plug for our SQL Server Health check – there are a myriad of companies that give great health checks. But get someone outside to look in. It’s not that you don’t trust your team, it’s that you want to verify, you want to get an outside eye. The cost of this is even less than the insurance of offsite backups and DR plans I ranted about above. Get that outside review.
So – it’s good they were transparent. It’s good they are trying. It’s sad it came to this. It could have been prevented most likely. These customers could have had their own backups someplace else – and I hope many did, but I bet many don’t. I hate learning opportunities like this, but it’s what it is. Use that feeling to take some action.
