Monday morning: you’re back at work, but your SQL Server estate had a weekend from hell. Some companies find that out the hard way—every single Monday.
It’s almost like SQL Server is taunting you, “How does it feel to treat me like you do?”
I See a Ship in the Harbor (Do you watch for SQL Server problems?)
“It’s worked like this fine for seven years; how could there be an issue?!” – I hear that way too often.
SQL Server problems start slowly and build up over time. The ship is in the harbor—there’s a fire on it—you just don’t see coming for you yet.
Without even logging into our ticket system and reading the notes from the most recent sales calls and new client onboardings, I’ll paint some familiar scenarios.
The main takeaway from each of these is that these problems were already there; they just hadn’t presented themselves yet.
- SQL Server Corruption – It was there. We still don’t know how long it was there for two recent clients. All we knew is the clients each found out when the users got an error.
- The backups weren’t what they seemed – This is a really bad one to mix with the above one. Sadly, they often mix.
- They were fine – a little slow – but fine – it just started getting a LITTLE BIT slower…. Until – BOOM! Clients now voting with their pocketbooks or with reviews on the app stores.
- SQL Injection vulnerabilities combined with wide open permissions and worst practices make it a matter of time for some clients to have a blue Monday in the security realm.
- The sysadmins just wanted to install an AV tool – no need for the DBA team to be involved, right? right?
At the end of the day. The ship is there in the harbor. I see it. You’ll eventually see it. We know it’s there. It’s smoldering. It’s coming for you some Monday…
Get paranoid, worry, run health checks, learn about your SQL Server, and ask it how it feels today. I’m posting this on a Thursday, so you can not have a Blue Monday next week. Read on for some tools to consider and approaches to take.
Thought I Was Mistaken
“A different team handles the backups, but they are probably fine…”
Add to the unseen unknowns lingering and waiting above; assumptions are out to get you—yours, your teams, all of the assumptions. So often, I’ve had to break the “you have data loss” news to a client only to follow it up with conversations with their teams fighting for who owned what part of the backup process…
I cannot tell you how many conversations with infrastructure teams I’ve had where they’ve said “oh no the business knows that they can lose up to a day of data,” only to watch the business leaders squirm when they realize more fully what the IT team said.
There is only one safe assumption for a DBA: Murphy’s Law is always correct.
If something can go wrong, it will—and usually in a completely new and excitingly awful way.
Your job? Fight that entropy.
Adopt these helpful “paranoid” attitudes as a SQL Server DBA (whether you are an accidental SQL Server DBA or an on-purpose one):
- Trust but verify – everything.
- Test!
- Be the data advocate – If someone touches a SQL Server for an update – make them explain why and show how it was tested. (Yes, even the CISO or CIO)
- Figure out who owns the backups – before you find out they don’t exist in a usable way.
- Never assume – see it yourself, understand it, then know.
There are folks out there all over the place who can tell you how it feels when their heart grows cold after they they finally realized that the most important resources in the DBA world aren’t actually assumption, presumption, and hope…
How Should I Feel?
Imagine waking up on Monday knowing your SQL Server environment is good, knowing what worked and what didn’t. Having a prioritized plan on what to focus on. Knowing instead of hoping..
It’s very much possible. We get a lot of “red x’s” on our daily health checks every Monday morning with clients. Stuff goes bump over the weekend. But it doesn’t make for a blue Monday—because we review our clients’ SQL Servers’ health checks, we see the issues on our proactive dashboard for each client. We are aware, and we are set up to be ahead of the game.
Mondays in IT will always be “interesting” days – the weekend is when a lot of scheduled and manual maintenace happens. Maybe large data loads, the weekly full backup (if you do a weekly full and daily diff as we recommend) runs. So Mondays can be busy. They can be interesting. They should not have to involve panic and running around trying to bring an environment back from a mistake you later will learn was preventable with just a little review, a little less assuming, and a little more awareness.
Do you want that kind of a Monday?
I Thought I Heard Your Words…
Your SQL Servers don’t have to feel blue every Monday. You don’t have to either. The tools, the checks, the training, and the resources exist. The signs are there. You need to listen and act.
Help Yourself Right Now
I’ll be sharing some of these resources throughout the series but they are worth calling out here as well.
- Run SP_CheckBackup – this is a free tool Jeff Iannucci and team here at Straight Path built to help you run a SQL Server backup check.
- Look for the errors, read the links the findings highlight.
- Find the other tools backing up your databases – and figure out why – make a clear owner.
- Find and fix the DBs with no backups at all.
- Find and fix the DBs in Full Recovery model with no log backups.
- This tool doesn’t phone home, it doesn’t cost anything. Use it. Fix your environment.
- Run SP_CheckSecurity – another free tool we released to help you run your own SQL Server Security check. Review and fix the items. Each finding has a post on our site with more of the how and why. Learn about security as you go. Again – this doesn’t phone home or cost anything. Just use it and get better.
- Run through our SQL Server Health Checklist – Pay close attention to the parts about maintenance – if you are not doing checkdbs – fix them second only after backups – if backups are good – GET CHECKDBS running now at a schedule that makes sense for your usage (best run at a time of lower usage). Learn about and use the Ola Hallengren maintenance scripts.
- Review the health of your SQL Servers.
- Use Brent’s First Responder toolkit.
- Or use Glenn Berry’s excellent SQL Server diagnostic queries.
- Run through the scripts after fixing the above items – and make sure things are good.
If you just do those things – you are ahead of the game. You are already off to not having a Blue Monday because of SQL server surprises. Sure, stuff can and will happen. But imagine knowing you are solid on the important things and can get the environment back if nothing else.
Your SQL Servers Don’t Have to Hate Mondays
The tools exist. The checks exist. The solutions exist. I just got you started.
You just have to use start.
But if you’d rather skip the stress and let our team handle the fire drills before they start, let’s talk.
- Need a 20-hour health check, teach, and fix engagement? We’ve got you.
- Need a 4-hour rapid triage? We’re here.
- Want 16 SQL Server geeks as your Remote DBA Team full-time alongside your team? It’s who we are.
Call us: 888.SQL.DATA or Contact us today.
(And if you fixed something after reading this—brag in the comments!)
