I live in New England. I expect snow in the winter. This winter, though, has been a bit of an anomaly from recent winters for us. We started out with very little snow, abnormally little snow. But then it came (and it is still coming) all at once. Over about 4ish weeks we had a good 7-8 feet of snow in a handful of good sized storms. Lot of blowing and drifting snow. Roofs have been collapsing here and there, roads in Boston have had to change to one way streets because they are too narrow now with the snow banks to allow two-way traffic. The other day while shoveling off a roof for the first time in my life, I thought of the lessons this winter has in store for Database Administrators.
Sometimes it is easier (and safer) to just do it, then debate it.
That picture is the view of my roof after one round of snow shoveling. Looks like there isn’t much up there, but that slope going to the ground is a lot of snow (in fact it is our new sledding hill, the kids love hiking up the roof with dad and sledding down it).
I wasn’t sure if I had to shovel it or not. There were some spots where drifts had made the snow pack almost 3 feet, and mostly everywhere there was at least 1.5 feet of snow on the roof. Some roofs nearby have collapsed, and the slope on our roof and way the wind blows in the neighborhood meant it wasn’t going anywhere anytime soon and we had a new storm on the way a few days later.
I had three choices here:
– Measure the snow precisely, talk to building inspectors, review the roof lines and figure out if I need to shovel it or not.
– Do nothing and hope it was fine.
– Do something even though it may be fine and it may be a waste – or I may be saving the day for my garage.
I opted for choice 3. I wrote a post about why I like to gamble in production environments. This is that gambling I am getting at. In my mind, the only right options are the first and last. The middle option is too great a risk, because there was a greater than 0 chance of that snow load causing an issue, and in the time I could have done that analysis, gotten the expert opinion and taken the proper action, I could have shoveled it off 2-3 times. Plus.. I got a cool sledding hill out of it.
My garage is still standing today. Was it because I shoveled it off? I don’t know. But I know have one less worry in my head.
Hey DBAs – You probably thought of similar situations you face in your day jobs while reading this. Could be that question of “do I take the downtime for the reboot while that strange memory dump keeps appearing?” The DBs aren’t complaining, Microsoft says a reboot will stop it, but you don’t want to take the planned downtime, but there is a greater than 0 risk of an unplanned downtime.. Do you get a little paranoid and build an “extra” redundancy into your cluster build out to prevent a failover? Even sometimes with resources – do you spend the money on hiring a consultant to try and tell you if you should with 128 or 256GB of RAM – when you can buy the RAM for about the same price as the consultant giving you an opinion?
You get the point – sometimes being a little liberal on the side of caution and saving time later goes a really long way. And sometimes you can’t prove it did, but in a lot of cases? That’s okay.
Work Hard Now
This picture of my deck and one set of stairs off the deck explains what I mean pretty well. When it comes to shoveling there are two approaches one can take.. The thorough approach – shovel everything, clear proper paths, go the extra mile and be sure you cleared everything. The other approach is the “just 
enough” approach. Make a spot for the cats and kids to get out but then let them climb the rest of the way themselves.
I elected for the latter approach around storm number 2 for the back deck (but the former for all storms still). I was in a rush and feeling a bit lazy. Well – what that has done is allowed the deck to get narrower and narrower because I didn’t take enough snow out initially. My job is harder now, those snow banks on the side where that snow shovel is are getting harder and harder as they freeze. Basically, they won’t be moving now. It’s not the end of the world there. If I took that approach on the driveway or front walkways, I’d have a real problem on my hands today though.
Hey DBAs – You have a choice with just about everything you touch. When it comes to installing SQL Server, when it comes to reviewing code, when it comes to implementing a proper monitoring or backup/recovery solution… You can race through it today and just get it done… Or you can spend a little extra effort today and save yourself real time in the future. If I cleared the deck off properly after each storm, my shoveling would have taken me maybe 15 minutes longer.. If I were to try and clear that off now? I would be looking at wasting an hour or more of effort to hack through the ice and move the snow. Please, bite off a little more than you think you can chew with that project you are working on today, and your future self will thank you.
Be Prepared
We knew winter was coming. It happens every year. In previous years, I’ve let the first snow surprise me and have had to do a race to finish putting the wood under the shed roof, finish helping the kids pick up their summer toys, run around making sure the decks and walks are ready for shoveling or snowblowing. It’s been an annual tradition to be surprised by the first snow. But that ends up with me being sloppy, quick and missing something ultimately. Something that gets eaten by the snowblower, that gets broken while shoveling the deck, that gets ruined by being left out in the snow, etc. This winter was better and a bit less stressful when the snow came, because I was ready for that first November snow. That was a really good feeling to know it was all done and waiting.
Hey DBAs – You know the lesson I’m getting at here already! Don’t just spend a little time with your tasks today. Spend more time preparing for them. Do proactive things. Setup alerts – I promise you it will take you 15 minutes to set up SQL Server alerts – I have videos that show you how.. Create recovery checklists, create migration checklists before a migration. Plan, rehearse, think of things that can go wrong and get yourself ready for the situations that you will someday face. Don’t let your first restore be while the CIO is darkening your cube wall asking “Is it done yet? Is it done yet?!” like a kid on a long car ride. Get proactive, your stress levels will go down every day, and when the disaster strikes, you’ll be that much more ready.
