Last week, I ripped a story from the headlines to draw lessons for Technologists from – it was about the Secret Service lapses that allowed a fence jumper to get into the threshold of the White House residence. That was bad. This week, though, we’ve learned of some additions to the story that makes it worse. You can read about those here or here.
These developments lead to a new set of lessons. Two parts this time.. Today’s post for DBAs with three tips to learn about Database Administration from the lapses. Another post coming out on Thursday for management.
Take Warnings Seriously
At the White House – The fence jumping suspect wasn’t unknown to the Secret Service.. In July he was pulled over. The police found a sawed off shotgun in his possession (a crime) and found a map of Washington DC with a circle drawn around the white house and a line to it… Secret Service were alerted. In August he was stopped and questioned while walking outside the White House grounds with a hatchet in his rear waist band…
At desks that looks like yours – A DBA has deemed certain error messages “acceptable”… Someone else has said “oh that’s just a warning message…” Others are saying “well, we can keep throwing hardware at the problem”.. Some developers are saying “works fine in dev!”
So… It’s the same thing.. I’m not saying we go 1984 and arrest and keep people who like hatchets or have touristy maps of the White House near the White House.. But there were warning signs. I’m not saying that you are evil if you ignore warning messages or are in the habit of saying that “these alerts are Safe To Ignore” – but I’ll tell you – When you let warnings go by unchecked – you’re going to find yourself missing a big failure when it was something that was stoppable.
Set Up (And Listen To) Alerts
At the White House – The White House ushers apparently didn’t like false alarms on perimeter door alarms. So the door the suspect came in through allegedly had its alarm deactivated.. I get it – I’ve been at airports. Those stupid gate
door open without code – or open for too long – alarms are annoying and distracting.. But.. Well the Ushers office shouldn’t dictate what the elite security team does with physical security measures.
At desks that look like yours – There are DBAs who have no clue that they are even getting the warnings I just talked about in the section above. They have no clue their systems are running into trouble. They don’t know a log file just filled up. They don’t know about alerts, or proactive tools or steps they can take. They’re just going about today’s work waiting for the affected users to start e-mailing about the problems that were smoldering.
So… There are great monitoring tools out there for SQL Server. There are some great free things you can do in less time than it takes you to read a blog post.. If you don’t have SQL Server alerts configured? Stop reading this post. Instead read this post and look at my 3 quick videos talking about how to set up SQL Server Alerts.. Take the 8 minutes you need and set up SQL Server alerts… And then listen to them and react appropriately to them. The weekend you save may be your own.
Don’t Give Up Too Easily
At the White House – A while back someone shot a gun at the White House, 8 shots hit the white house in fact. No one was hurt, no bullets entered the White House. Agents outside the White House were sure they were being shot at. They all radioed their concerns.. Eventually they were told by their command, “no silly.. it wasn’t gun shots, probably just construction… stand down and ignoreit…” They eventually found the 8 places the White House was shot and the suspect.. But it was days later…
At desks that look like yours – Someone has complained about a problem. A Jr. DBA thought they noticed something off.. Someone just read a blog post saying “You should setup SQL Server Alerts”.. But the DBA team is busy. Justifiably busy with too many tickets and requests and too small of a staff.. Or the Sr. DBA doesn’t think Jr. DBAs should be heard… Or the project manager says, “Damn the problems you just explained! Full speed ahead!!” Those agents? They followed protocol and stood down..
So… Will you stand down, too? If there is a problem in your environment, if something isn’t right? Fix it.. Explain why it’s an issue and see it to conclusion. You can be a little paranoid and be part of the chain of a successful dodge of disaster. You can be a little paranoid and have done a little extra and a day wasn’t really saved – it was sort of “wasted” (though I talk about that gamble here and why you should always take the bet).. Or you can stand down and be part of a really embarrassing miss.
In summary.. I love the Secret Service. I love the job they do – not just protecting national leaders but with computer crime and Treasury matters.. They are professionals who by and large do a great job. But there were some serious lapses here. Indicative of some system issues. I hope Congress gives them the time and space (with oversight) to work the problems out instead of do a bunch of grandstanding sound bite searches… In the meantime? Look at those failures, look at the parallels and do something about them in your environment. If you have already? Then this post isn’t about you.
Photo Credit – The Wisconsin National Guard
I recently passed through a similar experience – the only difference was it was in a virtual development environment that we have very limited control over. The drives on one of the storage arrays failed and it was only days later, when the database started throwing OS 32 CRC errors that we realized something was amiss and alerted the network guys. It was a development environment, so we were able to recover quite quickly from a backup, but that incident was a wake-up call to keep eyes open and keep monitoring.