DBA Tunnel Vision

One of the things hammered into you in Firefighting training, Law Enforcement training, defensive gun use training and many other forms of training is “Do not develop tunnel vision.” There is a reason they emphasize this concept so much, too. Lot of folks get killed each year by tunnel vision.

Do you think this phenomenon doesn’t apply to us in the DBA field? You’re wrong.I speak from mistakes I’ve made because of tunnel vision. I speak from helping mop up after tunnel vision made mistakes…

Tunnel Vision – Problem Defined

This image shows what happens when our senses hone in on a big, obvious problem. Different people respond differently to stress. But there are some biological constants. You’ve probably heard of “fight or flight” – the body is built to want to survive. It does all sorts of neat things to help us survive.  One of those things, though, is a response to stressful situations.

You’ve probably experienced it before – your heart rate goes up, you start sweating to keep your body cooled down as it heats up to prepare to deal with the situation. We also lose some peripheral vision – there is a problem – it is right there and it is scaring you and your body wants you to know it, respond to it and, well, just make it go away.  Now because I told you what this post is about and because you can probably already see the second image, you understand that there is something bad blocked out in the periphery here. I admit it, you caught me…

You see here in the center of the image  is an obvious threat.  When run through various simulators when presented with this picture in a series of pictures, a lot of folks receiving this type of training rightly point out a scary looking dude from the 80s (or something) holding a gun to a woman’s head. On the “good-bad” scale, that’s bad. And you know the whole job is about protecting and serving presumably – so that is something that should cause hackle to rise and a heart rate to increase. And the threat here in the center of the really cheesy tunnel vision effect graphic is real. And it requires a solution.

Tunnel Vision – To a DBA

There are exceptions, a friend of mine was at his office one day for one such exception, but in general we aren’t going to find the situation represented in the image above in our day jobs. We thankfully have relatively safe careers and we don’t take guns out of bad guys hands. Well sometimes we revoke developer access to environments – sort of the same thing. Every good DBA knows a developer who has made something “bad” on the good-bad scale… Enough picking on developers.. But seriously we don’t end up in horrible situations like the one pictured. But we can have terrible, no good, very bad days at work. And some of the one liners that I’ve witnessed tunnel vision in myself as a DBA or consultant or in colleagues or clients are:

• “Corruption! There is Corruption!”
• “This is the CIO.. I am standing right behind you. Please restore to a point in time NOW!”
• “The most important application of the most important company is down at the most important time!!!”
• “This performance is totally unacceptable, and our black Friday sales start in 15 minutes!!!”
• Fill in the blank here.. You get it.

The Anecdote To Tunnel Vision

I sort of gave this away in my post about life as an EMT preparing me for life as a DBA – especially about “being the duck” –  it’s simply really. It’s simple and, yet if it were easy we wouldn’t be stuck in the trap of tunnel vision so easily.  We have to force calm on ourselves. We have to force methodology, careful reflection and full analysis of all details. We’re fighting our biology, but it must be done. It’s easy to see a big problem, know that 80% of the time when the big problem comes the solution is complicated, takes forever and is difficult and start down the path of the “hard solution”. Well there’s a saying in the Fire Service “try before you pry” – it’s simple, but it’s effective. Don’t break out the big tools at a car accident to try and start cutting a car apart to get someone out – try and open the door first… Don’t take out the Paul Randal corruption hack attach and hex edit toolkit right away for corruption – Paul never takes that out first – he takes out his “Restore backup” toolkit first…

So the image here is what you expected it to be. But the point of the training is simple – train yourself to see everything. Be observant – be hyper observant and hyper aware. Make yourself scan the full situation and environment and take in all you can, process it all and pretty soon you’ll realize that the biggest threat to you isn’t the man preoccupied keeping a panic stricken, screaming and flailing hostage – it’s the man hiding around the corner with his gun aimed right at your head.

So it is with us DBAs – there’s no gun pointing at our head. But when we get stuck in the weeds and we forget to fly the plane while we focus on the narrowly focused on area – we miss the big picture and we make mistakes. Sometimes costly ones.

Break The Chain of DBA Tunnel Vision

The anecdote is simple here, too. Simple but we have to be deliberate and intentional about it. We have to practice at it and work hard at it and do things to make sure we are sticking with it. And we have to do it all the time – because if our normal day to day looks one way, and in the emergencies we expect to act differently – we won’t.. We act like our habits tell us to act. See how this almost got me hit by a train on Pinal’s blog…
Some Tips on breaking this cycle:

• Be Deliberate and Intentional – I get after my kids for not looking when they cross the road. Even the 9 and 13 year old ones. Even on our very unbusy road. I want them in the habit to look both ways. And I don’t want them to just feign obedience and turn their heads. I want them to pause… Look.. Pause absorb.. Pause look the other way.. pause..absorb.. And think about their decision to cross.  Let’s start doing that with everything we touch.. Sound out a checklist in your head. Read the dialog – even the one you’ve seen 476 times before.Take in information, slow down, and act. I have seen more emergencies blown by quick actions – with all the best of intentions – that have killed a restore process, killed a critical production fix, etc..

  When I am done shooting at the range  I make sure my firearm is clear of ammunition 2 times – every time. And I don’t do it as a habit without thinking. I intentionally say to myself “Look, Feel is it empty?” and I look (in a safe direction), sweep my pinky into the open chamber.. Then I look away for a couple seconds thinking about the task at hand and nothing else, and do it again a second time. Why? Because I want to slow down, pause and be observant, I do it again with those same firearms before I put them in the safe later that same day when back home…

  If you do nothing else? Slow down and read, listen, look, absorb, pause, look again and then act.. 

• Help Others Be Deliberate and Intentional – In EMS training when it comes to applying a defibrillator you’ll often hear the mantra “I’m clear, you’re clear, we’re all clear” repeated over and over. It’s excellent. It’s horrible when it is just quickly belted out like a rehearsed thing you are supposed to say. But when someone pauses and says “I’m Clear” – they look at themselves and where they are and if there is any direct or indirect contact. You’re clear – now everyone checks and the one pressing the button checks everyone else. Then one final “We’re all clear!” This one is the warning – the “here comes some joules!” message. And it is a final reminder to “Check yourself!” – I still say “Hands and Feet!” whenever I shut the door in my truck – even if it’s my wife in the passenger side (habit) – folks should know the door may be shutting. Folks should know what their job is in that instance. But I’m helping them remember their duty and warn them.. When I’m having someone else do a restore and I can’t watch them but I have to give instructions – if I want them to choose “WITH NORECOVERY” I tell them a second time – please note that I said NO recovery” – I called it out separately. Not to be a patronizing, patriarchal jerk – but I know it’s important to do that and I know folks skim and sometimes read what they want to or their brain told them they read.We can’t be responsible for everyone’s actions, but we can do much to make sure folks sweat the small stuff and give friendly reminders…

• Checklists  and Documentation – I know we all groan about documentation – well most of us do.  I blogged about each (Checklists, Recipes and Algorithms; Documentation – How I learned to stop hating it and embrace it) – they have tremendous value for us for this very purpose. There is a reason pilots use checklists and auditory validation with each other that the checklist is being run through properly. There is a reason NASA Mission Control relies on them. If you do something enough that it becomes second nature and you don’t need the documentation – you should consider using a checklist anyway… The point here is – never be fully comfortable that you “have it under control” – never just assume that everything will go exactly as planned. I’ve caused enough extra work for myself at 2AM doing something I have done hundreds of time in SQL Server simply because I was tired, lazy and not following a checklist.  If you have a hard time forcing yourself to be deliberate and intentional – this is a great way to build the habit.

• Advice – I can’t tell you how many problems I solved on my own just by striking up a conversation with someone else..  Ever have a conversation like this, “Hey Bob, help me out here.. The readerform is stuck on 11 and it is only supposed to go to 10, but someone was in the lab yesterday and they glued the… AHH! THAT’S IT!! THANKS BOB! I FIGURED IT OUT!” ?? I have. What happened is – by talking the problem out, you looked at more information, you slowed down and you analyzed the problem with a different part of your brain… And if that didn’t work? Maybe Bob could have the answer.. Ask for help.

• Look, Listen, Feel while SLOWING DOWN –  I know this falls under be intentional. But seriously – slow down a minute. If we remember one of the things I talked about in the ambulance post – it’s that the emergency isn’t yours we’ll be better off. If we go as fast as we can go, stop trying to rush a solution and realize that people will be beating down your fabric lined cube wall whether you solve it in 5 minutes, 15 minutes or 1 hour.. In other words.. it will never be fast enough for your users and customers – so rushing to save a few minutes and risking making a mistake that makes the problem worse? Not worthWhen I was in the ambulance, I’d ask the person driving to turn the lights and sirens off on the way to the hospital unless it was one of the dozen times we really needed every single second..  Statistics show that you get to the hospital in most areas not a whole lot faster with lights and sirens.. And some people do really really dumb things (those same people could read this article come to think of it.. nice.. full circle!) when they see lights and hear sirens.. Really dumb. So I figured I’d rather get there 2.5 minutes later but more guaranteed to not be in my patients lap, be on a backboard myself or have a sharps contained dumped on my head.. Some of the drivers didn’t like that – they were younger at heart than I was perhaps…Point here is – slow down! Get away from your problem even. Stop and breathe. Don’t get caught in panic. Realize it will be fixed when it gets fixed and apply a good methodology and take your time.