Hey DBAs: Be Prepared

I’m a big fan of bringing ideas and concept in from other industries to the world of technologists, particularly those of us in professions operating in and around data. I’ve written posts before about lessons from aviation, a string of failures at the Secret Service, some corollaries to farming or our gardens at home, some experiences from my time working in the Fire/EMS service, and even envious of chefs with their recipes and all. The other night at about 11:00 PM, I was at our local fire station helping put our trucks back “in service” after we were just at a structure fire in town for about 6 hours. That got me thinking of some pictures I took some time last year after a fire, for a blog post I had in mind then – also while putting our trucks back in service after we were at a different fire. So anyway…

Enough preamble. I’m going to steal a few thoughts about preparation from the Fire Service. The lesson is the same really, so I’ll share the lesson here so you can leave and begin working on it if you want the TL;DR version – Be Prepared. Always. Be. Prepared…   

That’s really what this whole post is going to be about – just a reminder to make your life easier. Then we’ll beat the dead horse of that reminder. We’ll try and draw some parallels and give some ideas. I’ll start with a concept and a quote. The Concept – I don’t love the word, “luck,” but I think what we think of as luck is a concept that you can help “manufacture” for yourself.  After doing DBA work of some sort for 20+ years now, I’ve lost count of the number of times that I’ve seen folks prepared or primed for “luck” to hit them. Either folks who calmly prepared and plotted and were ready for an issue (or prevented one!) or folks who sort of seemed like Murphy’s Law was written about them. For sure the best-prepared people can have a bad day, and the least-prepared people can have a good day. I like a quote from football coach Urben Meyer about this:

 I have yet to be in a game where luck was involved. Well-prepared players make plays. I have yet to be in a game where the most prepared team didn’t win.

So. That’s the name of the game here. Be prepared. Be ready.

Learn, Practice and Perfect Good Habits and Form.

That’s my locker at the firehouse (on the left. black helmet. I’m just a firefighter, no officer duties for me!). I can tell you a few things about that locker, what is where, and what state it is in. I’ll do that.

• That black bag hanging there is a mask that fits me and works with our air packs.
• In my inside left turnout coat pocket is a hood (think ski mask, keeps my face a bit cleaner from the crap people fill their houses with, and keeps my face, neck, and ears protected in several hundred-degree building fires. (plus it keeps you warm on an outside job in New England 😉 )
• My right lower coat pocket has my right vehicle extrication glove. My left lower pocket has the left. My right pocket also has a multi-tool typically. (I’m in bad form right now, I usually used to keep a length of rope, some cutters in the pockets and a harness around the bunker pants – I should get that going again – practice what you preach…
• In my right pant leg pocket is my right structural fire glove. Left in the left.
• Police officers put their pants on one leg at a time along with everyone else. Us firefighters put them on two legs at a time.  You go into any firehouse in America, and you’ll see all the pants just like mine are. We take them off and lay the pants around the boots – the boots in the bunker pants. When I take them off after a call, I’m careful to keep it all tight together, keep the boots in the pants and keep the suspenders right where I expect them.
• My helmet is up top (I’ve only forgotten it once.. long story…) and the ratchet strap is a little looser than I like for most calls – but it’s perfect for when I need to wear a mask and a hood. I’ll try and remember to loosen the ratchet strap a few clicks after each call.

You see. We put everything in its place. In the culinary arts, the chefs call this “Mise en Place” – it’s when they chop their ingredients up ahead of time and get it ready to go all where it should be – everything in its place. Why? I don’t want to forget one thing. I don’t want to leave without my gloves. Or without my hood. I put things back the same way and place each time because minutes count in our line of work, and sometimes seconds can also.

It’s more than just things in their places, though. We practice. We drill. We do repetitive things. Early in the fire service, in Firefighter I (and again in later classes often) we do “don and doff” drills – donning (putting on) and doffing (taking off) your gear. The goal is to go from street clothes and shoes into that full turnout gear as fast as we can. And to not just be in it – but to be in it properly and safely. So we spend a lot of time practicing the mundane things. To the point where this becomes second nature.

It’s not just putting our stuff away right, either…  After every call, we put things back where they belong for the next call. That may be something for us, or it may be something for someone else. We take care to do the little things.  In this picture is an air pack. Even in our vehicles, the seats are designed for the ability to put your pack on while sitting in the seat quickly. They are designed to get the straps easily on, have the release cord easily accessible and be able to get to the address, get out and be good to go. If you put your gear away right and know what you are doing – you can jump into the engine or truck with your equipment good to go, your hood on, your boots accessible, your helmet in hand, strap on the pack, even snap out and turn your pack on if it’s a real call as you get closer – and as soon as the engine your on pulls up – you can be to work. If you are the first arriving engine the driver who is well-versed and practiced in pumping the truck can have the truck switching into pump mode while those of you in the back are stretching the line(s) from the sides of the engine where we carefully layed the hose out in a pattern designed to get it out quickly and stretched into the fire. By the time the already readied firefighters stretch the line, the driver/engineer is typically ready and with a quick signal from the team on the hose line – the pump will start flowing water to the line, and we can be on our way pushing into the fire. If we practice and we lay the lines out right and the truck was put away right, and the same way it always is and our gear is operational, and care was taken a day or a week or even three months ago – it’s a beautiful thing. If someone jumps the gun or something wasn’t put away right, or the irons weren’t where they should have been or the hose lay wasn’t done well, or the driver/operator shouldn’t have been doing the pumping – it’s a scary thing. This isn’t luck! This is preparation.

I firmly believe that if you prepare ahead of time, you manufacture what many see as luck. I already said that. I’m reiterating it! When you start having one small thing after another go wrong, you begin to look unlucky. You aren’t unlucky. You are ill-prepared. There is a huge difference! You could say that you manufactured bad luck by not being read. I’ve noticed over these 20 years of DBA life. These 20 years of watching myself do something I’m frustrated with and these 15 years of Fire/EMS – when people aren’t ready, they psyche themselves out. They don’t have the comfort of a routine. They lack the confidence in having it all laid out as best as the circumstances allow (I still always laugh inside at the 6-minute prep a nurse in the hospital takes getting their little IV table all tidy and ready… We rarely were afforded that luxury on the critical calls in the ambulance to get each small piece of tape torn off just right on the edge of the table, with the Tegaderm good to go and all our tubes lined up nicely in a row, with a pleasant and clean non-moving environment… I digress..) And once you start the panic? You just became a manufacturer of bad luck.

It’s not just putting the packs away right, either…  When I’m putting an airpack back in service after a fire call, I have a few goals. 1.) Clean it (cancer is a real danger with the crap that burns in a house fire) thoroughly. 2.) Make sure it is easy and quick to use for the next person (I don’t want to manufacture bad luck for the next firefighter by tangling them up getting into their pack or taking the pack out of the holder… Remember… Once you make things go awry, you risk manufacturing “bad luck”).  And 3.) – that it is good to use. We have many things that run on batteries. Many wires and tubes and connectors are coming out of this stuff. The other night after our fire call, I was putting some packs back in service and do my mini-inspection on each one. I ask myself “If one of my brothers or sisters took this pack or I took it – did I just manufacture bad luck?”. Sure enough, one of the packs had a wire connector snapped off during the frigid night and the icy conditions. Maybe they fell, who knows, I was out at water supply. The pack was broken, though. I believe it was the wire the activates the PASS alarm – this is that alarm that starts going off once a pack has been turned on with air and the pack stays still for more than a few seconds. Once you stay still for a bit, it starts beeping. If you don’t shake your body to show it you are moving and alive, it gets progressively louder until the alarm is full bore tripped. That is a key tool to shout out “FIREFIGHTER DOWN!” if you are unable to do it for yourself. If you watched 9/11 coverage in the immediate aftermath, sadly, you know the sound an activated PASS alarm makes.

Sum it Up / Bring it Home

We say “we have a fire to put out” a lot in our line of work in the DBA world. We usually aren’t but let’s bring it back there anyway. There are a few big takeaways from this view into preparation:

1. Practice. Practice. Practice – We practice putting on and taking off our gear. We practice losing visibility. We practice getting stuck. We practice ice water rescues – in frozen lakes that we cut holes into. PRACTICE.
2. Know your tools – We know where our stuff is. We’re used to our tools. We put our irons back in the truck in the same spot each time.
3. Clean up after yourself – We don’t know when the next call is. We always prepare like it’s going to come just as soon as we put everything away.
4. Be proactive in maintenance and review – We catch things before they go bad and take them out of service.
5. Mike says you manufacture your own “luck” – And that can be good or bad luck. And luck is the wrong word. But. BE.PREPARED.
6. Cops put their pants on one leg at a time. Firefighters are better. – There’s no lesson there. It’s just a chance to tease our friends on team blue.

As DBAs and database people – we can pull so.many lessons from these points. These are just a few. What else am I missing?

• Practice restores – seriously – after 20 years of saying that, I thought I could stop. I can’t. Test your restores.
• Test failovers – do you have an HA/DR environment? Have you ever failed over after you went live? Did it work? Why haven’t you?
• Create checklists (Older blog post – but it still applies) create lists for what you do. Even the mundane things. ESPECIALLY the ordinary stuff. It’s the mundane things you forget and assume around.
• Take lots of saves. Don’t leave those query windows open on SSMS in an RDP window assuming they’ll be there tomorrow, darn it! How many times do you have to be burned before you stop doing that?!?!?!
• Create alerts – let your SQL Server tell you when it is unhappy.
• Go to learning events and absorb as much knowledge as you can – seek to improve.
• When you troubleshoot – be efficient and fast – but don’t be haphazard and crazy fast. Remember – you choose what kind of luck you make. Once you lose control of an ordered plan of attack and troubleshooting steps – you’ve started making bad luck. Watch out. You’ll pay for it if it continues.
• Check the basic stuff! At Straight Path one of the things our Remote Managed DBA clients love the most isn’t the experts or all of our advice and tuning – it’s the daily status check in that lets them know what is good or bad. Don’t assume. Check. Verify.
• I’m serious, and I’m a hypocrite – but when I have a clean workstation when I have a well-organized desk and computer desktop and things where I expect them? I’m more efficient. My brain has less work to do figuring things out. If we swapped our lockers at the firehouse each day? I’d be lost before I even get into those pants (… with both legs. at the same time…)

Be prepared. Be ready. Assume things can go sideways. Do all you can to minimize the distraction and risk!