Checklists, Recipes and Algorithms

What do checklists, recipes or algorithms have to do with Database Administration? A lot… If we study a few career paths we can see how a checklist or process flow  makes their lives easier. I love stealing ideas from everyday life, working them into something useful for my DBA career and I think we have an opportunity to do that here.

Let’s take a brief look at three careers and the organizational tool they use and see what we can steal from them.

Pilots – Checklists

The next time you board a flight, take a peek into the cockpit. You may notice checklists lying around or maybe even being gone through. Before, during and after most flights the air crew are going through a checklist. They use these for routine purposes (Pre-Flight walk through, preparation for takeoff, preparation for landing, takeoff itself, landing itself) and they use them for emergency procedures (Bird Strike, Engine Fire, Wing Stall, etc).

Look at preparing for takeoff. This is something they have done thousands of times. They can probably do it in their sleep yet they still use a checklist. They know that if something is missed, it could be catastrophe so they use their checklist. The aspects of their job they might see as mundane still get a checklist.  They question each other and make sure they follow the process the same way each time.

In an Emergency, an action only performed in training will definitely be aided by a checklist. It has been amazing to me that they use them in serious situations. My first thought was always, “How could they possibly have enough time to read one?!” After thinking about it, I now wonder how they could possibly do it without one. Some pilots fly different planes and there are different procedures on each, flight crews may have never worked together before. The checklist ensures that the right steps are done in the right order. Look at the history of airline disasters and you will probably see the genesis of some of these checklists.

What about us?

Hopefully most of us are already using checklists of some sort. If not here are some thoughts around them:

• Are you an overwhelmed DBA trying to get more staff to help? You get that staff and have no checklists? Now you have to work even more to ensure consistency in processes.
• Uh-Oh, the dreaded Restore request just came in and it’s an emergency. The CIO is in your office, upper management is drilling you with questions while you are trying to do it.  Sure wish you had your restore process documented and in checklist format, right?
• Ahhhh Clusters. They are so fun to setup and easy too, right? If you have ever set one up you remember the “oopses” you had the first time, I hope. You had to go back and create the groups because you aren’t domain administrator, you had to go find the e-mail with the link to the info about setting up DTC at the last minute, etc. What if you created a checklist and documentation as you did the first one, specific to your environment and then you followed that for future installations?

Doctors – Algorithms

If your loved one shows up to the hospital with one of several serious conditions, the medical staff my reach for an algorithm for their symptoms. In my spare time I belong to a volunteer fire department and have had various levels of EMT training and certification where I have seen flow-chart style algorithms used frequently.

One of the specialized certifications that comes to mind is one used in various cardiac situations from the AHA. This is a system used by Doctors, Nurses, Paramedics and other health professionals to deal with extremely urgent, life threatening situations. Various algorithms were created to deal with various patient situations. They ask questions, remind you to perform a certain course of treatment and then respond to changing patient responses.

The next time you are in a hospital, look for the “crash cart”, it is highly likely that you will see a flip book of flow charts attached to or on top of the cart. Many hospitals train with these flip books as part of their response. Can they work without them? Of course they can but having them readily available has proven to improve results. It isn’t weakness on their part but an understanding that in times of stress things can be overlooked. In fact many “code blue” response teams in a hospital setting will designate someone with the role of note taking/algorithm presenting. They will call out the sequences of steps and confirm actions and reactions.

Above is one example of such a flow chart used for a patient who is not breathing without a pulse – Cardiac Arrest. It boils down all of the training they have received, goes through the questions and actions that have been hammered into them for training and guides them through the proper actions for their patient.

What about us?

If a medical professional, who has likely received more training than you or I in our field, has found that an algorithm helps troubleshooting a person why can’t we see the need in troubleshooting a less complex database?

They can help us with:

• Troubleshooting – This seems to be my raison d’etre lately with posts about troubleshooting methodologies or not trusting every solution a quick web search shows you. Even still, imagine how laying out an algorithm for certain situations can help you:
  • Take a vacation.
  • Offload some work to more junior member you are trying to mentor.
  • Allow you to more easily fix a difficult problem in a stressful situation.
    
• Identifying what is important in your role – Even if you don’t make a flow for every scenario there is benefit in just using that mindset. It helps you to remember that your restores are far more important than your backups, it helps you to realize what you need to focus on with your maintenance tasks.
• Critical Thinking – If you try to take any problem you are faced with and boil it down to it’s basic parts or a flow you will be a step ahead of the problem solving game. Whether you are installing a closet system for your wife or wiring up an emergency generator after an ice storm with no electrician available. (I don’t recommend doing either of these yourself 😉 )

Chefs – Recipes

I like to eat at restaurants and when I find a great restaurant I want to tell all of my friends about it and go back and enjoy it again sometime. I hate it when I show up on another night and the taste or style is very different (and not for the better). I bet the executive chefs at most restaurants don’t like it when the dishes they have worked hard to make are changed dramatically by a clever line chef/cook. Recipes help avoid this problem. They help the restaurant staff properly order what they need for the expected guests, they help the prep staff properly ready all ingredients ahead of time and they help the line staff properly bang out quality food all night.

When my wife finds a great recipe online and makes it, it comes out awesome. Her secret? She follows the recipe rather closely the first time. She may tweak it the second time based on our feedback but she doesn’t just get the ingredient list and throw it together any old way. When I try and make something from a great looking recipe it doesn’t come out so great. My secret? I have myself fooled that I am a a Chef (hiding in DBA clothes) and I try it my way, ignoring the recipe by and large. This worked for one dish, I make a mean risotto. Everything else I have tried to make? Would have been better if I followed the recipe…

What about us?

When you work hard slaving over a setup of SQL Server, when you try to standardize where your data and log files should go, what port SQL should be listening on, etc. do you want the next installation to be radically different? I don’t, I want my environment to be consistent; there are enough customer created problems for me to deal with, why create my own?

To me the database administrator’s recipe would be the checklists, the algorithms and the documentation gluing them all together. Yes most techies hate documentation, myself included, but it is a necessary force. I think of the documentation or recipe as future relief. If I document a process well enough, I can hand that off if the need arises, I can go on a family vacation without the blackberry.

The trick is finding the time for it. As I discussed in my “Where do I start?!?!” post, sometimes you just have to make less important reactive stuff wait and do the important proactive tasks.

Want more content like this? Subscribe to this blog’s feed in your favorite feed reader and stay up to date with tips and tricks. I like to share techniques and free tools I use to help me with my day job, might help you out also.