What do checklists, recipes or algorithms have to do with Database Administration? A lot… If we study a few career paths we can see how a checklist or process flow makes their lives easier. I love stealing ideas from everyday life, working them into something useful for my DBA career and I think we have an opportunity to do that here.
Let’s take a brief look at three careers and the organizational tool they use and see what we can steal from them.
Pilots – Checklists
The next time you board a flight, take a peek into the cockpit. You may notice checklists lying around or maybe even being gone through. Before, during and after most flights the air crew are going through a checklist. They use these for routine purposes (Pre-Flight walk through, preparation for takeoff, preparation for landing, takeoff itself, landing itself) and they use them for emergency procedures (Bird Strike, Engine Fire, Wing Stall, etc).
Look at preparing for takeoff. This is something they have done thousands of times. They can probably do it in their sleep yet they still use a checklist. They know that if something is missed, it could be catastrophe so they use their checklist. The aspects of their job they might see as mundane still get a checklist. They question each other and make sure they follow the process the same way each time.
In an Emergency, an action only performed in training will definitely be aided by a checklist. It has been amazing to me that they use them in serious situations. My first thought was always, “How could they possibly have enough time to read one?!” After thinking about it, I now wonder how they could possibly do it without one. Some pilots fly different planes and there are different procedures on each, flight crews may have never worked together before. The checklist ensures that the right steps are done in the right order. Look at the history of airline disasters and you will probably see the genesis of some of these checklists.
What about us?
Hopefully most of us are already using checklists of some sort. If not here are some thoughts around them:
- Are you an overwhelmed DBA trying to get more staff to help? You get that staff and have no checklists? Now you have to work even more to ensure consistency in processes.
- Uh-Oh, the dreaded Restore request just came in and it’s an emergency. The CIO is in your office, upper management is drilling you with questions while you are trying to do it. Sure wish you had your restore process documented and in checklist format, right?
- Ahhhh Clusters. They are so fun to setup and easy too, right? If you have ever set one up you remember the “oopses” you had the first time, I hope. You had to go back and create the groups because you aren’t domain administrator, you had to go find the e-mail with the link to the info about setting up DTC at the last minute, etc. What if you created a checklist and documentation as you did the first one, specific to your environment and then you followed that for future installations?
Doctors – Algorithms
If your loved one shows up to the hospital with one of several serious conditions, the medical staff my reach for an algorithm for their symptoms. In my spare time I belong to a volunteer fire department and have had various levels of EMT training and certification where I have seen flow-chart style algorithms used frequently.
One of the specialized certifications that comes to mind is one used in various cardiac situations from the AHA. This is a system used by Doctors, Nurses, Paramedics and other health professionals to deal with extremely urgent, life threatening situations. Various algorithms were created to deal with various patient situations. They ask questions, remind you to perform a certain course of treatment and then respond to changing patient responses.
The next time you are in a hospital, look for the “crash cart”, it is highly likely that you will see a flip book of flow charts attached to or on top of the cart. Many hospitals train with these flip books as part of their response. Can they work without them? Of course they can but having them readily available has proven to improve results. It isn’t weakness on their part but an understanding that in times of stress things can be overlooked. In fact many “code blue” response teams in a hospital setting will designate someone with the role of note taking/algorithm presenting. They will call out the sequences of steps and confirm actions and reactions.
Above is one example of such a flow chart used for a patient who is not breathing without a pulse – Cardiac Arrest. It boils down all of the training they have received, goes through the questions and actions that have been hammered into them for training and guides them through the proper actions for their patient.
What about us?
If a medical professional, who has likely received more training than you or I in our field, has found that an algorithm helps troubleshooting a person why can’t we see the need in troubleshooting a less complex database?
They can help us with:
- Troubleshooting – This seems to be my raison d’etre lately with posts about troubleshooting methodologies or not trusting every solution a quick web search shows you. Even still, imagine how laying out an algorithm for certain situations can help you:
- Take a vacation.
- Offload some work to more junior member you are trying to mentor.
- Allow you to more easily fix a difficult problem in a stressful situation.
- Identifying what is important in your role – Even if you don’t make a flow for every scenario there is benefit in just using that mindset. It helps you to remember that your restores are far more important than your backups, it helps you to realize what you need to focus on with your maintenance tasks.
- Critical Thinking – If you try to take any problem you are faced with and boil it down to it’s basic parts or a flow you will be a step ahead of the problem solving game. Whether you are installing a closet system for your wife or wiring up an emergency generator after an ice storm with no electrician available. (I don’t recommend doing either of these yourself 😉 )
Chefs – Recipes
I like to eat at restaurants and when I find a great restaurant I want to tell all of my friends about it and go back and enjoy it again sometime. I hate it when I show up on another night and the taste or style is very different (and not for the better). I bet the executive chefs at most restaurants don’t like it when the dishes they have worked hard to make are changed dramatically by a clever line chef/cook. Recipes help avoid this problem. They help the restaurant staff properly order what they need for the expected guests, they help the prep staff properly ready all ingredients ahead of time and they help the line staff properly bang out quality food all night.
When my wife finds a great recipe online and makes it, it comes out awesome. Her secret? She follows the recipe rather closely the first time. She may tweak it the second time based on our feedback but she doesn’t just get the ingredient list and throw it together any old way. When I try and make something from a great looking recipe it doesn’t come out so great. My secret? I have myself fooled that I am a a Chef (hiding in DBA clothes) and I try it my way, ignoring the recipe by and large. This worked for one dish, I make a mean risotto. Everything else I have tried to make? Would have been better if I followed the recipe…
What about us?
When you work hard slaving over a setup of SQL Server, when you try to standardize where your data and log files should go, what port SQL should be listening on, etc. do you want the next installation to be radically different? I don’t, I want my environment to be consistent; there are enough customer created problems for me to deal with, why create my own?
To me the database administrator’s recipe would be the checklists, the algorithms and the documentation gluing them all together. Yes most techies hate documentation, myself included, but it is a necessary force. I think of the documentation or recipe as future relief. If I document a process well enough, I can hand that off if the need arises, I can go on a family vacation without the blackberry.
The trick is finding the time for it. As I discussed in my “Where do I start?!?!” post, sometimes you just have to make less important reactive stuff wait and do the important proactive tasks.
Want more content like this? Subscribe to this blog’s feed in your favorite feed reader and stay up to date with tips and tricks. I like to share techniques and free tools I use to help me with my day job, might help you out also.
14 thoughts on “Checklists, Recipes and Algorithms”
Another point to add about good documentation and lists, training. It will decrease the amount of time needed to get a new DBA up to speed with the current systems and conventions.
We use a Wiki for most of this stuff, this way the documentation is always up to date and we don’t waste bandwith emailing docs and pdfs around
Great point, Stephen. How many times have we been new DBAs and had to waste company time trying to gather basic information? We could have instead been reading those documents and handling issues right off the bat. Either helping the DBA team we were brought onto or helping our customers left hanging by the lack of a DBA.
Thanks for the comment.
Denis – Awesome suggestion to use a wiki site for this type of information. Great way to grow the documentation, store the history and allow for easy editing.
Plus it can actually trick us into actually doing the documentation :-)" Make sure to update the wiki site" just sounds like a nicer order than "Please provide documentation." = )
But what can chefs, pilots, and doctors learn from DBAs? 🙂 🙂
TV chefs always seem to have the same dish prepared multiple times (redundancy) at various steps in the process they can pull another dish out and proclaim "this is what it will look like when it is finished!", (backup and restore)
Doctors don’t need to count sponges during an operation. If they discover they are missing a sponge, just do a rollback.
Pilots use simulators (POCs) for training. After taking off, when a flock of geese flies into the engines, and they lose all power, they react calmly and put the plane down in the nearest river (stress testing). When they are flying smaller equipment such as a Saab 20-seater, they often reposition passengers to balance the ariplane (load balancing)
That’s great, Bill!
The doctors do more of a "pull back" or "rip back" than a rollback though, eh? Thanks for the comments. Great lessons 😉
Does anybody have a checklist before you actually start on a long trip. Half the way through your dont want to think , Oh! did I lock the doors or did I switch off the oven, did I throw the garbage out? …
That will be really really useful
As a pilot and former trainer for the US Nuclear Navy I can say that checklists are CRITICAL for safe and correct operation/response. I will add that when the fit hits the shan there are IMMEDIATE ACTIONS required for various emergency scenarios that are MEMORIZED and rehearsed so they truly become muscle-memory type responses. You simply cannot reference a checklist when doing so would a) take important moments away from the initial response or b) distract you from the need to keep flying the plane or monitoring the nuclear power plant. Once the situation is stabilized after completion of the memory-based responses THEN you go to the checklist and verify you completed the immediate actions and continue down with further monitoring/response.
I believe we as DBAs could benefit from checklists for a number of routine and emergent situations – nice article!