I get to tell you an embarrassing story and I get to kick off a series of posts I’ve wanted to do for awhile now. Why? Because it’s “meme Monday” Check out Tom LaRock‘s post about the topic and the idea he came up with a couple months back.
This week Tom asked us to talk about a “Dumb SQL Question” we’ve asked or wanted to ask. I am twisting the topic a bit and will talk about a question I was once asked (a really good question, if only a bit late). I’m also going to talk about a disaster causing attitude. In fact, I am going to start a weekly (or more frequent) series of posts on lessons from disasters. I’ll focus on aviation disasters and see what sort of improvements we can make in our day jobs as a result. This is a topic that interests me to no end – learning lessons from others’ (and my own) mistakes – In fact I had a really fun time interacting with the audience at the SQL Rally sharing a presentation of such lessons (The title of that talk, Iceberg, Dead Ahead! is a bit misleading, it was all aviation).
All of the posts in this series will be categorized as “DisasterLessons“ and I’ll get to a proper introduction and all later in the week. Consider the ending of this post a quick preview.
But on to that story, first…
Pride Goes Before Destruction, A Haughty Spirit Before a Fall
Let’s go back about 9 years. I’m a few years into my SQL Server career. I know some stuff but mostly I didn’t know a lot – in fact I still don’t know (or appreciate) what I don’t know yet. But I knew a bit more than “the new guy.” He was a VB developer, a talented, smart and experienced guy. He’d been around about 5-6 years more than I had in the industry but hadn’t done much SQL yet. It was my job to train him on some of the aspects of the job. We worked for a software vendor. We had some pretty prestigious clients whose names you’d know instantly. In fact we had a tremendous market share in this particular industry. Well we did some DBA work, some database development and custom SQL work for clients whose needs ran outside of the typical “out of the box” functionality. We were doing a day time dial-in (and I mean dial in) to our biggest client. They were the client whose money trumped all other clients. We added new features because this client said “jump”. We were doing some cleanup work for them….
I was proudly showing this new (and older, more experienced, more talented) colleague all about SQL Server and our product. I was probably showing him how well I could navigate Query Analyzer and write a query on the fly. How well I had mastered our data model and really stupid table names and relationships. I forget what we were doing but we were changing a flag or column value for a large table (maybe it was customers, or companies that had done business for/with this large client of ours). They had typed values wrong for a few of the customers/companies because of a bug and it would be easier for us to fix them with a script. I was so busy talking and probably showing off that I went ahead and quickly typed up that query. To keep it simple let’s say the query looked like this:
UPDATE PoorlyNamedCustomerTable SET SomeColumnTheCustomerCaresAbout = ‘some value that is right for these 5 rows’
I had my keywords in caps, my indentation was sharp looking. I think I even had to do a join for the update to grab some info from a relationship table that linked people and companies. Everything looked nice. I clicked Execute.
At that same moment I clicked execute this colleague threw out today’s question or maybe it was a statement. Either way, my stomach must have landed in our shipping department a floor or two below us when he said that one word in that questioning tone of voice –
All that flashed through my head was something like a mixture of “Oh, Crap! The Customer! This is a day time change! I have to call them!” and
probably something along the lines of “Young grasshopper, your teacher has become the student” He got to see my troubleshooting skills and customer service skills at work. Probably with a healthy dose of panic and wounded pride thrown in. Customer ended up fine and we had a good one word joke that has lasted through the years we’ve known each other but yeah… That was a pretty dumb question. Well not the question but the fact it had to be asked.
Disaster Causing Attitude?
Had I been a bit slower on hitting F5, I bet the timing would have worked differently. His question would have instantly made me say “oh you nummy” and I would have added the appropriate WHERE clause. Here was a new guy unafraid to ask the person assigned to him as a trainer a question that, well, could have been construed as insulting or embarrassing. In a cockpit that ability is crucial. On a technology team that attitude is also pretty important. You can’t be afraid to ask a “tough” question of a colleague/superior/PM/manager/etc. When I fly on a plane, I want the first officer to feel comfortable asking the Captain, “WHEREE?!!” if he or she does something silly.
In aviation, cockpits didn’t always work this way, though. In fact it wasn’t until a number of accidents pointed out a problem with too much respect for the chain of command, titles, seniority or hierarchy that a new flight crew training methodology called Crew Resource Management (CRM) was established.
Briefly – CRM abolished the notion that the Captain was the one and only authority. Before CRM, questioning the Captain was seen as a high insult. Mentioning something of concern wasn’t necessary because the Captain was all seeing and all knowing. CRM really worked to put all on an equal footing when it came to responsibility for the safety. Leadership training aspects helping senior crew members and Captains to listen and respond to advice, suggestions, questions was big. Training on how to communicate assertively and be bold enough to state a concern or potential problem was big for all parties. CRM was the natural response to the acknowledgment that human factors were behind most accidents in the aviation world.
“Where?!?” “How’s the Fuel?”
In a pre-CRM world that person being trained would have likely kept his mouth shut – “Mike is training me, he must know what he is doing”. In our case he was a bit too late but the impact was easily mitigated by a restore to a new DB and some quick joined updates with no impact. In aviation the pre-CRM mentality caused unfortunate accidents.
Take United Flight 173, for example. Basically this flight had a landing gear indicator issue. The flight crew spend too much time troubleshooting beyond their checklist and too much time focusing on cabin preparations and being absolutely sure that they were going to try to land. After nearly an hour of troubleshooting and preparations that should have taken half that time at most, they made their approach to land. The only problem? They hadn’t enough fuel to make it and crashed, 10 people were killed, another 24 were seriously injured. While many factors contributed to this accident (and we’ll get to them all as we go through this series), the one we care about here is summed up in the NTSB findings –
“The failure of the other two flight crewmembers either to fully comprehend the criticality of the fuel state or to successfully communicate their concern to the captain.”
If you read the Cockpit Voice Recorder transcript from this flight you can hear a couple really passive references to fuel. Almost as though they could have been said, “Hey uhh Captain, sir, we ahh don’t want to bother you and it’s probably nothing, I mean you know what you’re doing but the fuel situation is well, do you think we’re okay? ahh nevermind, sorry to bother you.”
United Airlines subsequently became the aviation’s industry first implementer of CRM. It’s now mandatory training in the aviation industry as well as many others.
The takeaway from United 173’s passive fuel warning (and my colleagues lack of fear to question my stupidity) –
If You See Something, Say Something (Stolen from the Department of Homeland security’s Orwellian posters)
It’s that simple. If you believe something is wrong, may be wrong or could go wrong – Tell someone. Don’t be afraid of being the one to put the brakes on the project. Don’t be afraid to be the one that gets laughed at during lunch because you misunderstood something. Because maybe, just maybe, your concern is valid and the project you save could be your own. You could spend your weekend having fun instead of picking up pieces from an implementation gone wrong. If you were wrong, so what, at least you paid attention. If you were right and said nothing? Well then you just screwed up that chance to be part of a successful team.
Subscribe to my feed to come back next week (or maybe later this week) when I properly kick off the DisasterLessons series and go through some of the attitudes and actions we talked about in that SQLRally presentation and work together an idea of what the series will look like.
13 thoughts on “If You See Something, Say Something”
Great post Mike! Thanks for sharing a valuable and important message. I really liked how you used a real world event to drive the message home.
All too often I see technical professional’s biting their tongue when something should really be said. For the DBAs out there it’s especially important to voice any concerns you may have. Never go out quietly into the night, if you’re on my team I would much rather you go out kicking and screaming in protest 🙂 than knowingly let a potentially bad thing play out.
Good story, Mike! I’m going to remember to ask myself “WHERE?!” whenever showing new guys the ropes on our production systems. 🙂
I’ve had a similar unfortunate experience, and your description of your stomach is right on the mark.
I’ve gotten in the habit of putting a BEGIN TRAN at the beginning of any query window where I’m doing a sensitive update. Then if I see a “xxxx row(s) affected” message that looks okay, I’ll manually execute a COMMIT, but if I see a huge number and realize I screwed up, I can ROLLBACK.
It’s saved me several times… I type so fast sometimes that I might hit an F5 or CTRL+E by mistake.
Terrific post! How well I know the stomach-churning feeling of seeing an entire table updated instead of the two or three rows you expected. I think the worst thing I’ve ever seen was when a young DBA colleague, experimenting with SQL and with the system views on an Oracle database, thought he’d write himself a little script that would drop each and every table in the current schema. It was, of course only after he’d set it running, and phones started ringing, that he realised he’d unleashed this engine of destruction on the production system and not in a dev environment as he’d thought. The lesson? ALWAYS be sure of your current context when launching potentially deadly code.
Great post and I completely agree! It has been my experience that the ‘dumbest’ questions are usually the ones I learn the most from – both as the questioner and the questioned.
I’ve asked lots of dumb questions along the way, but I’ve also been caught speechless more than once when trying to explain a concept to someone and they fire off a seemingly basic question…and then enjoy my blank “Ummmm…” face. Delicious Humble Pie. =)
Thanks for the comment (and the modesty!), David.. Humble pie doesn’t taste that great but when we can use our deserved serving to improve ourselves it isn’t a horrible experience (in hindsight). I know the “ummm” you’ve muttered. Those are the times when I realize I don’t know something as well as I thought I did. If I can’t explain the simplest elements of something then I probably haven’t grasped it as well as I thought I did.
If You See Something, Say Something. I would like to see a blog that says,”If you have done something(to foul up a SQL server) say something”, but tell the truth, man up to it. After I realized early in my career that lying about my inadequacies and mistakes don’t get you anywhere. Admitting a mistake, asking for help and getting involved solving the problem does advance your career. The Truth will set you free.
Ha – I agree Ian (and apologies for the delay, you were hidden among some old spam comments and I happened to catch this while cleaning some spam. I actually wrote a blog post sort of about that before – Bill Clinton Wasn’t Impeached For… Shared an example of my own bumbling towards finding that same solution out.
Thanks for the great comment!