If You See Something, Say Something

I get to tell you an embarrassing story and I get to kick off a series of posts I’ve wanted to do for awhile now. Why? Because it’s “meme Monday” Check out Tom LaRock‘s post about the topic and the idea he came up with a couple months back.

This week Tom asked us to talk about a “Dumb SQL Question” we’ve asked or wanted to ask. I am twisting the topic a bit and will talk about a question I was once asked (a really good question, if only a bit late). I’m also going to talk about a disaster causing attitude. In fact, I am going to start a weekly (or more frequent) series of posts on lessons from disasters. I’ll focus on aviation disasters and see what sort of improvements we can make in our day jobs as a result. This is a topic that interests me to no end – learning lessons from others’ (and my own) mistakes – In fact I had a really fun time interacting with the audience at the SQL Rally sharing a presentation of such lessons (The title of that talk, Iceberg, Dead Ahead! is a bit misleading, it was all aviation).

All of the posts in this series will be categorized as “DisasterLessons“ and I’ll get to a proper introduction and all later in the week. Consider the ending of this post a quick preview.

But on to that story, first…

Pride Goes Before Destruction, A Haughty Spirit Before a Fall

Let’s go back about 9 years. I’m a few years into my SQL Server career. I know some stuff but mostly I didn’t know a lot – in fact I still don’t know (or appreciate) what I don’t know yet. But I knew a bit more than “the new guy.” He was a VB developer, a talented, smart and experienced guy. He’d been around about 5-6 years more than I had in the industry but hadn’t done much SQL yet. It was my job to train him on some of the aspects of the job. We worked for a software vendor. We had some pretty prestigious clients whose names you’d know instantly. In fact we had a tremendous market share in this particular industry. Well we did some DBA work, some database development and custom SQL work for clients whose needs ran outside of the typical “out of the box” functionality. We were doing a day time dial-in (and I mean dial in) to our biggest client. They were the client whose money trumped all other clients. We added new features because this client said “jump”. We were doing some cleanup work for them….

I was proudly showing this new (and older, more experienced, more talented) colleague all about SQL Server and our product. I was probably showing him how well I could navigate Query Analyzer and write a query on the fly. How well I had mastered our data model and really stupid table names and relationships. I forget what we were doing but we were changing a flag or column value for a large table (maybe it was customers, or companies that had done business for/with this large client of ours). They had typed values wrong for a few of the customers/companies because of a bug and it would be easier for us to fix them with a script. I was so busy talking and probably showing off that I went ahead and quickly typed up that query. To keep it simple let’s say the query looked like this:

UPDATE PoorlyNamedCustomerTable SET SomeColumnTheCustomerCaresAbout = ‘some value that is right for these 5 rows’

I had my keywords in caps, my indentation was sharp looking. I think I even had to do a join for the update to grab some info from a relationship table that linked people and companies. Everything looked nice. I clicked Execute.

At that same moment I clicked execute this colleague threw out today’s question or maybe it was a statement. Either way, my stomach must have landed in our shipping department a floor or two below us when he said that one word in that questioning tone of voice –

WHEEERRRRE?!?

All that flashed through my head was something like a mixture of “Oh, Crap! The Customer! This is a day time change! I have to call them!” and

probably something along the lines of “Young grasshopper, your teacher has become the student” He got to see my troubleshooting skills and customer service skills at work. Probably with a healthy dose of panic and wounded pride thrown in. Customer ended up fine and we had a good one word joke that has lasted through the years we’ve known each other but yeah… That was a pretty dumb question. Well not the question but the fact it had to be asked.

Disaster Causing Attitude?

Had I been a bit slower on hitting F5, I bet the timing would have worked differently. His question would have instantly made me say “oh you nummy” and I would have added the appropriate WHERE clause. Here was a new guy unafraid to ask the person assigned to him as a trainer a question that, well, could have been construed as insulting or embarrassing. In a cockpit that ability is crucial. On a technology team that attitude is also pretty important. You can’t be afraid to ask a “tough” question of a colleague/superior/PM/manager/etc. When I fly on a plane, I want the first officer to feel comfortable asking the Captain, “WHEREE?!!” if he or she does something silly.

In aviation, cockpits didn’t always work this way, though. In fact it wasn’t until a number of accidents pointed out a problem with too much respect for the chain of command, titles, seniority or hierarchy that a new flight crew training methodology called Crew Resource Management (CRM) was established.

Briefly – CRM abolished the notion that the Captain was the one and only authority. Before CRM, questioning the Captain was seen as a high insult. Mentioning something of concern wasn’t necessary because the Captain was all seeing and all knowing. CRM really worked to put all on an equal footing when it came to responsibility for the safety. Leadership training aspects helping senior crew members and Captains to listen and respond to advice, suggestions, questions was big. Training on how to communicate assertively and be bold enough to state a concern or potential problem was big for all parties.  CRM was the natural response to the acknowledgment that human factors were behind most accidents in the aviation world.

“Where?!?” “How’s the Fuel?”

In a pre-CRM world that person being trained would have likely kept his mouth shut – “Mike is training me, he must know what he is doing”. In our case he was a bit too late but the impact was easily mitigated by a restore to a new DB and some quick joined updates with no impact. In aviation the pre-CRM mentality caused unfortunate accidents.

Take United Flight 173, for example. Basically this flight had a landing gear indicator issue. The flight crew spend too much time troubleshooting beyond their checklist and too much time focusing on cabin preparations and being absolutely sure that they were going to try to land. After nearly an hour of troubleshooting and preparations that should have taken half that time at most, they made their approach to land. The only problem? They hadn’t enough fuel to make it and crashed, 10 people were killed, another 24 were seriously injured. While many factors contributed to this accident (and we’ll get to them all as we go through this series), the one we care about here is summed up in the NTSB findings –

“The failure of the other two flight crewmembers either to fully comprehend the criticality of the fuel state or to successfully communicate their concern to the captain.”

If you read the Cockpit Voice Recorder transcript from this flight you can hear a couple really passive references to fuel. Almost as though they could have been said, “Hey uhh Captain, sir, we ahh don’t want to bother you and it’s probably nothing, I mean you know what you’re doing but the fuel situation is well, do you think we’re okay? ahh nevermind, sorry to bother you.”

United Airlines subsequently became the aviation’s industry first implementer of CRM. It’s now mandatory training in the aviation industry as well as many others.

The takeaway from United 173’s passive fuel warning (and my colleagues lack of fear to question my stupidity) –

If You See Something, Say Something (Stolen from the Department of Homeland security’s Orwellian posters)

It’s that simple. If you believe something is wrong, may be wrong or could go wrong – Tell someone. Don’t be afraid of being the one to put the brakes on the project. Don’t be afraid to be the one that gets laughed at during lunch because you misunderstood something. Because maybe, just maybe, your concern is valid and the project you save could be your own. You could spend your weekend having fun instead of picking up pieces from an implementation gone wrong. If you were wrong, so what, at least you paid attention. If you were right and said nothing? Well then you just screwed up that chance to be part of a successful team.

Subscribe to my feed to come back next week (or maybe later this week) when I properly kick off the DisasterLessons series and go through some of the attitudes and actions we talked about in that SQLRally presentation and work together an idea of what the series will look like.