SAN Administrators and DBAs… Developers and DBAs… Project Managers and DBAs… The… the… the business users and DBAs… Oh, my… It seems like everywhere we look there is some group that us DBA types seem to have a problem with. I know we all don’t… But it’s one of those relationship issues we are -supposed- to have.. You know, like mother-in-laws? There’s a saying out there that goes something like, “If you seem to have a problem with everyone else – maybe everyone else isn’t the problem.”
I think we suffer from that as DBAs. Sometimes we are the problem, even when we aren’t. Why? Because we don’t chat, we don’t
Part of the solution is expressing ourselves appropriately and effectively. Let’s chit chat about that here. I’ll start off by saying I am not the one stop answer shop here and I would love to see your thoughts on how we get this wrong in the comments. Please – discuss. I’ll get the conversation going.
Improper Communication Leads To
We can choose to just talk to the other DBAs and our management and whine about the lack of understanding we see. We can vent to friends at User Group meetings or the SQL PASS Summit. That won’t solve anything. Improper communication has led to a lot of bad in the world. At NASA and some of it’s contractors you could say it was at least part of what led to the Challenger and Columbia tragedies – avoidable tragedies based on info teams had before either. We could blame a fair number of our armed conflicts on this as well as some of the tragic way certain conflicts were handled when politicians stuck in Washington failed to effectively communicate with the military tacticians in the field. It has been the cause of several avoidable aviation disasters – So much so that a simple – but effective – training process, Crew Resource Management, was created in the aviation industry.. And in our workplaces – it leads to downtime, increased IT expenditures, project delays, and tension. I’ve witnessed the effects of this failure with my clients repeatedly… One example –
Just recently I was at a shop that told me their disk layout was rated for high performance and would scale fine so we wouldn’t really need to test it like I wanted. I asked them how they knew this. Their response was basically, “Oh.. We told the off-site SAN team that this was to be high performance and we cared most about performance for the logs and the tempdb LUNs.” Sounded good but I asked if we could have a call with the SAN team to get a few things answered from them about how their particular SAN was laid out. We had that call to ask about SAN cache, RAID types, LUN layout, etc. During the call I asked how the LUNs for this particular cluster were setup… The answer back was, “Oh.. they are all slices off of different 5 disk RAID 5 groups” I asked what else was on them and they said not much yet but they were the last RAID groups available and would get the most allocations since the SAN was full until/unless more trays were purchased. I also said, “now the team here said they told you that the LUNs were all performance sensitive and that the TempDB and Log file LUNs needed to be high performance, those are also on slices of 4+1 RAID 5 raid groups?” The answer back was, “Yeah… That’s how we do all drives unless the team specifically asks for a specific RAID type or different setup… Everyone says their stuff is ‘High Performance’ but if all they tell us is the size, they get a slice off of a RAID 5 for the space they need.” I thanked them for their time and then had a conversation about communication with the SAN team with the local DBA team….
If we use this as our example and add in a little basics from crew resource management we can come up with at least a few simple rules –
- Doesn’t leave room for assumption – The DBA team assumed High Performance meant dedicated RAID 10 in this case.. It didn’t mean anything, actually. Airliners have literally crashed because the crew each assumed the other was watching fuel or that fuel was fine..
- Is Precise – “High Performance” is a bad request… It will almost always mean something different to the asker and to the asked… Spell out what you need. When you are complaining to a developer instead of saying (pardon me..), “your code sucks, I won’t implement this” – try and precisely tell them what’s wrong and how they can improve it.. They’ll learn something and you’ll have an ally in development as a result.
- Seeks clarification – “High Performance” may be a bad request. But to see that and then just deliver the standard is intellectually dishonest and, well, lazy. If someone asks for something and there is room for assumption – clarify it. I talk about this in my “If you see something, say something” post.
- Is Free of Attitude – I know… Developers are supposed to hate DBAs and DBAs are supposed to be grumpy to developers.. It ought not be that way, though. Listen, understand where the other person is coming from and help them out.
- Is Concise – Yeah…… I’m up to 1,000 words already. I’ve been doing this for 13 years and I’m still working on it. It helps though. If your request is lost in a novel you won’t see a response. Especially in this age. Be concise, Mike…
- Seeks Confirmation – In crew resource management there are a few steps to communication.. The process is something to the effect of 1.) Get their attention, 2.) State your concern/intention 3.) Explain the problem/reason from your view, 4.) Propose a solution or approach and 5.) Seek agreement/confirmation… Close the deal! “Hey SAN Team – these drives are housing our critical app that will have a lot of transactions and runs our entire operation.. Performance is going to be visible and important here, I think we should go dedicated RAID 10 with 6 disks for TempDB and a dedicated mirror for logs because this is what we’re expecting for load. Do you have the spindles for that and does this approach make sense to you?” This is clear, concise and gives them an opportunity to counter, seek clarification, disagree or agree. If they agree and don’t deliver you now have something to fall back on and you figured out about the issues up front instead of the week of go live deployment.
Speaking of being concise, I’ll stop here. What are your tips and tricks to avoid “What we got here… Is a failure to communicate?” syndrome? How do you get along with your SAN team? What irks you about the way your DBAs talk to you? What could politicians learn from us? Share your thoughts in the comments.