I’ve lost track of the number of long term clients we have at Straight Path who started out with “It’s the vendor’s fault! They had the nerve to tell me my servers weren’t running great!” or “Our environment is perfect, it’s this (ERP/CRM/eDiscovery/Integration/etc.) software we installed that is just running like garbage!” True to human nature, the “reporting party” is often grasping for someone to take their side.
And in about 98% of the cases…
They’re Both to Blame
I’ve been a SQL Server consultant for 15 years now, and I can think of 3 out of hundreds of cases where it was all one-sided.
I both love and fear these engagements. There was that one time the vendor called – they had already told their client that this was all on them. They had the client send them their backup, and they took it and ran it in their lab – see! All was fine! (Their lab without any other uses, without any integrations, but hey – something was fast…) The vendor was paying my bill. They’d already told their client it was their client’s fault. And I was supposed to be the smoking gun testimony. I told them upfront that I’d look at everything and tell the truth, regardless of who paid the invoice.
Enough about them for now, I’ll tell you what happened at the end. For now, let’s walk through how these things usually go.
What We Usually Find on the Vendor’s Side
- Queries and DB Design tested in a clean lab or with an “ideal client” profile – not the client’s exact skew of data. Indexes that work in theory but weren’t stress-tested. Often, tests that don’t truly replicate the concurrency of live.
- Documentation that assumes clients already know all the best-practice defaults and would pass one of our onboarding health checks, not a next-next-next-install “set it and forget it” installation like we see all the time.
- A support process that asks for a backup and then tests in their own offline environment, declares “all good!” and closes the ticket. (No integrations, no other users, no other DBs, no power-saving mode, no MAXDOP set wrong, no missing maintenance.)
- Surprise when the application doesn’t function at scale the same way it did in a limited perf test environment.
The list goes on… We’ve seen all sorts of things (like the vendor who is so in love with query hints AND has scores of permanent tables in tempdb that get built on startup because they couldn’t think of more creative ways to mess up the optimizer…), but it’s usually just a simple case of a client hitting a workload that hasn’t been caught yet. Or a client of a size that is finally stressing an application out, and the vendor hasn’t really had a thorough review with SQL Server experts yet.
What We Usually Find on the Client’s Side
- Ron Popeil installed the SQL Server (showing my age now… Remember those late-night infomercials about the rotisserie chicken? “Set it and forget it!”). They clicked Next a bunch of times, didn’t do any best-practice review ahead of time, and Poof! They have a production SQL Server. I’ve officially lost track of the number of servers in balanced mode.
- SQL Server on a VM that was sized for a file server instead of a database server. No consideration of neighbors, no VMware best practices or Hyper-V best practices were considered before they started.
- No DBA, No Maintenance – One client who called about corruption was downright angry when he told me, “But I thought the vendor’s support team was my DBA team!” when I asked about backups. Their vendor had called us in to see if we could help. We probably see more servers without maintenance running than those with it.
Again, the list could go on forever. We stopped getting surprised during SQL Server health checks. I have an ongoing “SQL Server regret” series of blog posts about that. We see the same things time and time again. The default settings, the IO setup, the VM, or Cloud VM or Cloud PaaS setup steal so much performance before you even get started looking at the code and indexing opportunities.
There’s Enough Blame for All!
I think an earworm from the 70’s might sum this up the best:
There ain’t no good guy, there ain’t no bad guy, there’s only you and me and we just disagree…. – Dave Mason (written by Jim Krueger)
I tell our team we are 1 part SQL Server/Data/Database expert, 1 part “business therapist”. It’s true, too. We are like relationship counselors, even when someone thinks they are hiring a divorce attorney.
Clients should ask vendors questions before making a purchase. I wrote a list ages ago to help get you started. Minimum requirements are there for a reason and should be considered. You can’t throw software on a noisy-neighbor laden VM or an underpowered burstable cloud VM and expect amazing performance.
Vendors should work to make sure their code is the best it can be. They should listen when their clients bring in SQL Server expertise; it’s free advice to help all their clients. They should do a better job of letting their clients know they should have a DBA on staff full-time or part-time, and they should document best practices for setup and maintenance for clients without a dedicated DBA team.
Both should remember they’re on the same team…. They just forgot that somewhere between the support ticket escalation and the finger-pointing.
And that’s usually how it goes… The answer to whose fault is it? Is almost always, “Yes.” And when each side is ready to hear it and work on their part of the story, everyone wins. And if you don’t like spending money on consultants, remember that we’re kind of like divorce lawyers – the more you fight your part of the issue, the more we make.
… And you already know how the story at the start went, don’t you?
If your team is caught in the middle of one of these conversations right now — or if your customers are blaming your software and you’re not sure they’re wrong — we do this work every day. DM me on LinkedIn or grab 20 minutes.