“It was the best of database consulting times…it was the worst of database consulting times…”
One of the most common things we do here at SSG is help new customers who have hired us because they have run into big problems, usually performance related. In many of these cases, there is either no DBA on staff at all, or the one they do have is an “accidental DBA”—someone who hasn’t had the training and experience necessary to really understand what is going on “under the hood.”
When we end up on the scene, the situation is:
- There is someone on the staff whose job is to take care of the databases, whether as a “real” DBA or not.
- They haven’t been able to fix the problems, and have sometimes made issues worse.
- Management has brought in consultants to fix the problems.
The on-staff DBA in this situation has two choices: They can get defensive and allow their pride to prevent them from learning from experienced consultants. Or, they can look at it as an opportunity to learn from an expert, improve not only the situation in their shop but also their skills, and in the process look good to the boss.
Fortunately, the latter is much more common. And yet we’ve run into enough of the former that I thought I’d tell a couple of stories. These are of course fictional but are based on real experiences I’ve had as a consultant.
Katie’s Story
Katie is a web developer at ACME Corp who has had the role of DBA fall to her by default because nobody else really understands databases and she’s the closest thing they’ve got. She’s pretty good at writing T-SQL code that works functionally, but she hasn’t ever been mentored by a senior person. As a result, she’s unaware of many best practices and things that should be done for the database to perform well.
As ACME’s database grows due to the success of its product, the company begins to have major performance problems. Management realizes Katie is in over her head (through no fault of her own) and they need to bring in some outside expertise to help. When her boss tells her this, Katie is excited! Finally, she’ll have someone who can mentor him and show her the ins and outs of SQL Server—the techniques and insights that will help her get ACME’s databases really humming.
By welcoming the support of an outside consultant and not feeling threatened by him or her, Katie can see first-hand what a DBA can and should do, taking her beyond her 101-level “just the basics” approach to all sorts of ways she can manage and optimize ACME’s systems. In the process, she’s making himself more valuable to the company, and more marketable should she ever decide it’s time for a change.
For us, the consultants, this kind of interaction with on-site staff is always encouraging. When our efforts are done, obviously we know the situation is better, but more importantly, we know the on-staff personnel should be better equipped to handle things. And should our help ever be needed again, we’ve already established a positive relationship with people we’ll be working with.
Bill’s story
Bill has been working in the DBA role for XYZ Inc., and established himself with some early and easy wins. The problem is that those accomplishments have given him a false sense of his own abilities. He doesn’t know as much about SQL as he thinks he does. He compounds his shortcomings by not getting involved in any of the myriad free resources available to a DBA looking to improve his or her skills. His shadow never darkened the halls of the local user’s group meeting, and the Internet is merely a tool for solving his immediate problems.
So Bill had good intentions when he checked to make sure that backups were running correctly, but he never verified his backups. He didn’t take any action where database integrity checks and index optimization maintenance scripts weren’t being utilized. He had some disk space problems, so he decided that a nightly shrink database might be a good thing.
Then one day the data drive on one of his database servers had a hard failure. The system admins fixed the problem, but he had to restore the databases. The full backups restored as desired. However, the differentials failed to restore. Ouch! Three days of data gone, just like that.
At this point, higher ups brought in consultants. In contrast to Katie’s welcoming attitude, Bill is surly and uncooperative when they arrive. In his mind, they are a threat because they have been asked to fix what he couldn’t. But instead of watching and learning, he grumbles. He misses the opportunity to understand how to avoid the situations that caused the problem.
That’s bad enough, but this is even worse: Once the consultants finish the job, he starts to undo all of the fixes they implemented. It’s mindboggling to consider someone would resent the help so much that they would, in essence, sabotage the completed work. Needless to say, his actions put his employment in jeopardy.
Don’t be a Bill
The moral of the story is simple enough: when we or other consultants show up to help, just know we are there, in fact, to help. We have plenty of clients with on-staff DBAs, accidental or not. We don’t want to replace you, and if you take advantage of the opportunity, you can gain from the experience. It’s your choice.
