TSQL Tuesday: Why Are DBA Skills Necessary

 

Quote: “Database stuff, all this programming stuff, is easy. Anyone can do it. That’s why everyone in the company has sa privileges.”

For nine months, I worked in an environment where everyone, from developers to QA to the sales people to the receptionist, had SA privileges. You know what? DBA skills are necessary.

I speak from the point of view of someone that has had to recover a server after a salse person helpfully “cleaned up the temporary stuff on the server” by dropping tempdb, causing a late deployment for a client. I speak from the point of view of the guy who kept a window open on his desk with the database restore script ready to run, all day long, because of “accidents” that stopped development until I could get the database restored. I speak from the point of view of someone who had to explain, multiple times, “No, you can’t hit the undo button to get that data back” in client systems. And yes, I could go on and on about this nine month gig, but I won’t. You might consider it an extreme case, mainly because, it was.

Instead, I’ll talk about another  job. It was a free-wheeling dot com. We were doing some amazing stuff. We were running  well over 300gb of data into SQL Server 7.0 in a 24/7 environment… until the day one of the managers of the organization said “You database guys are spending too much time working on maintenance of the server. SQL Server manages itself. You will stop all maintenance, immediately, and don’t work on it any more.” It only took two weeks for the server to crash. It took us three days (72 straight hours)  to rebuild the system and recover the data, with no web site available for our thousands of clients during that entire time. Funny enough, a memo came down from management, not thanking us for putting in all the effort, no, but instructing us that we should spend adequate time performaning maintenance routines on the server.

And before you ask, yes, I can keep going. I have tons of examples. How about the application development team (different company) that built the entire database out of multi-statement table valued UDFs that called other UDFs that called other UDFs that called other UDFs… which led to a delayed production roll-out.

What I’m saying is, these are not isolated or extreme examples (OK, the first one was extreme, but only a little). This is real world stuff occurring on regular basis, every day, all over the world that negatively impact the businesses that we support. It’s not that people are stupid. They’re absolutely not. It’s just that databases and database servers, and data for that matter, are still pernicious and difficult. It’s not rocket science, not by any stretch of the imagination. Why do I say that? Because I can do it. In fact, I’ve been quite successful as a DBA and Database Developer. If I can do it, it’s not that hard. But it is a specialized set of knowledge, and one that is quite extensive. There’s just so much to learn about how data is stored, retrieved, and managed within SQL Server, it’s crazy to assume you wouldn’t need a specialist as you move deeper and deeper into managing more and more information. The lack of that knowledge, can, and does, have an impact on the bottom line.

If you’re a business person or a developer (and by developers, I’m not talking about the exceptional Brainiac that can really do it all, I’m talking a normal human being), ask yourself, do you know how to restore a database to a point in time? Just in case you don’t know, this means recover the database from backups and log backups to a point, usually just before a failure or a bad update or some other problem, again, that will cost your company money.  Ask yourself, do you know how to set up backups so you can do that point in time recovery? Do you know how to tell why a query is running really slow and keeping customers from buying your stuff? Do you know how to tell why you’re getting deadlocks and losing transactions? Do you know how to tell if a process is blocked or just taking a long time? I’m clearly belaboring the point, and this is still fairly basic information. I haven’t even started delving into configuring systems for mirroring, or gathering metrics for a performance tuning effort, or any other of hundreds of tasks that need to be performed to build large scale data management systems that keep your business in business. If you don’t know what all this stuff is, or you don’t know why it’s needed, you might just need someone in your organization who can help out.

And yeah, there are tools out there now, like Object Relational Mapping (ORM) tools, that take away vast amounts of the labor that used to go into building databases. But, having worked with teams building tools with an ORM, while they’ve successfully removed the database from their thoughts & designs by using the ORM tool, they’re still storing data on a database system. Guess what? The problems, such as deadlocks, index scans, blocking, still exist. Sometimes, these problems are even exacerbated by the ORM tool. Now what? You need a specialist.

Fine, what if you toss all the mess, get rid of relational storage, go for one of the NOSQL database systems. Yeah, that might work, a bit. But guess what. You’re still storing data. It’s still going to, in most circumstances, outlive the application that built it. It’s still going to need to be reported against, backed up, recovered, protected, performance tuned… In other words, you’re going to run into situations where, despite the fact that you’ve eliminated the things that irritate you, like SQL, like constraints, you’re still dealing with pernicious data and you’re going to need someone who has been working out how to deal with that.

Does every single company with any installation of SQLExpress need to hire a DBA? No, of course not. But as soon as your business starts to rely on having data around, all the time, you just might need the skills of someone who has been working with this stuff for a while.

12 thoughts on “TSQL Tuesday: Why Are DBA Skills Necessary

  • datachix2

    Great manifesto! I love the point that the data will outlive the applications that created it. I think that’s a point that cannot be made often enough. Just one question: Does the sales person who dropped tempdb have scars from the beating you gave him? That one is truly cringe-worthy.

    –Audrey (Datachix2)

  • SQLChap

    It is sometimes hard to explain to people why you need to spend time and effort on maintenance or why you need skilled people.

    It all comes back to Frank Brooks’ observations on “two guys in a garage”, yes you can run systems without skilled DBAs (or documentation, or maintenance or upgrade strategies or offsite backups) but if systems are important to the business then people need to treat them as such and not just some inconvenient extra expense.

  • scarydba

    Audrey,

    Thanks.

    And no, he was in Chicago at the time, so it was days later when I finally saw him. All that nerd rage was gone by then.

  • scarydba

    Thanks SQLChap

    Matt, no, there are people out there with TONS more experience than I have. As well as more smarts, more skills, etc. I know a bunch of them personally, and a constantly humbling experience it is too.

    I do my part.

  • SQLChap

    “Database stuff, all this programming stuff, is easy. Anyone can do it”

    This kind of quote is just annoying, do concert pianists have people saying “Piano playing is easy, it’s just pressing some keys, the piano does all the work”.

    So any tips for educating the un-educated types who may such comments? Surely we can’t just leave them in ignorance waiting for systems to fail?

  • scarydba

    I sure don’t equate myself with a concert pianist. I’m just a laborer with some specialized skills. But to answer your question, no, I don’t know how to educate people. Many of the stories I relate up above, I’ve been in front of the management types or the lead developers, waving my arms & crying, “Stop, stop” but they ran into the embankment anyway. It’s not simply education that’s needed. It’s somehow getting people to understand that they need an education in the first place. And I sure don’t know how to do that.

  • SQLChap

    ” It’s somehow getting people to understand that they need an education in the first place”

    I totally, agree with that. I like to use the analogy of lego bricks, anyone can stick two bricks together but it takes some skill to build a model of the fourth rail bridge that does not fall down. Probably helps that I’ve been working with Danish clients though.

  • scarydba

    Hey Wendy! See you in a few days… man I love saying that.

    Yeah, the commonality of so many of the silly, stupid, or scary experiences is a hoot.

Please let me know what you think about this article or any questions:

This site uses Akismet to reduce spam. Learn how your comment data is processed.