Dec 10 2008

SQL Quiz Part 2

Gee, thanks TJay. I have apparently been tagged to take part in a quiz started by Chris Shaw. It’s an experiment in learning where a series of people answer the same question, hopefully spreading some useful knowledge. The question:

What are the largest challenges that you have faced in your career and how did you overcome those?

This is tough. You could answer based purely on technical issues, what was my toughest technical challenge. You could talk about career issues, tough bosses, ignorant co-workers, incompetent subordinates, clueless clients. You could also talk about shifting jobs as companies fail or are mismanaged or you move. You could also talk about missed opportunities, when you faced the large scale challenges and didn’t quite rise to the occasion. Anyway… My Answers)

1) My first challenge is ongoing, but about five years old at this point. Five years ago, I was acting as the gatekeeper on development machines. I would vet applications prior to their release to ensure they met our companies minimum standards. The first part of this challenge was actually the easiest to fix. Checking applications right before they’re due to release is a pretty bad scheduling. We now monitor apps as they’re developed, rather than at the end of the process attempt to make changes. However, back then, that was how it was getting done. I received word that a major application was about three weeks away from release so I needed to take a look at it. I checked it out. It was beautiful. You’ve never seen so crystaline a structure in all your life. The developers were geniuses. Evil and misguided, but geniuses. They had built an entirely object oriented mechanism within the database with user defined functions (UDF’s) calling UDF’s that called other UDF’s, all in this incredible chain. It was amazing. And it worked, very well… As long as you only had about 10 rows in the tables and no more than one user at a time. Since this was going to support millions of rows and hundreds of simultaneous users… So, I went back to them and said that they couldn’t do this, that the entire database needed to be rewritten. “Why?” was the response. Well, because it won’t work. “Why?” Because these things behave poorly. “Why?” Well, because they do. “Why?” I didn’t have THE answer (which is, that since multi-statement UDF’s have no statistics, they make execution plans based on a single row of data, which is fine as long as there is only a single row, but stinks on toast when there are more rows). Their argument was, if the database let them do it, it was OK and any performance problems were on my end, not in the TSQL code that was written. The challenge, and it continues to this day, was to find THE answer and be able to supply it… on demand. Our developers are somehow under the impression that if you have to go and look something up or research something it means you don’t actually know what you’re talking about. I don’t know how that impression has come to them, but there it is. So I am daily trying to get in front of whatever it is they’re doing that day (ORM, nHibernate, LINQ, SCRUM, Agile, whatever the latest development buzzword might be) so that when they come to the database and declare the next PERFECT solution (like the UDF’s) I can answer “WHY” instantly. It’s an incredible challenge that I have to struggle with constantly.

2) I worked at a Dot Com that I probably shouldn’t name (DASH.COM) for a bunch of guys who graduated from… TOP SCHOOLS. These guys were IVY LEAGUE! You have to use all caps and exclamation points because, that’s the way they talked about their education. They really looked down on anyone that didn’t go to a TOP SCHOOL and tended to get really pissed when those of us who didn’t go to TOP SCHOOLS opened our mouths. The core challenge was not strangling these little twerps with their frigging Harvard Business School diploma’s, but there were other challenges. Such as the time when they came to the DBA group and declared that we had spent too much time doing maintenance and we were slowing down the development process. Therefore, no more maintenance on the databases, just concentrate on development to the exclusion of all else. OK. We told them things weren’t going to go well, but these guys didn’t know enough to trust us, despite the TOP SCHOOL education. What they needed was more education, and it was up to us, the DBA’s to provide it. Unfortunately, at the time, I didn’t have enough experience to realize that. The new method worked great for almost a month. Then one day, the log backups failed the same day a new piece of software was implemented in the production system. We hadn’t completed our monitoring processes, so we didn’t find out about the log failure. This was back in 7.0 and the system actually could fill the drive. Our drive filled and the entire server locked up. We were down. Two co-workers and I spent three days straight, no sleep, trying to recover the system, failing, and finally rebuilding the system, losing one day of data. Herculean effort. I wrote up full documentation on what went wrong and why. A couple of days later we received word that we should spend as much time as we needed ensuring that proper database maintenance was completed. While it took a catastrophe to pound it home, the one thing I learned from this is that documentation for why you’re doing something is the best thing you can provide management. Assuming they know enough to trust you is a bad assumption. You need to make sure you communicate what can go wrong ahead of time. By the way, they don’t teach apologies at TOP SCHOOLS.

Not great answers I suppose, but both helped push me down the track a bit farther.

Now, my tags… Ah, first, of course, GilaMonster at SQL In the Wild. Take it away Gail.┬áNext… Tim Mitchell at Bucket of Bits.

7 Comments

  • By brento, December 10, 2008 @ 8:04 am

    DAMN, I had that exact same thing happen with UDFs. We had a group of developers in a remote office, and they were learning SQL and .NET by reading books, but not talking to anybody who’d actually done it. Their application came into our shop when it was time to have the first customer proof-of-concept, and of course performance went through the floor. I had to pinch myself to believe they were actually using UDFs with a data warehouse – 300m+ rows. What a mess.

  • By Jack D Corbett, December 10, 2008 @ 8:27 am

    Interesting Grant. I have a good friend who has an IVY LEAGUE education. He’s probably one of the smartest and most gifted developers I know. His issue is that he he looks to take things to theoretical perfection so we mere mortals can’t understand or fix what he has done. Common issue with developers, they want to apply OOP to the DB and it doesn’t usually work well.

  • By scarydba, December 10, 2008 @ 9:02 am

    I’ve met more people from the Ivy League since then and lots of them are decent human beings. However, there really does seem to be a clustering of jerks that attend TOP SCHOOLS. I’m not sure if the school draws them or creates them, but there it is.

  • By Jack D Corbett, December 10, 2008 @ 9:18 am

    I think the schools DRAW them. I mean, either they have lots of money and are a legacy, or they are insanely intelligent and are never challenged and never develop people-skills or common sense.

    Now remember I said this is a good friend, he just is TOO smart. I should mention he got me started in IT and was a GREAT mentor as well.

  • By scarydba, December 10, 2008 @ 9:57 am

    Sorry. Didn’t mean to impune (sp?) your friend. If he was a good mentor, then he’s clearly not a jerk. I don’t think you can do both at the same time.

Other Links to this Post

  1. SQL Server Central — December 11, 2008 @ 12:22 am

  2. SQL in the Wild » Blog Archive » SQL quiz 2 - Greatest challenges — December 18, 2008 @ 4:45 pm

RSS feed for comments on this post. TrackBack URI

Leave a comment