I have lists. Lots of lists. I even have them in different locations sometimes. Some of them are carefully written down in my notebook, others are typed into OneNote and I’ve been experimenting with Remember the Milk and Trello (Trello is winning). These lists include ideas for presentations, blogs, articles. Notes from sessions I’ve attended or meetings. Lots and lots of plans and ideas and all that sort of stuff. I try to keep it organized, but sometimes it runs away from me. However, I find writing things down helps me to keep things organized. Between very carefully scheduling out my calendar and all these notes, I only occasionally completely drop the ball.

One ball I dropped was coming up with some goals, some plans, for 2012. I just plowed through 2011 and then 2012 and here I am, two years an employee with Red Gate (and I’m sorry to say, terribly happy there, my apologies) and I’m not sure where I should be going. I don’t mean within the company. We’ve got lots going on there (more than I can keep up with). No, I mean career goals, work outside work, all the extra stuff.

I’m going to break these down into pure technical issues that I want to spend more time on and professional development. Let’s start with the easy stuff, technical.

First, HDInsight. I absolutely believe there’s something big possible with this technology. I could be utterly wrong (I sure thought SCOM had legs too, oops). But I’m going to keep slogging on this stuff til I get something of a more thorough understanding of it. I doubt I’ll be presenting on it except maybe for introductory sessions, and not even those for the next 3-6 months. But I’m going to get my mind wrapped around it and understand how it works. Maybe then I can better determine if it’s just a highly specialized impact wrench or a true paradigm shift.

Next up, more Azure. Go ahead, giggle. Microsoft keeps doing more and more stuff with it. You know you can store spatial data in Azure now? I didn’t either until earlier this week. I’m going to be presenting on how to do query tuning using Azure SQL Database tools. Actually, I’m thinking I’ll only use the online tools. Everything you need is there. You can access most of the Dynamic Management Objects (DMO) and you can read execution plans from queries there. I’ve got presentations already scheduled for this. I’m pretty excited about others that might be scheduled (LOTS of noise if those come to pass). I’m going to continue expanding my learning here.

As to professional/personal development, I’ve got something somewhat trivial I want to work on, and something that, frankly, scares me. I would also like to find a coach or coaches.

The trivial task is that I want to work on pumping up my slide decks. I’ve long simply used them as markers for my talks. Bullet points as reminders of the topics I want to make sure I cover. They somewhat act as a point for taking notes for the attendees, but they are excessively dull. I’m going to work with Mrs. Scary (a graphic artist, good at her job, freelance if you need some help and are willing to pay for it, get in touch) and punch up my presentations. I had about 12 or so different presentations last year. This year, I’m dropping that number down a little.

Scary. Really scary. No, not me. Me. I mean I’m scared, not scary. I’m scared because I’ve taken on the task of acting as a mentor to an actual human being. I don’t know about the rest of you, but I find that extremely scary. I’ve got a level of responsibility that I’ve never had with putting up a blog post or even making a presentation. It’s as bad as writing a book (and if you think putting your name on a book isn’t scary, wait til you see your first bad review). But, I’m excited by it. This is something new and different. I’m really looking forward to it, and I think the person who approached me has a lot of potential. They’ll be great without my ever getting involved, so if I can take some of the credit, I win. Sorry, joke. I’m going to try to help someone who doesn’t need it, but will benefit from the help. Assuming I get it right… Scared again.

Coaches. Am I doing the marketing bits of my job right? I don’t know. I need some help in that area. Am I approaching learning HDInsight correctly? I could use a coach to talk with and get some guidance from. Maybe some suggestions on my slides (although my wife is going to supply a lot of that). Pro athletes do it. Good surgeon’s do it. They must know something. It’s an approach I want to try.

That’s it. I’ll be continuing things like working on books (yes, I’ve committed to writing one and editing one, so far), presentations, light consulting, and working for a fantastic organization. But these are the current plans for 2013.

By default statistics are created automatically within SQL Server. And, by default, these stats are updated automatically based on a set of triggers. The triggers are defined as

• If 0 rows – Any data added leads to a statistics update
• If < 500 rows – 500 rows added causes a stats update
• If > 500 rows – 500 rows + 20% of the number of rows causes a stats update (unless you enable a traceflag in 2012 in which case you get a proportional value instead of 20%).

There are some exceptions for temporary tables and some variations for filtered statistics and filtered indexes, but you get the idea. I was writing an article on statistics in preparation for another Oracle/SQL Server discussion (on, you guessed it, statistics) and I came to these rules and wrote something along the lines of “So even if you have 499 rows, in order to see the stats update you would need to have to add more than 500 additional rows, meaning you would get to 999 before you saw a statistics update.” Which made sense, but I was suddenly concerned, is that valid. I decided to clarify it for my own piece of mind.

Quick set up of a test table and loading 499 rows into followed. I also ran a simple query to generate some statistics on the column:

CREATE TABLE dbo.StatsTest (StatsColumn INT);

WITH Nums
  AS (SELECT TOP (499)
ROW_NUMBER() OVER (ORDER BY (SELECT 1
)) AS n
  FROM master.sys.all_columns AS ac
CROSS JOIN master.sys.all_columns AS ac2
)
INSERT INTO dbo.StatsTest
(StatsColumn)
SELECT n
FROM Nums;

--auto create stats
SELECT *
FROM dbo.StatsTest AS st
WHERE StatsColumn = 42;

This results in the follow set of statistics:

If you click on it to look at the details you can see that it shows 499 rows sampled.

If I add two more rows to the statistics and then run a different query, guaranteed to require a statistics update:

SELECT	*
FROM	dbo.StatsTest AS st
WHERE	StatsColumn < 42;

If I rerun DBCC SHOW_STATISTICS I still see the same output. In short, it appears that my original assumption was correct.

I ran several more tests, and I could get to 999 without updating the statistics, but adding just one more row and breaking 500 additional rows and the statistics update fired. I needed to validate this behavior for myself and it worked.

If you’re over in the UK, or close to it, you’ve probably heard of SQL Bits even if you haven’t attended. If you’re across the pond here in the US or somewhere else you may have heard of SQL Bits, but few of you have gone. Well, let me tell you, from everything I’ve heard, this is a community event you want to find your way to. So, I’m trying. This is my first time submitting to this outstanding event and I really want to go. But, in order to make the cut, I need your help. They base who presents on community feedback. This means you go over and vote. You’ll have to have a SQL Bits login, but if you’re even thinking about attending you’re going to have one of those anyway. I’ve submitted two sessions:

Database Backups for the Accidental DBA

and

Query Tuning in the Clouds

Read through those abstracts. If you think they’re worthy, please go here and vote for them. You’ll also find there are a huge number of sessions being potentially offered by some of the best speakers in the business. Check it out and vote on your favorites.

Just a quick little blog post about one more app you’re going to want to track down for your Surface. Great Windows Apps.

Why do you need this? Well, how to put this, the Store, looks nice. But it doesn’t work terribly well. Trying to see the top rated apps, or looking for new apps, you can get different results at different times and I find it quite frustrating. But, Great Windows Apps is running some sort of queries or something against the Store and displaying the information in a much more meaningful, and quite accurate, representation of the apps. It’s just good.

After living with the Surface for a month and trying hard to make it work as a tool for production, not just consumption, I feel like I’ve got a handle on this thing. There are several that are built in that I’m using all the time like Word, Mail, Internet Explorer, OneNote, Music and Video. But there are a few others that you need to know about if you’re looking to get more functionality out of your Surface. One note, I tried to find links to the Windows Store for all the apps, but I can’t seem to do that, or at least I couldn’t find how to do it. Here’s the list:

Tweetro: This is the single best Twitter app that I’ve tried so far. It functions as I would expect a decent Twitter app to function, and it works with the Windows 8 interface nicely. It looks good, performs well, hasn’t crashed on me yet. I’m quite happy. Unfortunately for most everyone else, you won’t be able to get this app for free. They’re one of the first apps to hit Twitter’s new policy that roughly reads “you can’t make better apps than we do.” But, they have an app you can purchase, and it will be absolutely worth it. I’m digging the free version, but I think I’ll upgrade.

Evernote: You are either already using Evernote, or you will be shortly. The main strength of the app is, well, it’s Evernote. And it’s on your Surface. But more than that, the interface is very much a Windows 8 interface. The notes are laid out in a great way, showing you when you edited them, the title and the initial text in a set of tiles. Clicking on a tile opens up the note. From there you can edit the note. It just works and works nicely.

Kindle: Duh. I still carry an actual Kindle, but you really should have the app anyway. This one isn’t all special Windows 8′sy at all. It’s just a nice layout of the books you have available from Amazon and the books you have on your desktop. It’s actually nice to see the book covers, and in color, since I’m used to only ever seeing their titles these days. Not much to really say about this app, but you need it.

Skype: You must own this one if, like me, you travel and at least some of that travel takes you overseas. I want to be able to talk to my family. Luckily, with a WIFI connection, I can, and see them too. The Skype app works wonderfully on the Surface (and since Microsoft now owns Skype, I’d be horrified if it were otherwise). It works nicely with the camera. I’ll also be using it occasionally when I need to talk to my Lords & Masters back at Red Gate. I like the way the app works. You can switch between the two cameras really easily by clicking on the preview window low on the screen. Other than that, it’s Skype. It works.

WODTimer: OK, maybe not for everyone, but when I’m on the road I still need to work out. I’ve been using this Workout of the Day timer. It lets you do countdowns, times, and tabata’s. Plus it does rounds in between. Everything I want or need.

Remote Desktop: Yes, it’s built-in, but believe it or not, you have to download it and install it. Make sure you do. I have hit some snags with the Windows 8 version as opposed to the desktop version, but it’s still one of those apps that makes the whole thing work.

Burn After Reading: I’m just getting going using Pocket for managing reading web pages later  and tracking them, but, after trying a couple of apps on the Surface, this one wins, hands down. Yeah, you’ll have to pay for it. But, that gets you a very nice looking app with a full blown Windows 8 interface. The tiles are graphics from the web sites you have marked. You’ll also see the title and a piece of text. Best of all, when you’re in the browser, you can use the Share charm to mark pages from there within the Surface that will then be saved out to your Pocket list. Exactly what I would expect.

That’s it. And if that list seems a little short, that’s because, we really still don’t have a great selection of apps on the Surface. But, it’s growing and the thing is turning into a real, functional tool. Slowly though. Ever so slowly.

After multiple talks with Jason Strate (blog|twitter) and Denny Cherry (blog|twitter) about how they are working on automating more of their blogging and social media interactions, I’m going down that road… a little ways.

The first thing that I’m trying out is Pocket. It’s a way to capture web sites that you want to read later. You don’t want to have to try to manage book marks in your browser. They all suck (and Google Chrome on the Mac sucks even worse) in this task. So this app lets you quickly mark a site and tag it for reading later. Further, you have an account that you can access from multiple devices. I’m already trying out two different readers that will work with Pocket on the Surface and I already have one for the phone. (yes, still using the Surface, more blog posts coming).

This then leads to IFTTT, or If This Then That. It’s a way to automate some of the interactions of things you do on the internet. I’m just getting started. I just set up my first trigger to post any articles I’ve read from Pocket to my Facebook account with a link. It was insanely, stupidly simple. Oh boy! More stuff to come as soon as I can find more things to do with this. Oooh, there are suggested triggers, such as RSS feeds to Pocket. Oh yeah, this is dangerous, slick and cool. Stand back.

I’m also looking at HootSuite. I’ve already been using automating a Facebook post and a Tweet when I post to my blog. But, it’s just the one time. You know how any one Tweet can get lost in your Twitter stream. So, maybe a few (a very few, I refuse to become a spam engine) extra tweets after a blog post would be useful. The basic setup there is quite simple, but management looks like a bear. I’ll have to see if IFTTT can help there. Ah, there it is, a nice easy way to link an RSS feed to accounts. Can’t do all accounts at once without moving to the paid service. I’ll have to wait to see if I need that.

All experiments are starting out on Facebook, primarily because I care the least about my Facebook account. I do care about G+ and Twitter. These seem to have a lot more professional interactions.

As I see more possibilities here, I’ll share them.

See this:

That’s right. The install worked. All I had to do was get a completely clean server set up. No domain controller. Now to get my learn on.

Microsoft has a web site with a number of introductory samples. I’ll start there and work through them. The very first example gets me set up with some data that it builds by running a Powershell script, importdata.ps1. But I’m not going to just blindly follow along. I want to see what the heck is happening so I can start understanding this stuff. By the way, thank you Microsoft for making the samples in PowerShell and not forcing me to relearn Python or something else. That would have been frustrating.

The script is really simple. It has two scenarios you can pass it, w3c or RunTraffic. They just change directory and run another PowerShell script, import.ps1, from two different directories. I’ll be the scripts are different. I’m running the w3c scenario, so let’s see what that script is doing.

Ah, now things are getting interesting. There are two functions, one for data generation which uses an executable to make up test data. The other a mechanism for calling Hadoop. Basically it uses two objects, System.Diagnostics.ProcessStartInfo and System.Diagnostics.Process. The ProcessStartInfo is for defining startup information for a process that you then define using the Process command. In this case it’s setting the location of hadoop:

$pinfo.FileName = $env:HADOOP_HOME + "\bin\hadoop.cmd";

Then it sets up arguments, if any. The actual calls to this from the code use a command, dfs, which has different settings -mkdir and -copyFromLocal. From what I can tell, it’s creating a storage location within Hadoop and then moving the data generated over. I’m good with all the scripts I can see except knowing where this dfs thing comes from.

Data load ran just fine:

Data loaded, time to test out a Map/Reduce job. Again there’s a powershell script included for running a simple job, so I check it out. First run, fails. Great. More stuff to try to troubleshoot in order to be able to see this work. This is not going to be easy.

Stepping through and running the scripts might not be the best way to learn this. So, I’m going to now start reading the Big Data Jumpstart Guide. I’ll post more as I learn it.

Just a few blog posts that you ought to go and read.

First up, Tom LaRock maintains a listing of SQL bloggers split up into various cleverly named groups to show you where to go to get good information. This really is an excellent collection of bloggers. It’s the people I go to when I need information. Some of them are better resources than the Books Online when they post something. Personally, I’ve made the list for the last several years, but Tom has decided that I’m worth of elevation, so I’ve gone from the Model database to the Master database. Thanks Tom. One blog that’s not on Tom’s list is Tom’s blog. You should be reading that regularly too. And congratulations to Tom again on making MCM.

Next, one of the bloggers on Tom’s list, and a friend, is Aaron Bertrand. Aaron has posted pretty much everything you need to know about how to get the most out of DBCC, not at his blog. I was considering a blog post on this myself but after reading this, why bother. He has it covered, up to and including linking over to Paul Randal’s advice on how to break up your DBCC checks (and Paul would know since he wrote the silly thing). Seriously, go and read Aaron’s blog post right now. It’s must reading for all the DBAs who are not Paul.

And I thought this would be so easy.

Latest attempt. I just did an uninstall of HDInsight. Then I found all the Hadoop folders on the drive and removed them. I validated that there were no services left running or anything else that might interfere with a fresh install. Then, I rebooted the server (have you tried turning it off and on again?). After all that, I tried rerunning the HDInsight Community Preview. This time I ran it as an administrator on the system, even though I already am logged in as one. Paranoia at every step is my new approach since there have been no errors until I try to turn the silly thing on.

And the install went perfectly. I can see two folders, Hadoop and HadoopFeaturePackSetup. Last time there was a third folder HadoopInstall or something like that. This doesn’t bode well. Let me check the services. Nope. I can’t see anything added. Crud. So, I try the powershell command anyway, just to see what I’ll get. It’s a new error, which is nice:

Join-Path : Cannot bind argument to parameter ‘Path’ because it is null.
At C:\hadoop\start-onebox.ps1:2 char:23
+ $start_cmd = Join-Path <<<< $env:ISOTOPEJS_HOME “bin/start_daemons.cmd”;
+ CategoryInfo : InvalidData: (:) [Join-Path], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.JoinPathCom
mand

Crud. My assumption is, I’m doing something wrong. I just can’t figure out what. Lots of searching later, I think I may have found the issue. I’ve been trying to install this on my local server that I use for lots of my tests because it’s a physical box that I can access on the road. It’s also a domain controller. I finally tracked down this thread on Microsoft’s forums. Great. OK. Next step, setting up a virtual server. May as well make it Windows 2012 while I’m at it. More to come when I get past this hurdle. Oy.

I installed HDInsight with no errors. The install is so brain-dead, I couldn’t see how there could be errors. But, I decided to fire up the service to start playing with it. That’s when I hit the errors. First up, I was getting logon errors for the Hadoop service itself. I tried resolving them manually, but was digging a hole. So, I uninstalled, and reinstalled, figuring I had missed an error message somewhere along the line. After this install, I didn’t get logon errors. Instead I got internal errors. Some more investigation and it appears that the service was marked for deletion. The uninstall worked, but somehow wasn’t complete. Yay! So, I did another uninstall after rebooting the server. Install #4 coming up.  And we’re back to logon errors. Specifically:

Starting IsotopeJS services
Starting isotopejs
System error 1069 has occurred.

The service did not start due to a logon failure.

Let’s check the logs, if any. Nothing in the c:\Hadoop directory. Checking the system logs to see if there’s anything there. Nothing I can see. No joy. I’ll go through it all again tonight.