Learning R: Hard Lessons

Home / Data Science / Learning R: Hard Lessons

I’ve always found the best way to learn a new programming language is to start building stuff, solving problems, using the language. Even if you do things badly or inefficiently, you’re figuring out how to put the language to use. I tried the same thing with R.

Fair warning, there is no happy conclusion to this story. It’s a lesson about learning, not about solving a problem.

After poking at the R language for a little while, I decided I was ready to solve a problem. I have a fantastic idea for demonstrating the usefulness of the language specifically for DBAs. I won’t go into what it is here because I’m still hoping to solve this problem and it will provide a fantastic blog post. Anyway, I have a very good understanding of the problem (or so I thought) I’m looking to solve, so, what the heck, let’s throw R at it.

First thing I learned was how to connect R up to a SQL Server database. Easy. Then it’s query the database to create a data frame. Easy. Referencing columns and rows from the data frame, again, really simple. Spit out some interesting information about the data such as density or a histogram? A couple of Bingle searches later, easy. Here’s a starter (formatted as text, my code formatter in the blog doesn’t have an R language choice):

Seriously, this really is an easy enough language to learn. It’s kind of quirky, but it’s easy to understand as long as you are quick to look up syntax(it really is quirky). So what’s the problem?

Algorithms.

But that’s not even true. For example, I can pretty quickly run a Chi Square Independence test against my data like this:

Then you get the output and all hell breaks loose…. or… more accurately, no hell breaks loose. You’re looking at some data and trying to figure out what it means. Suffice to say, I finally realized that I had the wrong algorithm. More web searches. Questions posted to various forums, defining my problem (as I saw the definition), which lead me to the fact that what I really needed to solve my problem was not to run standard comparison tests, but to compare continuous distributions. Awesome. Fixed that algorithm problem, right?

Wrong.

That’s because algorithms are not the problem… the only problem. The real problem is data preparation. A lot of the examples you’ll read online are very straight forward with nice neat data sets. That’s because they were carefully groomed and prepared. Here I am looking at the wooly wild real data and I’m utterly lost in how to properly prepare this so that it’s appropriately set up as a continuous distribution(or a distribution at all). WOOF! The reason this is so hard is because I actually don’t understand the data fundamentals of the problem I’m trying to solve in exactly the way needed to solve the problem. More cogitation is necessary.

My lesson. I’m going to continue learning R syntax. I’m absolutely going to keep studying the mathematics in order to better understand the algorithms. But I’ve got to also spend time figuring out how to groom the data. It’s pretty clear that is one of the biggest challenges to making this stuff work.

Oh, and make sure you close your connection to the database:

8 Comments

  • LORD! That sounds like lots of problems in the Excel world. Pivot Tables are really easy … but not if the source data isn’t ready.

    Thanks for this post and pointing out that dirty side of working with data: the *doing* may not be so hard but the prep work can be the killer.

  • if your going to learn any R, you really need a book. and the reason is that you read the book with one eye, and type in RStudio with the other. interactive training/instruction with R just doesn’t work so well. the reason for that is, R is a language from another brother. semantics and syntax are unhinged from the ALGOL/C lineage, so nothing that “should work” does.

    I recommend Crawley’s “The R Book” for basics (it’s quite large because it covers a lot of ground) and Wickham’s “ggplot2” for graphics.

    as to the major benefit of R/RDBMS, running R inside the engine, SS’s approach isn’t as useful as Postgres or Oracle or SAP. learn Postgres too, if that’s where you see yourself going.

  • Thanks for the suggestions Robert. The lessons I was getting were pretty good in terms of typing out the language, but I was considering books to pick up.

    I don’t think I’ll be going the Postgres route any time soon. I’m staying mainly inside the MS playground for now. Lots of moves into all the Data Platform offerings though. SQL Server is becoming an “also” instead of an “only.”

  • Larry

    Right there with ya’. I’ll second Robert’s motion that some kind of interactive training with R is pretty much required unless you’ve already got a solid background in statistical analysis (and maybe even then).

    I took statistical courses on Coursera which utilized R and it helped tremendously. I’m sure there are other online learning sites that have similar courses. One particular course had optional lessons which demonstrated how the instructors used R to produce the material for the lessons. Great insight on not just the language but how to use it to accomplish specific tasks and analyses.

OK, fine, but what do you think?