An introductory comparison of using the two languages.

Background

R was made especially for data analysis and graphics.  SQL was made especially for databases.  They are allies.

The data structure in R that most closely matches a SQL table is a data frame.  The terms rows and columns are used in both.

A mashup

There is an R package called sqldf that allows you to use SQL commands to extract data from an R data frame.  We will use this package in the examples. There are two basic steps to using an R package:

• it must be installed on your machine (once)
• it must be available in each R session where it is used

You can use sqldf by doing (one time only, and assuming an internet connection):

install.packages("sqldf")

Then in each R session where you want to use it:

require(sqldf)

To simplify the examples, we’ll slightly modify one of the inbuilt data frames:

myCO2 <- CO2
attributes(myCO2) <- attributes(CO2)[
  c("names", "row.names", "class")]
class(myCO2) <- "data.frame"

Note that the character between C and 2 is a capital-O and not a zero.  The CO2 object has a complicated value for its class but the result of the sqldf function has only "data.frame" in its class.  We want to cleanly see if two objects are the same, and hence we want the classes to match.

Column names

In R the colnames function returns the names of the columns:

> colnames(myCO2)
[1] "Plant"     "Type"      "Treatment" "conc" 
[5] "uptake"

The result is a vector of character strings.

Subsetting columns

Columns in SQL are also called “fields”.  In R it is rather common for columns to be called “variables”.

In SQL the subset of columns is determined by select.  Here we want to get the Type and conc columns:

s01 <- sqldf("select Type, conc from myCO2")

Subsetting in R (commonly called “subscripting”) is done with square brackets.  When subscripting a data frame there will be two places inside the square brackets separated by a comma.  The R equivalent of the command above is:

r01 <- myCO2[, c("Type", "conc")]

The first part inside the square brackets (corresponding to rows) is empty.  The second part (corresponding to columns) has a character vector with the names of the two columns we want.

We can test that r01 and s01 are the same:

> all.equal(s01, r01)
[1] TRUE

In R the vector of column names could be created as an object and then used in the subscripting:

someCols <- c("Type", "conc")
r01b <- myCO2[, someCols]

The r01 and r01b objects are the same.

All columns

An asterisk is used in SQL to indicate that you want all columns:

s02 <- sqldf("select * from myCO2")

When you want all items in a dimension in R, you leave it blank:

r02 <- myCO2[ , ]

You might have been able to guess that because we’ve seen that done for rows already.  Note that spaces almost never matter in R — the command above has spaces either side of the comma, but would be exactly the same with no spaces.

Only one column

How to select only a single column is no surprise in either language:

s03 <- sqldf("select Type from myCO2")
r03 <- myCO2[ , "Type"]

But there is a surprise when you test if these two objects are equal:

all.equal(s03, r03)

The command above results in a bunch of stuff, indicating they are quite different.

The r03 object is not a data frame, it is an object of the type of the column.  While surprising to those used to SQL, this is quite natural for R’s purposes.  For example, we give the mean function a vector of numbers, not a data frame:

> mean(myCO2[, "uptake"])
[1] 27.2131

You can get a one-column data frame by slightly modifying the command:

r03d <- myCO2[ , "Type", drop=FALSE]

The s03 and r03d objects are the same.

Data frames are not natural inputs to some functions:

> mean(myCO2[, "uptake", drop=FALSE])
[1] NA
Warning message:
In mean.default(myCO2[, "uptake", drop = FALSE]) :
 argument is not numeric or logical: returning NA

Case sensitivity

SQL is not case-sensitive:

s04 <- sqldf("select type, coNC from myCO2")

s04 is the same as s01.

On the other hand, R is case-sensitive:

> r04 <- myCO2[, c("type", "coNC")]
Error in `[.data.frame`(myCO2, , c("type", "coNC")) : 
  undefined columns selected

R extensions

We've seen how to select columns of an R data frame with the names of the columns.  There are other ways of selecting columns as well.

The order of the columns in an R data frame is of significance.  You can select columns by number.  For example, you can select column 5 and then column 2:

myCO2[, c(5, 2)]

You can use negative numbers to exclude columns.  Here you are asking for all columns except the first and the fourth:

myCO2[, c(-1, -4)]

Column selection in R can also be done with logical values:

myCO2[, c(TRUE, FALSE, FALSE, TRUE, FALSE)]

Those logical values can be created by a command:

myCO2[, colnames(myCO2) > "d"]

Subsetting rows

In SQL a common synonym for “row” is “record”.  In R a common synonym is “observation”.

Conditions

The common way of getting a subset of rows in SQL is with the where command:

s05 <- sqldf("select * from myCO2 where uptake < 20")

In R the equivalent of the where is put in the first position inside the square brackets:

r05 <- myCO2[ myCO2[, "uptake"] < 20, ]

s05 and r05 are in most respects the same.  The difference is that the row names are different.  r05 has the row names from the original data frame while s05 has new ones that are sequential from 1.

with

The command that created r05 is a little convoluted (but logical once you stare at it long enough).  The with function allows a command that is more in the spirit of what is done in SQL:

r05w <- with(myCO2, myCO2[uptake < 20, ]) # same as r05

Inside the with call the columns of the data frame named in the first argument can be used as objects.  In this example uptake is used directly instead of pulling that column out of the data frame.

Logical operators

Logical comparisons in SQL are combined with AND and OR:

s06 <- sqldf("select * from myCO2 where uptake < 20 and Type='Quebec'")

Also note that testing equality is with =.

In R this type of ‘and’ operation is done with & and the ‘or’ is |:

r06 <- with(myCO2, myCO2[uptake < 20 & Type == 'Quebec', ])

A possible trouble spot is that equality in R is tested with == (while = is an assignment operator).

The s06 and r06 objects are the same except for their row names.

First few

The limit command in SQL limits the number of rows that are given:

s07 <- sqldf("select * from myC02 limit 6")

One way to see just the column names is to limit the number of rows to zero.

You can get the first few rows in R with head:

r07 <- head(myCO2)

The tail function gives you the last few rows, and the corner function is a logical extension of head and tail.

Row names versus numbers

A source of possible confusion is that row names are character even though they are, by default, representations of numbers.  Let’s experiment with r06:

> r06
   Plant   Type  Treatment conc uptake
1    Qn1 Quebec nonchilled   95   16.0
8    Qn2 Quebec nonchilled   95   13.6
15   Qn3 Quebec nonchilled   95   16.2
22   Qc1 Quebec    chilled   95   14.2
29   Qc2 Quebec    chilled   95    9.3
36   Qc3 Quebec    chilled   95   15.1

Select the first three rows:

> r06[1:3,]
   Plant   Type  Treatment conc uptake
1    Qn1 Quebec nonchilled   95   16.0
8    Qn2 Quebec nonchilled   95   13.6
15   Qn3 Quebec nonchilled   95   16.2

Now let’s select the characters one through three:

> r06[c("1", "2", "3"), ]
   Plant   Type  Treatment conc uptake
1    Qn1 Quebec nonchilled   95   16.0
NA  <NA>   <NA>       <NA>   NA     NA
36   Qc3 Quebec    chilled   95   15.1

What happened?  The first row is correct — the first row name is 1.  In the second row it looked for a row name called “2″ and didn’t find one, so it put in missing values.  The third row is even weirder: it looked for a row name called “3″; there was a single row name starting with “3″ so it did a partial match and gave us that row.

Trying to give numbers instead of the actual names doesn’t necessarily work either:

> r06[c(1, 8, 15), ]
     Plant   Type  Treatment conc uptake
1      Qn1 Quebec nonchilled   95     16
NA    <NA>   <NA>       <NA>   NA     NA
NA.1  <NA>   <NA>       <NA>   NA     NA

Additional details

NULL

In SQL NULL means missing value.  Confusingly R also has NULL but the equivalent of SQL NULL is NA in R.

Let’s create some data to play with:

r08 <- r06
r08[2:4, 1] <- NA
r08[5, 4] <- NA

This looks like:

> r08
   Plant   Type  Treatment conc uptake
1    Qn1 Quebec nonchilled   95   16.0
8   <NA> Quebec nonchilled   95   13.6
15  <NA> Quebec nonchilled   95   16.2
22  <NA> Quebec    chilled   95   14.2
29   Qc2 Quebec    chilled   NA    9.3
36   Qc3 Quebec    chilled   95   15.1

Get the rows where Plant is not missing:

s09 <- sqldf("select * from r08 where plant is not null")
r09 <- with(r08, r08[!is.na(Plant), ])

We can also get the rows where Plant is missing:

s10 <- sqldf("select * from r08 where plant is null")
r10 <- with(r08, r08[is.na(Plant), ])

To get the rows that have no missing values in R, you can do:

> na.omit(r08)
   Plant   Type  Treatment conc uptake
1    Qn1 Quebec nonchilled   95   16.0
36   Qc3 Quebec    chilled   95   15.1

Quotes

In SQL single quotes are used to delimit character strings.  A single quote inside a string is given with two single quotes in a row.  Some implementations allow you to specify the delimiter.

In R either single quotes or double quotes can be used.  You can use whichever you find more convenient but R always prints using double quotes.  The backslash is used to escape a quote character that is the same as the delimiting quote:

> c("he's", 'he\'s', "she has \"it\"")
[1] "he's"           "he's"           "she has \"it\""

Semicolons

Semicolons are sometimes used at the end of statements in both SQL and R.

Some SQL implementations require a semicolon at the end of a statement.

Semicolons are used to separate R commands on the same line.  They can be used after all R commands, but probably shouldn’t be.

Single subscript

Note that both

r06[1:3, ]

and

r06[1:3]

are legal R commands, but they do different things.  The first gives the first three rows and all of the columns; the second gives all of the rows and the first three columns (for a reason you need not be concerned about initially).

Efficiency

If a data frame is large and the manipulation is complex, then R can be inefficient.  Why should someone with access to a database put up with such inefficiency?  One reason is the flexibility that R gives you.  It might surprise some people that not all data naturally fit into a structure of rows and columns.  Besides it usually doesn’t matter.  As has been asked facetiously by a certain someone: “What are you going to do with that extra millisecond?”

But if there are millions of those milliseconds, then you might start to care.  The data.table package provides an alternative form of data frames that is highly efficient.

Resources to learn R

“Impatient R” is a minimal set of things to learn about R.

“Some hints for the R beginner” suggests additional resources.

Resources to learn SQL

I’m certainly no expert at learning materials for SQL — please make suggestions.  But here are things I’ve found that seem at least okay:

SQLZoo provides quite a nice interactive set of exercises.

Tutorialspoint has information easily arranged for learning and refreshing.

See also

“R database interfaces”

“Select operations on R data frames”

The post Translating between R and SQL: the basics appeared first on Burns Statistics.

Please comment on the article here: Burns Statistics

The post Translating between R and SQL: the basics appeared first on All About Statistics.

One great thing about working in statistics and political science is, between them, these two subjects are connected to just about everything.

From the day’s news (sort of):

Pat Robertson Thinks Low-Carb Diets Violate God’s Principles: I wonder what Art De Vany will think of this. I had the impression that lo-carb is vaguely connected to conservative politics, that whole paleo-caveman-thing. I’d think that if Pat were a bit more plugged into the zeitgeist, he’d be pushing lo-carb as a poke in the eye at the liberal health establishment. But maybe I’m the one who’s out of touch, as can be seen from these links:

A Week Into the Shutdown, Government Buys $47,174 Mechanical Bull: They should see some of the things we pay for from our NSF grant. Remember that setup we bought at the state fair where people could throw a ball at a target and try to send Mister P into the water tank?

Nielsen’s New Twitter TV Ratings Are a Total Scam. Here’s Why.: This one I can definitely relate to. Social media has become a bit of a fad in quantitative social science, and for good reason (new data capturing aspects of people’s lives that are not always accessible via survey interviewing), but can also get oversold. So I’m happy to see some pushback on this in the popular media. Next step, perhaps: a counterintuitive story in Slate explaining why Twitter ratings are not all crap. . . .

Tech’s Gender Problem in One Chart: Uh oh. This looks a lot like the gender mix of the Stan team. And, no, we’re not planning to put Chelsea Clinton on our board.

Robo-Vaping Is the Sad Future of Smoking: I’m seeing these on the street all the time.

The Daily Show Definitely Showed an Uncensored Penis [NSFW]: No comment.

The post A day with the news! appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post A day with the news! appeared first on All About Statistics.

I came across a new source of data which I think is really worth sharing: ThinkNum. It gathers around 2,000 sources of data but more importantly it allows the user to manipulate this data via functions and graphics and there is an R package available on CRAN. Interested readers can find a very good post exploring some of the functionalities here.

Data Description

• Yahoo: Free stock quotes, up to date news, portfolio management resources, international market data, message boards, and mortgage rates that help you manage your financial life
• FRED: Download, graph, and track 149,000 economic time series from 59 sources
• Oanda: Currency information, tools, and resources for investors, businesses, and travelers
• Google: Stock market quotes, news, currency conversions & more
• Quandl: Futures prices, daily. Quandl is a search engine for numerical data^. The site offers access to several million financial, economic and social datasets
• TrueFX: Tick-By-Tick Real-Time And Historical Market Rates, Clean, Aggregated, Dealer Prices
• Bloomberg: Financial news, business news, economic news, stock quotes, markets quotes, finance stocks, financial markets, stock futures, personal finance, personal finance advice, mutual funds, financial calculators, world business, small business, financial trends, forex trading, technology news, bloomberg financial news
• Interactive Broker: Interactive Brokers Group, Inc. is an online discount brokerage firm in the United States
• Datastream: Datastream Professional is a powerful tool that integrates economic research and strategy with cross asset analysis to seamlessly bring together top down and bottom up in one single, integrated application
• pwt: The Penn World Table provides purchasing power parity and national income accounts converted to international prices for 189 countries/territories for some or all of the years 1950-2010
• Thinknum: Thinknum brings financial data from a variety of useful sources together on one platform. We use this data to develop applications

Package Detail

• Quantmod: Specify, build, trade, and analyse quantitative financial trading strategies
• Quandl: This package interacts directly with the Quandl API to offer data in a number of formats usable in R, as well as the ability to upload and search
• TFX: Connects R to TrueFX(tm) for free streaming real-time and historical tick-by-tick market data for dealable interbank foreign exchange rates with millisecond detail
• Rbbg: Handles fetching data from the Bloomberg financial data application
• IBrokers: Provides native R access to Interactive Brokers Trader Workstation API
• rdatastream: RDatastream is a R interface to the Thomson Dataworks Entreprise SOAP API (non free), with some convenience functions for retrieving Datastream data specifically. This package requires valid credentials for this API
• pwt: The Penn World Table provides purchasing power parity and national income accounts converted to international prices for 189 countries/territories for some or all of the years 1950-2010
• fImport: Rmetrics is the premier open source software solution for teaching and training quantitative finance. fImport is the package for Economic and Financial Data Import
• Thinknum: This package interacts directly with the Thinknum API to offer data in a number of formats usable in R

Please comment on the article here: The R Trader

The post Financial Data Accessible from R – part III appeared first on All About Statistics.

Please comment on the article here: Gianluca Baio's blog

The post Rescue remedy appeared first on All About Statistics.

Please comment on the article here: Adventures in Analytics and Visualization

The post Replication of few graphs/charts in base R, ggplot2, and rCharts appeared first on All About Statistics.

Please comment on the article here: No Hesitations

The post The e-Writing Jungle (Part I): LaTeX and MathJax and MathML and LaTeXtoHTML5 and Python and Sphinx and iPython and R and Knitter and Firefox and Chrome and … appeared first on All About Statistics.

After seeing a document sent to me and others regarding the crisis of spurious, statistically-significant research findings in psychology research, I had the following reaction:

I am unhappy with the use in the document of the phrase “false positives.” I feel that this expression is unhelpful as it frames science in terms of “true” and “false” claims, which I don’t think is particularly accurate. In particular, in most of the recent disputed Psych Science type studies (the ESP study excepted, perhaps), there is little doubt that there is _some_ underlying effect. The issue, as I see it, as that the underlying effects are much smaller, and much more variable, than mainstream researchers imagine. So what happens is that Psych Science or Nature or whatever will publish a result that is purported to be some sort of universal truth, but it is actually a pattern specific to one data set, one population, and one experimental condition. In a sense, yes, these journals are publishing false claims. But it’s not clear to me that the framing in terms of “false positive” etc. is the way to go. It’s not like I think the true effects are zero, I just think there’s not much of a relationship between what is found in a particular study, and more general phenomena of interest.

I can understand if my preferred framing in terms of Type M and Type S errors is too new and unfamiliar to use here, but I do think that a key part of this effort should be to move away from the whole Type 1, Type 2, false positive, false negative, etc. framework.

The post I’m negative on the expression “false positives” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post I’m negative on the expression “false positives” appeared first on All About Statistics.

After seeing a document sent to me and others regarding the crisis of spurious, statistically-significant research findings in psychology research, I had the following reaction:

I am unhappy with the use in the document of the phrase “false positives.” I feel that this expression is unhelpful as it frames science in terms of “true” and “false” claims, which I don’t think is particularly accurate. In particular, in most of the recent disputed Psych Science type studies (the ESP study excepted, perhaps), there is little doubt that there is _some_ underlying effect. The issue, as I see it, as that the underlying effects are much smaller, and much more variable, than mainstream researchers imagine. So what happens is that Psych Science or Nature or whatever will publish a result that is purported to be some sort of universal truth, but it is actually a pattern specific to one data set, one population, and one experimental condition. In a sense, yes, these journals are publishing false claims. But it’s not clear to me that the framing in terms of “false positive” etc. is the way to go. It’s not like I think the true effects are zero, I just think there’s not much of a relationship between what is found in a particular study, and more general phenomena of interest.

I can understand if my preferred framing in terms of Type M and Type S errors is too new and unfamiliar to use here, but I do think that a key part of this effort should be to move away from the whole Type 1, Type 2, false positive, false negative, etc. framework.

The post Nix on the expression “false positives” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Nix on the expression “false positives” appeared first on All About Statistics.

Jeremy Fox points us to this compilation of data visualizations in R that went wrong, in a way that ended up making them look like art.

They are indeed wonderful.

The post Data visualizations gone beautifully wrong appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Data visualizations gone beautifully wrong appeared first on All About Statistics.

Please comment on the article here: Junk Charts

The post Light entertainment: when up is down (double bill) appeared first on All About Statistics.