All About Statistics

RStudio Webinar with Hadley Wickham: The Grammar and Graphics of Data Science

Francis Smart — Thu, 17 Jul 2014 13:53:00 +0000

(This article was originally published at Econometrics by Simulation, and syndicated at StatsBlogs.)

RStudio has recently announced a series of free webinars open to the public. The first of these
seminars is given by Hadley Wickham, Rice University Professor, RStudio Chief Scientist, and general super-star of the R development world. Contributing author of the popular R packages ggplot2, plyr, testhat, reshape, and several of the other most innovative and widely used R packages currently available.

The first seminar is entitled The Grammar and Graphics of Data Science and I would imagine it involving much information with regards to ggplot2 as well as many other essential graphical tools in the R environment.

Upcoming seminars include one focused on "Reproducible Reporting" for which I believe will lead to a revolution in the Social Sciences in the next decade in terms of a tremendous increase in the quality and quantity of scientific work.

The third upcoming seminar is entitled "Interactive Reporting" and is focused on the development of tools to create dynamic reporting, in particular the popular package and server environment named "shiny".

To find a link to the registration site for these webinars go to:
pages.rstudio.net/Webniar-Series-Essential-Tools-for-R.html

Please comment on the article here: Econometrics by Simulation

The post RStudio Webinar with Hadley Wickham: The Grammar and Graphics of Data Science appeared first on All About Statistics.

Ethics and statistics

Andrew — Thu, 17 Jul 2014 13:36:47 +0000

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

I spoke (remotely) recently at the University of Wisconsin, on the topic of ethics and statistics. Afterward, I received the following question from Fabrizzio Sanchez:

As hard as it is to do, I thought it was good to try and define what exactly makes for an ethical violation. Your third point noted that it needed to break some sort of rule. Could you elaborate on this idea in the context of statistical rules? From my understanding, most statistical rules are not 0 or 1, but somewhere in between. (Removing an outlier comes to mind as an example).

He was responding to my statement that “An ethics problem arises when you are considering an action that (a) benefits you or some cause you support, (b) hurts or reduces benefits to others, and (c) violates some rule.”

I thought the bit about violating a rule was necessary because it’s generally considered acceptable to try to get more for yourself, if you’re doing so within the context of an accepted set of rules. Here I wasn’t thinking so much of statistical rules (for example, the idea that for statistical significance you need p=0.05 not p=0.06) but rather social rules. But maybe there’s more to be said on this.

The big new idea in my talk (which, unfortunately, I didn’t get to during the 20 minutes that were allocated to me) is near the end of the presentation, when I suggest that mainstream statistical methods (Bayes included) can themselves be unethical. Maybe this will the subject of a future Chance column.

P.S. One difficulty in posting slides is that they can be misleading without the accompanying speech. In particular, near the end of the slides I show the notorious third-degree polynomial regression discontinuity fit, under the headline, “Find the ethical problem!” Just to be clear, let me explain that I think the ethical problem here is not with the people who did the analysis and made the graph; rather, I think the ethical problem arises in our scientific publication system itself, which rewards dramatic claims based on statistical significance and dis-incentivizes more realistic, sober assessment of evidence. Also contributing to the ethical problem has been the publication of papers recommending something as goofy as this sort of high-degree polynomial fit.

The post Ethics and statistics appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Ethics and statistics appeared first on All About Statistics.

Why hadn’t I written a function for that?

Karl Broman — Thu, 17 Jul 2014 04:05:35 +0000

(This article was originally published at The stupidest thing... » R, and syndicated at StatsBlogs.)

I’m often typing the same bits of code over and over. Those bits of code really should be made into functions.

For example, I’m still using base graphics. (ggplot2 is on my “to do” list, really!) Often some things will be drawn with a slight overlap of the border of the plotting region. And in heatmaps with image, the border is often obscured. I want a nice black rectangle around the outside.

So I’ll write the following:

u <- par("usr")
rect(u[1], u[3], u[2], u[4])

I don’t know how many times I’ve typed that! Today I realized that I should put those two lines in a function add_border(). And then I added add_border() to my R/broman package.

It was a bit more work adding the Roxygen2 comments for the documentation, but now I’ve got a proper function that is easier to use and much more clear.

Update: @tpoi pointed out that box() does the same thing as my add_border(). My general point still stands, and this raises the additional point: twitter + blog → education.

I want to add, “I’m an idiot” but I think I’ll just say that there’s always more that I can learn about R. And I’ll remove add_border from R/broman and just use box().

Please comment on the article here: The stupidest thing... » R

The post Why hadn’t I written a function for that? appeared first on All About Statistics.

Riding With the Reverand

Dave Giles — Thu, 17 Jul 2014 04:00:00 +0000

(This article was originally published at Econometrics Beat: Dave Giles' Blog, and syndicated at StatsBlogs.)

Last week, outside Peter Jackson's ("Lord of the Rings", "The Hobbit") Weka Studios in Wellington, New Zealand, I happened to spy..........

All aboard! Let's get this bus on the road.

Please comment on the article here: Econometrics Beat: Dave Giles' Blog

The post Riding With the Reverand appeared first on All About Statistics.

Compile R and OpenBLAS from Source Guide

Lindons Log » Statistics — Wed, 16 Jul 2014 20:54:15 +0000

(This article was originally published at Lindons Log » Statistics, and syndicated at StatsBlogs.)

This guide is intended to aid any R and Linux user who desires a threaded version of BLAS. In particular I hope this will allow other grad students, who like me do not have many user privileges on their office computer, to follow suit and exploit multiple cores to speed up their linear algebra computations within R. The following will be performed on Scientific Linux 6.4 but has should be completely general. If you are a Ubuntu user, then there is an elegant and streamlined process for changing BLAS libraries and a recommended post about it here. I use Fedora on my laptop, and the following has also been tested thereupon.

My office computer has a quadcore processor with two threads per core but I also have access to a departmental computer with 4 sockets and 12 cores per socket (1 thread per core), so it really makes sense to use a threaded version of BLAS. If you are curious about the hardware on your own computer you can run the command “cat /proc/cpuinfo” or “lscpu”.

Unfortunately my office computer is part of a network upon which I do not have permissions to change ‘/usr/lib64/R/lib/libRblas.so’. Moreover R appears to be running serially: if you start up R and get the PID (process ID) from ‘top’ or ‘ps aux | grep R’ or something and then execute ‘cat /proc/PID/status | grep Threads’ you can see there is only one thread available.

[msl33@cabbage ~]$ cat /proc/13605/status | grep Threads
Threads:	1

(where 13605 was the process ID of my R process. That is using the default R on the network. One could appeal to the network administrator to change things for you but they probably won’t because a parallel BLAS implementation may cause problems for other users who require a serial BLAS, such as those that use the multicore environment to perform inherently parallel algorithms such as parallel tempering instead of using idle cores to speed up the linear algebra. There are also some known conflicts with the multicore package in R. There is, however, nothing stopping the user from compiling one’s own custom R build in one’s home directory and just changing the executable path thereto. In addition, you then have the power and freedom customize R to your needs – at the moment I have some very large matrices which would benefit from a threaded BLAS but at some point I may want to revert to a tuned serial BLAS such at ATLAS for certain parallel algorithms.

Firstly, go ahead and create a directory in which to keep all your custom software.

[msl33@cabbage ~]$ pwd
/home/grad/msl33
[msl33@cabbage ~]$ mkdir software

Download OpenBLAS

Make a directory “openblas” in the “software directory.

[msl33@cabbage ~]$ cd software/
[msl33@cabbage software]$ mkdir openblas

Next, grab the tarball from the OpenBLAS homepage. Change directory into where you downloaded the tarball and extract the files from it.

[msl33@cabbage ~]$ cd Downloads/
[msl33@cabbage Downloads]$ tar -xvf xianyi-OpenBLAS-v0.2.9-0-gf773f49.tar.gz

While this is running, fill a kettle with some water and turn it on, this stage is very important.

Change directory into where you extracted the files and run make.

[msl33@cabbage Downloads]$ cd xianyi-OpenBLAS-347dded/
[msl33@cabbage xianyi-OpenBLAS-347dded]$ make

Now is a good time to “make” some tea with the water prepared earlier. When done successfully one will see

Now, as instructed above, install to the “software” directory made earlier.

[msl33@cabbage xianyi-OpenBLAS-347dded]$ make PREFIX=/home/grad/msl33/software/openblas install
...
Install OK!

In openblas/lib there will be a file “libopenblas.so”, needed for later. That’s it for openblas, next we will do R.

Download R

Let’s create an R directory in software. Go onto the R homepage, then download, then choose a mirror and grab the tarball of the latest version. Download it to your “software” directory and extract it as before with “tar -xvf R-3.1.1.tar.gz”. Once extracted, remove the tarball and change directory into R-3.1.1. Before running the configure script one might bring some customizations into consideration in the name of efficiency. One might consider upping the optimization level from 2 to 3 and adding march or mtune by editing “config.site” and changing “## CFLAGS=” on line 53 to “CFLAGS=’-O3 -march=native’” and making similar changes for FFLAGS and CXXFLAGS. It is noted in the R Installation and Administration documentation that these can produce worthwhile speedups but come with a warning that the build will be less reliable, with segfaults and numerical errors creeping in. It is safest to leave things regular (reccommended link) but I’ll take the risk. Now run the configure script

[msl33@cabbage R-3.1.1]$ ./configure --prefix=/home/grad/msl33/software/R --enable-R-shlib --enable-BLAS-shlib --enable-memory-profiling --with-tcltk=no

################ADDITIONAL INSTRUCTIONS FOR DUKE STATISTICAL SCIENCE STUDENTS##################
On my office computer I needed in addition ‘–with-readline=no’ because the readline headers weren’t found (I’ll work on that but for now I’m not too bothered). In addition, the first time I tried to make on my office computer I encountered this error. “jni.h” could not be found. The error was resolved by locating it and then export the environment variable JAVA_HOME.

[msl33@cabbage software]$ locate jni.h
/usr/lib/jvm/java-1.7.0-sun-1.7.0.11/include/jni.h
[msl33@cabbage software]$ export JAVA_HOME=/usr/lib/jvm/java-1.7.0-sun-1.7.0.11/

#####################################################################################

Once the configure has completed, you’ll see a summary below like

You can execute “make install” to populate the software/R directory but you don’t need to. Change directories to lib and make a backup of libRblas.so and create a symbolic link to the openblas library that was made earlier.

[msl33@cabbage ~]$ cd software/R-3.1.1/lib
[msl33@cabbage lib]$ pwd
/home/grad/msl33/software/R-3.1.1/lib
[msl33@cabbage lib]$ mv libRblas.so libRblas.so.keep
[msl33@cabbage lib]$ ln -s /home/grad/msl33/software/openblas/lib/libopenblas.so libRblas.so

That was the last step. The R executable in the bin directory should now use openblas. Note this is the R executable you now need to run in order to use the custom built R with openblas. Just typing R in terminal will load the old /usr/lib64… which we students didn’t have the permissions to alter. You can, however, create an alias in your .bashrc file by inserting the line ‘alias R=”/home/grad/msl33/software/R-3.1.1/bin/./R”‘. Now when you type R in a terminal it will load the new R and not the old one. One can check that R executable depends on the correct linked shared blas library with the “ldd” command.

[msl33@cabbage bin]$ pwd
/home/grad/msl33/software/R-3.1.1/bin
[msl33@cabbage bin]$ ./R CMD ldd exec/./R | grep blas
	libRblas.so => /home/grad/msl33/software/R-3.1.1/lib/libRblas.so (0x00007f62e3fb7000)
[msl33@cabbage bin]$ ls -lt ../lib | grep openblas
lrwxrwxrwx  1 msl33 grad      53 Jul 16 15:35 libRblas.so -> /home/grad/msl33/software/openblas/lib/libopenblas.so

In addition, execute “./R” from the “bin” directory (or just R if you set up the alias) and grab the process id.

[msl33@cabbage bin]$ ps aux | grep R | grep software | awk '{print $2}'
2412
[msl33@cabbage bin]$ cat /proc/`ps aux | grep R | grep software | awk '{print $2}'`/status | grep Threads
Threads:	8
[msl33@cabbage bin]$

Evidently the R session now has 8 threads available. Finally, lets perform an eigen-decomposition and look at the cpu usage using top. You’ll see it light up all of your cores.

The post Compile R and OpenBLAS from Source Guide appeared first on Lindons Log.

Please comment on the article here: Lindons Log » Statistics

The post Compile R and OpenBLAS from Source Guide appeared first on All About Statistics.

The Econometrics of Temporal Aggregation – II – Causality Testing

Dave Giles — Wed, 16 Jul 2014 18:54:00 +0000

(This article was originally published at Econometrics Beat: Dave Giles' Blog, and syndicated at StatsBlogs.)

In my recent post about my presentation at the recent conference of the N.Z. Association of Economists, I promised some follow-up posts of a more specific nature. I thought I'd begin with some brief comments about the effects of temporal aggregation on Granger causality.

By temporal aggregation, I'm referring to the situation where the economic activity takes place at some frequency (say daily), but our time-series data are recorded at only a lower frequency (say monthly). My use of the word "aggregation' tells you that we're "adding up" the data over time - so, at least implicitly, I'm thinking about flow data.

How does this sort of aggregation impact on Granger causality?

A number of authors have addressed this question from different perspectives. Some useful references include the papers by Sims (1971), Wei (1982), Christiano & Eichenbaum (1987), Marcellino (1999), Breitung and Swanson (2002), and Gulasekaran and Abeysinghe (2002).

I'll use y_t and x_t to refer to the high-frequency data; and Y_t and X_t to refer to the (aggregated) low-frequency data. When I write x_t → y_t, this will mean "x Granger-causes y". Similarly, y_t ↔ x_t will denote bi-directional Granger causality between y and x. On the other hand, X_t ↛ Y_t will mean "X does not Granger-cause Y", etc.

First, let's look at a summary of a few situations that can arise when time-series data are temporally aggregated:

If x_t ↛ y_t, then X_t ↛ Y_t.
If x_t → y_tor y_t → x_t , then we can find X_t ↔ Y_t; or X_t ↛ Y_t; and / or Y_t ↛ X_t.
If y_t ↔ x_t, then we can find only Y_t → X_t, or vice versa.

Next, let's look a simple illustrative empirical example. It involves the use of daily data for the price of crude oil, P_C, (on the Edmonton hub), and the wholesale (rack) price, P_W, of gasoline in Vancouver. As well as using the daily data, I've also aggregated the numbers to get end-of-week prices, and end-of-month prices.The prices are stocks, not flows, but we'll see that this type of temporal aggregation can also impact on Granger causality testing.

The testing that I've undertaken allows for the unit roots and cointegration in the data by using the Toda-Yamamoto methodology. I've discussed the latter in detail in several earlier posts - most notably here. Here are the results, with "p" denoting the p-value, and "v" denoting the degrees of freedom, for the Wald test of the hypothesis of "no Granger causality":

As you can see, the upper part of the results table illustrates situation #2 noted above. The daily data indicate that there is Granger causality from crude oil prices to gasoline prices; whereas the aggregated data suggest the opposite (unlikely) result.

References

Breitung, J. and N. R. Swanson, 2002. Temporal aggregation and spurious instantaneous causality in multiple time series models. Journal of Time Series Analysis, 23, 651-665.

Christriano, L. J. and M. Eichenbaum, 1987. Temporal aggregation and structural inference in macroeconomics. Carnegie-Rochester Conference Series on Public Policy, 26, 63-130.

Gulasekaran, R. and T. Abeysinghe, 2002. The distortionary effects of temporal aggregation on Granger causality. Working Paper No. 0204, Department of Economics, National University of Singapore.

Marcellino, M., 1999. Some consequences of temporal aggregation in empirical analysis. Journal of Business and Economic Statistics, 17, 129-136.

Sims, C. A., 1971. Discrete approximations to continuous time distributed lags in econometrics. Econometrica, 39, 545-563.

Toda, H. Y. and T. Yamamoto, 1995. Statistical inferences in vector autoregressions with possibly integrated processes. Journal of Econometrics, 66, 225-250

Wei, W. W. S., 1982. The effect of systematic sampling and temporal aggregation on causality – A cautionary note. Journal of the American Statistical Association, 77, 316-319.

Please comment on the article here: Econometrics Beat: Dave Giles' Blog

The post The Econometrics of Temporal Aggregation – II – Causality Testing appeared first on All About Statistics.

Comparing localsolver with Rglpk on k-medoids example

Bogumił Kamiński — Wed, 16 Jul 2014 18:13:00 +0000

(This article was originally published at R snippets, and syndicated at StatsBlogs.)

Recently I have coauthored a new localsolver package that can be used to solve large scale optimization problems from R. It is a wrapper around commercial solver that is free for academia. If you are interested why it is worthwhile to give it a look - read on.

There is a host of packages on optimization CRAN Task View so why bother with a new one? There are two reasons: (1) flexibility and speed of LocalSolver optimization engine and (2) ease of use - which is often a problem with traditional packages like Rglpk - especially for large scale problems.

Today I will introduce the package and focus on ease of use. Next time I will provide some performance comparisons to existing solvers.

Before we begin

First you need to setup your environment. There are two steps.

You should start with downloading LocalSlover and installing it.

Next simply install localsolver package (if you have problems with installation on Mac use type="source" option of install.pacakges should solve them).

In case LocalSolver is not in search path of your system after installation either add it or remember its path as in such cases you can pass it to computational routines.

Now we are ready to go.

The test problem

As this is an example of ease of use we will focus on an elementary integer programming task. Suppose that we have 20 locations. Out of them we have to select 5 depots and we want to assign every location to nearest depot. We know distances between all stores (distance matrix does not have to be symmetric). Our objective is to minimize sum of all distances between depots and locations assigned to them. We can see that the problem is simply a k-medoids clustering task but without assumption of symmetry of distance matrix.

Let us start with code that generates test matrices for our examples:

kmedoids.generate.data <- function(seed, N, P) {
if(!is.null(seed)) {
set.seed(seed)
}
InfiniteDist <- 10000.0
Dist <- matrix(round(runif(n=N * N) * 100, digits = 0),
nrow = N)
diag(Dist) <- 0

return(list(
N = as.integer(N),
P = as.integer(P),
Dist = Dist,
InfiniteDist = InfiniteDist))
}

data <- kmedoids.generate.data(seed = 1234, N = 20L, P = 5L)

One thing that is worth explaining is that in the code we define InfiniteDist which will serve us as surrogate for infinity (all actual distances are shorter than it).

Now let me present how the problem can be solved with localsolver and Rglpk.

localsolver approach

Here is the code:

library(localsolver)

lsp.model <- "function model() {
x[1..N] <- bool() ; // point i is in P iff x[i] = 1

constraint sum[i in 1..N](x[i]) == P ;

minDist[i in 1..N] <- min[j in 1..N](
x[j] ? Dist[j][i] : InfiniteDist);
// minimize sum of distances
objective <- sum[i in 1..N]( minDist[i] ) ;
minimize objective;
}"

lsp <- ls.problem(model.text.lsp = lsp.model)
lsp <- set.params(lsp = lsp, lsTimeLimit = 1)
lsp <- set.params(lsp = lsp, lsNbThreads = 1)
lsp <- add.output.expr(lsp=lsp, expr.text.lsp = "x",
dimensions = data$N)
lsp <- add.output.expr(lsp=lsp, expr.text.lsp = "objective",
dimensions = 1)
lsp.solution <- ls.solve(lsp = lsp, data = data)

Notice that we can define the optimization problem using a simple domain specific language as a string. Interestingly in the string we can use all variables defined in data list as localsolver is able to interact with extrernal data source. Additionally using set.params and add.output.expr we can set up the parameters of optimization engine and optimization output we will want to collect.
As a result we get a list with optimization output. Importantly the contents of the list can be controlled programatically as mentioned above.

Rglpk approach

Now let us solve the same problem using Rglpk:

library(Rglpk)

objective <- c(as.numeric(data$Dist), rep(0, data$N))

types <- c(rep("C", length(data$Dist)),

rep("B", data$N))

# Exactly P centroids

# sum[i in 1..N] y[i] == P

mat <- matrix(data = c(rep(0, length(data$Dist)),

rep(1, data$N)), nrow=1)

rhs <- c(data$P)

dir <- c("==")

# Each datapoint only to one centroid

# x[i,j] - ith datapoint to jth centroid

# For each j in 1..N sum[i in 1..N] x[i,j] == 1

for(j in 1:data$N) {

mat <- rbind(mat,

rep(0, (j-1)*data$N),

rep(1, data$N),

rep(0, ncol(mat) - j*data$N)))

}

rhs <- c(rhs, rep(1, data$N))

dir <- c(dir, rep("==", data$N))

# Client cannot be assigned to notselected centroid

# x[i,j] <= y[j]

# x[i,j] - y[j] <= 0

for(i in 1:data$N) {

for(j in 1:data$N) {

row <- rep(0, ncol(mat))

row[(i-1)*data$N + j] <- 1

row[length(data$Dist) + j] <- -1

mat <- rbind(mat, row)

}

rhs <- c(rhs, rep(0, data$N*data$N))

dir <- c(dir, rep("<=", data$N*data$N))

glpk.sol <- Rglpk_solve_LP(

obj = objective,

mat = mat,

dir = dir,

rhs = rhs,

types = types,

max = FALSE,

control = list(verbose = TRUE,

canonicalize_status = TRUE))

In my opinion the code looks cryptic in comparison to localsolver specification. There are two reasons for such a situation:

we have to have two mix two types of decision variables: location of depots and assignment of locations to depots
we have to construct a complex constraints matrix combining different types of conditions in one big data structure

In fact even in order to directly learn which locations are selected as depots we have to dig into Rglpk output with tail(glpk.sol$solution, n = data$N).

Summary

We have tested localsolver on linear integer programming problem. I wanted to show that the package allows the user to solve optimization tasks:

using natural (mathematical) formulation of objective function and constraints
allowing to dynamically update reference data: that lsp.model string did not have any hard coded constants - everything is loaded from metadata provided as R list.

In my opinion it is not only more elastic in comparison to Rglpk solution presented above but also more robust. Coding Rglpk solution took much more time as it required much more thought in the design of Rglpk_solve_LP input variables and a lot of corrections of errors in the code.

What is more localsolver is not only more user friendly but is also not limited to mixed integer linear optimization - it can solve highly complex non-linear tasks as well.

Now a natural question arises: what is the performance of localsolver in terms of speed and the quality of obtained solution. I will tackle them in my next post.

Please comment on the article here: R snippets

The post Comparing localsolver with Rglpk on k-medoids example appeared first on All About Statistics.

Jan de Leeuw owns the Internet

Roger Peng — Wed, 16 Jul 2014 15:22:51 +0000

(This article was originally published at Simply Statistics, and syndicated at StatsBlogs.)

One of the best things to happen on the Internet recently is that Jan de Leeuw has decided to own the Twitter/Facebook universe. If you do not already, you should be following him. Among his many accomplishments, he founded the Department of Statistics at UCLA (my alma mater), which is currently thriving. On the occasion of the Department's 10th birthday, there was a small celebration, and I recall Don Ylvisaker mentioning that the reason they invited Jan to UCLA way back when was because he "knew everyone and knew everything". Pretty accurate description, in my opinion.

Jan's been tweeting quite a bit of late, but recently had this gem:

As long as statistics continues to emphasize assumptions, models, and inference it will remain a minor subfield of data science.

— Jan de Leeuw (@deleeuw_jan) July 15, 2014

followed by

@j8feng @rdpeng Statistics is the applied science that constructs and studies techniques for data analysis.

— Jan de Leeuw (@deleeuw_jan) July 15, 2014

I'm not sure what Jan's thinking behind the first tweet was, but I think many in statistics would consider it a "good thing" to be a minor subfield of data science. Why get involved in that messy thing called data science where people are going wild with data in an unprincipled manner?

This is a situation where I think there is a large disconnect between what "should be" and what "is reality". What should be is that statistics should include the field of data science. Honestly, that would be beneficial to the field of statistics and would allow us to provide a home to many people who don't necessarily have one (primarily, people working not he border between two fields). Nate Silver made reference to this in his keynote address to the Joint Statistical Meetings last year when he said data science was just a fancy term for statistics.

The reality though is the opposite. Statistics has chosen to limit itself to a few areas, such as inference, as Jan mentions, and to willfully ignore other important aspects of data science as "not statistics". This is unfortunate, I think, because unlike many in the field of statistics, I believe data science is here to stay. The reason is because statistics has decided not to fill the spaces that have been created by the increasing complexity of modern data analysis. The needs of modern data analyses (reproducibility, computing on large datasets, data preprocessing/cleaning) didn't fall into the usual statistics curriculum, and so they were ignored. In my view, data science is about stringing together many different tools for many different purposes into an analytic whole. Traditional statistical modeling is a part of this (often a small part), but statistical thinking plays a role in all of it.

Statisticians should take on the challenge of data science and own it. We may not be successful in doing so, but we certainly won't be if we don't try.

Please comment on the article here: Simply Statistics

The post Jan de Leeuw owns the Internet appeared first on All About Statistics.

“The Europeans and Australians were too eager to believe in renal denervation”

Andrew — Wed, 16 Jul 2014 13:21:30 +0000

(This article was originally published at Statistical Modeling, Causal Inference, and Social Science, and syndicated at StatsBlogs.)

As you can see, I’m having a competition with myself for the most boring title ever.

The story, though, is not boring. Paul Alper writes:

I just came across this in the NYT.

Here is the NEJM article itself:

And here is the editorial in the NEJM:

The gist is that on the basis of previous studies without a control arm, renal denervation was thought to be a blockbuster treatment for those suffering from very high blood pressure. The randomized clinical trial with a sham procedure as the control (placebo) found that the effect seems to be mainly psychological. I suppose the moral of the story is that unless there is a control arm, enthusiasm must be tempered. Note too that the U.S. FDA comes out appearing properly slow and skeptical while the Europeans and Australians were too eager to believe in renal denervation.

Paul followed up with this:

Our local newspaper today had an AP article about drugs which lower cholesterol but speculated that despite the lower cholesterol, the lowering of mortality by the drugs are yet to be proved.

To see whether you have already blogged about “surrogate criteria,” I googled *surrogate Gelman* and came up with this and this, which contains this sentence:

The objective of NRL’s research is to develop tissue surrogate materials that simulate the mechanical and acoustical properties of biological tissues. These are then assembled into an experimental test system of the human thorax, called “GelMan,” for assessing blunt forces and blast dynamics.

Hey, I used to work at NRL!

Paul continued:

While I still don’t know what a “GelMan” is, I also found this by a Gelman which has to do with vomiting.

I assume he was referring to my encounter with the Barfblog.

Paul continued:

Note the reference to a “Fisher,” but not R.A.

Just for giggles, I googled *surrogate Alper* and sure enough found many web sites for conceiving a new life and dealing with the end of life by people who have Alper as a last name. One wonders if *surrogate any-name-what-so-ever* would also produce a bunch of google hits.

Hmmm, let’s see what comes up:

Cool—it works!

P.S. I would’ve given this post the title, “GelMan Thoracic Surrogate for Underwater Threat Neutralization,” but that wouldn’t have been boring enough.

The post “The Europeans and Australians were too eager to believe in renal denervation” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post “The Europeans and Australians were too eager to believe in renal denervation” appeared first on All About Statistics.

The SAS/IML File Exchange is open

Rick Wicklin — Wed, 16 Jul 2014 09:20:47 +0000

(This article was originally published at The DO Loop, and syndicated at StatsBlogs.)

Have you written a SAS/IML program that you think is particularly clever? Are you the proud author of SAS/IML functions that extend the functionality of SAS software?

You've worked hard to develop, debug, and test your program, so why not share it with others? There is now a central location for SAS/IML programmers to upload data, programs, and documentation. It is called the SAS/IML File Exchange. The File Exchange is a Web site where authors can upload programs that they have written and where SAS/IML programmers can search for useful programs.

SAS programmers have a long history of sharing programs:

Papers in the proceedings of SUGI or SAS Global Forum conferences often include a complete program.
Each book from SAS Press has a Web page where you can download the programs and data from the book.
The sasCommunity.org site has a user-created "sasopedia" that contains hundreds of tips, articles, and programs that were written by experts.
Beyond the SAS language, there are many other active communities that exchange programs. For example, the JMP user community has contributed a large collection of JMP programs, scripts, and data. There are also active communities for R and MATLAB users.

The File Exchange is a subcommunity of the popular SAS/IML Support Community, where programmers discuss algorithms, ask questions, and get help writing, debugging, and improving their programs. In 2013, there were more than 500 posts to the SAS/IML Support Community.

The new SAS/IML File Exchange provides a much-needed repository for ready-to-use programs so that programmers can spend less time reinventing the wheel and more time writing amazing programs. To participate in the File Exchange, do the following:

Search your files for that awesome SAS/IML program that you wrote last year. Add comments to the program so that others can appreciate how clever you are! :-)
Read the article "How to Contribute to the SAS/IML File Exchange," which describes how to upload files to the File Exchange.
If you do not yet have a SAS profile, go to the SAS Support Communities and establish a profile. If you already have a SAS profile, log in by using your email address.
Go to the SAS/IML File Exchange and upload your files individually or as a ZIP file.

The community of SAS/IML programmers has been growing rapidly. Because the free SAS University Edition includes the SAS/IML product, I expect the community to grow even faster in the future. I hope that programmers of all skill levels will use the SAS/IML File Exchange to post programs that everyone can use. Why not post your program today?

tags: File Exchange, Statistical Programming

Please comment on the article here: The DO Loop

The post The SAS/IML File Exchange is open appeared first on All About Statistics.