Finding observations that satisfy multiple conditions: The LOC-ELEMENT technique

May 11, 2015
By
Finding observations that satisfy multiple conditions: The LOC-ELEMENT technique

A common task in data analysis is to locate observations that satisfy multiple criteria. For example, you might want to locate all zip codes in certain counties within specified states. The SAS DATA step contains the powerful WHERE statement, which enables you to extract a subset of data that satisfy […] The post Finding observations that satisfy multiple conditions: The LOC-ELEMENT technique appeared first on The DO Loop.

Read more »

The Economist gets in on the AI Fluff

May 11, 2015
By
The Economist gets in on the AI Fluff

The Economist leads with an editorial and an article on The Dawn of Artificial Intelligence. The editorial starts of with: “THE development of full artificial intelligence could spell the end of the human race,” Stephen Hawking warns. Elon Musk fears...

Read more »

Fast SQL moving average calculation without windowing functions

May 11, 2015
By
Fast SQL moving average calculation without windowing functions

In this post, I show a trick to do moving average calculation (can be extended to other operations requiring windowing functions) that is super fast. Often, SAS analysts need to conduct moving average calculation and there are several options by the or...

Read more »

arbitrary distributions with set correlation

May 10, 2015
By
arbitrary distributions with set correlation

A question recently posted on X Validated by Antoni Parrelada: given two arbitrary cdfs F and G, how can we simulate a pair (X,Y) with marginals  F and G, and with set correlation ρ? The answer posted by Antoni Parrelada was to reproduce the Gaussian copula solution: produce (X’,Y’) as a Gaussian bivariate vector with […]

Read more »

JPMorgan, Data-Rich Analyses, and the Public Good

May 10, 2015
By

I recently received an invitation to the JPMorgan Chase event below.Reaction 1: JPMC should stick to its business, which is business, working to maximize the shareholder wealth with which it is entrusted, leaving to others (like me) the "provision of d...

Read more »

Collaborative filtering, hierarchical modeling, and . . . speed dating

May 10, 2015
By
Collaborative filtering, hierarchical modeling, and . . . speed dating

Jonah Sinick posted a few things on the famous speed-dating dataset and writes: The main element that I seem to have been missing is principal component analysis of the different rating types. The basic situation is that the first PC is something that people are roughly equally responsive to, while people vary a lot with […] The post Collaborative filtering, hierarchical modeling, and . . . speed dating appeared first…

Read more »

Visualizing statistical distributions with javascript

May 10, 2015
By
Visualizing statistical distributions with javascript

For the past few years, I've been developing and using a library I created that allows me to easily generate visualizations of statistical distributions for teaching. One can specify a distribution along with a parametrization, and the library sees it ...

Read more »

U-boats in WW-II

May 10, 2015
By
U-boats in WW-II

This is the time when we celebrate the end of the second world war in The Netherlands, so I thought to do somwthig with data from that era. One of the things I enjoyed were books on the sea warfare, such as 'The Cruel Sea'by Nicholas Monsarrat. In that...

Read more »

Stephen Senn: Double Jeopardy?: Judge Jeffreys Upholds the Law (sequel to the pathetic P-value)

May 9, 2015
By
Stephen Senn: Double Jeopardy?: Judge Jeffreys Upholds the Law (sequel to the pathetic P-value)

Stephen Senn Head of Competence Center for Methodology and Statistics (CCMS) Luxembourg Institute of Health Double Jeopardy?: Judge Jeffreys Upholds the Law “But this could be dealt with in a rough empirical way by taking twice the standard error as a criterion for possible genuineness and three times the standard error for definite acceptance”. Harold […]

Read more »

AI, Artificial Birds and Aeroplanes

May 9, 2015
By
AI, Artificial Birds and Aeroplanes

The Turing Test for artificial intelligence is a reasonably well understood idea: if, through a written form of communication, a machine can convince a human that it too is a human, then it passes the test. The elegance of this...

Read more »

Social networks spread disease—but they also spread practices that reduce disease

May 9, 2015
By
Social networks spread disease—but they also spread practices that reduce disease

I recently posted on the sister blog regarding a paper by Jon Zelner, James Trostle, Jason Goldstick, William Cevallos, James House, and Joseph Eisenberg, “Social Connectedness and Disease Transmission: Social Organization, Cohesion, Village Context, and Infection Risk in Rural Ecuador.” Zelner follows up: This made me think of my favorite figure from this paper, which […] The post Social networks spread disease—but they also spread practices that reduce disease appeared…

Read more »

Vienna Workshop on High-Dimensional Time Series In Macroeconomics and Finance

May 8, 2015
By

Program looking good:  https://www.conftool.net/timeseries2015/sessions.php.  Presumably papers will be posted, or at least you can email the authors.

Read more »

Statistics: P values are just the tip of the iceberg : Nature News & Comment

May 8, 2015
By
Statistics: P values are just the tip of the iceberg : Nature News & Comment

Statistics: P values are just the tip of the iceberg : Nature News & Comment: This article is very important. Yes, p-values reported in the literature (or in your own research) need scrutiny, but so does every step in the analysis process, starting...

Read more »

Yeah ………… That’d Be Great

May 8, 2015
By
Yeah ………… That’d Be Great

"Bill Lumbergh" will continue to terrorize the office via my tweets at @DEAGiles:p.s.: Actually, I do own a "Swingline" stapler, and I'm thinking of re-spraying it fire-engine red.© 2015, David E. Giles

Read more »

What Can We Learn from the Apps on Your Smartphone? Topic Modeling and Matrix Factorization

May 8, 2015
By
What Can We Learn from the Apps on Your Smartphone? Topic Modeling and Matrix Factorization

The website for The Burning House begins with a simple question:If your house was burning, what would you take with you? It's a conflict between what's practical, valuable and sentimental. What you would take reflects your interests, background and pri...

Read more »

The tyranny of the idea in science

May 8, 2015
By

There are a lot of analogies between startups and academic science labs. One thing that is definitely very different is the relative value of ideas in the startup world and in the academic world. For example, Paul Graham has said: Actually, startup ideas are not million dollar ideas, and here's an experiment you can try

Read more »

What I got wrong (and right) about econometrics and unbiasedness

May 8, 2015
By

Yesterday I spoke at the Princeton economics department. The title of my talk was: “Unbiasedness”: You keep using that word. I do not think it means what you think it means. The talk went all right—people seemed ok with what I was saying—but I didn’t see a lot of audience involvement. It was a bit […] The post What I got wrong (and right) about econometrics and unbiasedness appeared first…

Read more »

Updated DBDA2E programs for number of MCMC chains and parallel chains in runjags

May 7, 2015
By
Updated DBDA2E programs for number of MCMC chains and parallel chains in runjags

The DBDA2E programs have been updated so they deal better with parallel chains in runjags and the number of cores available on your computer. The new programs are available as this zip folder also linked at the book's software page. There are 29 modifi...

Read more »

Books to Read While the Algae Grow in Your Fur, March 2015

May 7, 2015
By

Attention conservation notice: I have no taste. Anthony Shadid, House of Stone: A Memoir of Home, Family, and a Lost Middle East Shadid's memoir of restoring his family's ancestral home in a small Christian town in south Lebanon, inter-cut with the ...

Read more »

On the Invariance of MLE’s

May 7, 2015
By
On the Invariance of MLE’s

The Maximum Likelihood Estimator (MLE) is extremely widely used in statistics, and in the various "metrics" disciplines such as econometrics. This is because this estimator has several highly desirable properties, as long as the sample size is sufficie...

Read more »

What is new in the vtreat library?

May 7, 2015
By

The Win-Vector LLC vtreat library is a library we supply (under a GPL license) for automating the simple domain independent part of variable cleaning an preparation. The idea is you supply (in R) an example general data.frame to vtreat’s designTr...

Read more »

Deflategate 3: nature of evidence

May 7, 2015
By
Deflategate 3: nature of evidence

Last time we heard about Deflategate on this blog, Warren Sharp compiled some statistics on fumble rates, showing that the Patriots were unusually good at avoiding fumbles. (link, link) I thought the level of analysis was "above average" and remarked that statistical evidence of this type can only get you so far. The metric is indirect, and it does not speak to causation. The official investigators have now issued their…

Read more »

Mendelian randomization inspires a randomized trial design for multiple drugs simultaneously

May 7, 2015
By
Mendelian randomization inspires a randomized trial design for multiple drugs simultaneously

Joe Pickrell has an interesting new paper out about Mendelian randomization. He discusses some of the interesting issues that come up with these studies and performs a mini-review of previously published studies using the technique. The basic idea behind Mendelian Randomization is the following. In a simple, randomly mating population Mendel's laws tell us that at any

Read more »


Subscribe

Email:

  Subscribe