Bad SQL habits

August 5, 2014
By

For those in Boston/Cambridge, I will be speaking at the Chief Data Scientist meetup on Wednesday night. See you there. *** Warning: this post may be hard to understand if you don't know SQL. SQL is one of the most fundamental tools in data science. It is used to manipulate data. Its simplicity is a big reason for its popularity. There are lots of things it can’t do but the…

Read more »

Stigler’s seven pillars of statistical wisdom

August 5, 2014
By
Stigler’s seven pillars of statistical wisdom

Wisdom has built her house; She has hewn out her seven pillars.      – Proverbs 9:1 At the 2014 Joint Statistical Meetings in Boston, Stephen Stigler gave the ASA President's Invited Address. In forty short minutes, Stigler laid out his response to the age-old question "What is statistics?" His answer was […]

Read more »

Thanks to R Markdown: Perhaps Word is an option after all?

August 5, 2014
By
Thanks to R Markdown: Perhaps Word is an option after all?

In many cases Word is still the preferred file format for collaboration in the office. Yet, it is often a challenge to work with it, not so much because of the software, but how it is used and abused. Thanks to Markdown it is no longer painful to inclu...

Read more »

Neyman, Power, and Severity

August 5, 2014
By
Neyman, Power, and Severity

Jerzy Neyman: April 16, 1894-August 5, 1981. This reblogs posts under “The Will to Understand Power” & “Neyman’s Nursery” here & here. Way back when, although I’d never met him, I sent my doctoral dissertation, Philosophy of Statistics, to one person only: Professor Ronald Giere. (And he would read it, too!) I knew from his publications that he […]

Read more »

Estimation & Accuracy After Model Selection

August 4, 2014
By
Estimation & Accuracy After Model Selection

This was the title of Brad Efron's invited paper at the 2014 Joint Statistical Meetings in Boston this morning. It was a great presentation, with excellent discussants - Lan Wang, Lawrence Brown, and Soumendra Lahiri.The paper and discussion are s...

Read more »

Video Tutorial – Calculating Expected Counts in a Contingency Table Using Joint Probabilities

Video Tutorial – Calculating Expected Counts in a Contingency Table Using Joint Probabilities

In an earlier video, I showed how to calculate expected counts in a contingency table using marginal proportions and totals.  (Recall that expected counts are needed to conduct hypothesis tests of independence between categorical random variables.)  Today, I want to share a second video of calculating expected counts – this time, using joint probabilities.  This method uses […]

Read more »

Correlation does not even imply correlation

August 4, 2014
By

The above title is my response to a discussion that began with this email sent to be by Steve Roth: Noah Smith had a great tweet recently, a real keeper for me [Roth]. Causation is correlated with correlation. I would reword it: Correlation correlates with causation. (Just not very much.) And I wonder if the […] The post Correlation does not even imply correlation appeared first on Statistical Modeling, Causal…

Read more »

On deck this week

August 4, 2014
By

Mon: Correlation does not even imply correlation Tues: When doing scientific replication or criticism, collaboration with the original authors is fine but I don’t think it should be a requirement or even an expectation Wed: Scientific communication by press release Thurs: Nate Silver’s website Fri: Estimated effect of early childhood intervention downgraded from 42% to […] The post On deck this week appeared first on Statistical Modeling, Causal Inference, and…

Read more »

One guaranteed to make Stephen Few cry

August 4, 2014
By
One guaranteed to make Stephen Few cry

Vox published this chart: This sort of chart is, unfortunately, quite common in business circles. Just about the only thing one can read readily from this chart is the overall growth in the plug-in vehicle market (the heights of the...

Read more »

The Black Swan Spectrum

August 4, 2014
By

Speaking of the newly-updated draft of Econometrics, now for some fun. Here's a question from the Chapter 6 EPC (exercises, problems and complements). Where does your reaction fall on the A-B spectrum below?Nassim Taleb is a financial markets trader (a...

Read more »

Reversing the limits of integration in SAS

August 4, 2014
By
Reversing the limits of integration in SAS

In SAS software, you can use the QUAD subroutine in the SAS/IML language to evaluate definite integrals on an interval [a, b]. The integral is properly defined only for a < b, but mathematicians define the following convention, which enables you to make sense of reversing the limits of integration: […]

Read more »

Men set to live as long as women by 2030?

August 3, 2014
By
Men set to live as long as women by 2030?

A few months ago, in Men set to live as long as women, figures show, it was mentioned that (in the U.K.) the gap between male and female life expectancy is closing and men could catch up by 2030, according to an adviser for the Office for National Statistics. (the slides are available online http://cass.city.ac.uk/…). I don’t really know U.K. demography, but I was surprised to read such an optimistic conclusion. So…

Read more »

Guns are Cool – Differences between states

August 3, 2014
By
Guns are Cool – Differences between states

Last week my blog showed that there are differences between states in the shootingtracker database. This week it is attempted to understand why states are different. A number of variables were extracted from a few sources, among which gun laws, % ...

Read more »

qdap 2.1.1 Released

August 3, 2014
By
qdap 2.1.1 Released

We’re very pleased to announce the release of qdap 2.1.1 What is qdap? qdap (Quantitative Discourse Analysis Package) is an R package designed to assist in quantitative discourse analysis. The package stands as a bridge between qualitative transcripts of dialogue … Continue reading →

Read more »

Correlation and Causation

August 3, 2014
By
Correlation and Causation

A hat-tip to Judea Pearl, whose e-newsletter alerted me to this interesting post on the EvaluationHelp blog. It shows the original sixteen diagrams in Sewall Wright's classic 1921 paper on correlation and causality.Philip and Sewall Wright were re...

Read more »

Customer Segmentation Using Purchase History: Another Example of Matrix Factorization

August 2, 2014
By
Customer Segmentation Using Purchase History: Another Example of Matrix Factorization

As promised in my last post, I am following up with another example of how to perform market segmentations with nonnegative matrix factorization. Included with the R package bayesm is a dataset called Scotch containing the purchase history for 21 brand...

Read more »

The odds of a cluster of airplane accidents

August 2, 2014
By
The odds of a cluster of airplane accidents

Recently, there have been a lot of airplane accidents. July, 17th 2014, Hrabove, Ukraine, Malaysia Airlines, Boeing 777, fatalities 298 (/298) July, 23rd 2014, Magong, Taiwan, TransAsia Airways, ATR 72-500, fatalities 47 (/58) July, 24th 2014, Aguelhok, Mali, Air Algerie, Mc Donnell Douglas MD-83, fatalities 116 (/116) It is simple to find a lot of datasets about airplane crashes. For instance on http://ntsb.gov/aviationquery. The dataset is nice, with a lot…

Read more »

Testing an R package’s interactive graphs

August 2, 2014
By
Testing an R package’s interactive graphs

I’ve been working on an R package, R/qtlcharts, with D3-based interactive graphs for quantitative trait locus mapping experiments. Testing the interactive charts it produces is a bit of a pain. It seems like I pretty much have to just open a series of examples in a web browser and tab through them manually, checking that […]

Read more »

The “scientific surprise” two-step

August 1, 2014
By

During the past year or so, we’ve been discussing a bunch of “Psychological Science”-style papers in which dramatic claims are made based on somewhat open-ended analysis of small samples with noisy measurements. One thing that comes up in some of these discussions is that the people performing the studies say that they did not fish […] The post The “scientific surprise” two-step appeared first on Statistical Modeling, Causal Inference, and…

Read more »

Adjusted Momentum

August 1, 2014
By
Adjusted Momentum

David Varadi has published two excellent posts / ideas about cooking with momentum: VIX-Adjusted Momentum Error-Adjusted Momentum I just could not resist the urge to share these ideas with you. Following is implementation using the Systematic Investor Toolbox. Please enjoy and share your ideas with David and myself. To view the complete source code for […]

Read more »

Stanny Stanny Stannitude

July 31, 2014
By
Stanny Stanny Stannitude

On the stan-users list, Richard McElreath reports: With 2.4 out, I ran a quick test of how much speedup I could get by changing my old non-vectorized multi_normal sampling to the new vectorized form. I get a 40% time savings, without even trying hard. This is much better than I expected. Timings with vectorized multi_normal: […] The post Stanny Stanny Stannitude appeared first on Statistical Modeling, Causal Inference, and Social…

Read more »

Roger Berger on Stephen Senn’s “Blood Simple” with a response by Senn (Guest posts)

July 31, 2014
By
Roger Berger on Stephen Senn’s “Blood Simple” with a response by Senn (Guest posts)

Roger L. Berger School Director & Professor School of Mathematical & Natural Science Arizona State University Comment on S. Senn’s post: “Blood Simple? The complicated and controversial world of bioequivalence”(*) First, I do agree with Senn’s statement that “the FDA requires conventional placebo-controlled trials of a new treatment to be tested at the 5% level two-sided but […]

Read more »

Open Econometrics Text Updated for Fall Use

July 31, 2014
By

I have just posted an update of my introductory undergraduate Econometrics (book, slides, R code, EViews code, data, etc.). Warning: although it is significantly improved, it nevertheless remains highly (alas, woefully) preliminary and incomp...

Read more »


Subscribe

Email:

  Subscribe