Below is a plot of per capita GPD (in log scale) against years since women received the right to vote for 42 countries. Is this cause, effect, both or neither? We all know correlation does not imply causation, but I see many (non statistical) arguments to support both cause and effect here. Happy International Women's Day ! 

The data is from here and here. I removed countries where women have had the right to vote for less than 20 years.

pd -What's with Switzerland?

update - R^2 and p-value added to graph

Please comment on the article here: Simply Statistics

The post Per capita GDP versus years since women received right to vote appeared first on All About Statistics.

I’ll reorder this week’s posts a bit in order to continue on a topic that came up yesterday.

A couple days ago a reporter wrote to me asking what I thought of this paper on Money, Status, and the Ovulatory Cycle. I responded:

Given the quality of the earlier paper by these researchers, I’m not inclined to believe anything these people write. But, to be specific, I can point out some things:

- The authors define low fertility as days 8-14. Oddly enough, these authors in their earlier paper used days 7-14. But according to womenshealth.gov, the most fertile days are between days 10 and 17. The choice of these days affects their analysis, and it is not a good sign that they use different days in different papers. (see more on this point in sections 2.3 and 3.1 of this paper: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf)

- They perform a lot of different analyses, and many others could be performed. For example, “Study 1 indicates that ovulation boosts women’s tendency to seek relative gains when given the opportunity to possess products superior to those of other women. However, we expect that ovulation should not have the same effect on women’s choices if they have the opportunity to possess better products than men who are potential mates.” But if they found the pattern in both groups they could argue that this is consistent with their theory in a different way. They’re essentially playing a lottery where they make the rules, and they can keep coming up with ways in which they win.

- For another example, “Ethnicity, relationship status, and income had no effect on the dependent measures.” But if they had found something, this could’ve fit the story. Recall that their previous paper was all about relationship status!

- Yet another example: “A repeated measures logistic regression . . . revealed a significant interaction . . . however, when women compared their house relative to that of men, there was no difference . . .” Again, any pattern here would fit their story.

- Here’s another example: “As in Study 1, we next examined women’s choices relative to women across the full 28-day cycle.” But in their earlier paper, they did not do this: rather, they only compared days 7-14 to days 17-25, completely excluding days 1-6, 15-16, and 26-28.

- The authors say their results fit their model, for example, “Consistent with H3, we predicted that ovulation would lead women to give smaller financial offers to other women but not to men.” Where exactly is this “prediction”? Did the authors really predict this ahead of time in a public way, or are they just saying they predicted it? In either case, how many other things did they predict? It wouldn’t be so impressive if they predicted (or could predict) thousands of possible comparisons and then they show what worked.

Perhaps the biggest problem with this study is that it purports to be all about effects within women, but it is a between-subjects design. That is, they do _not_ interview women at multiple points during their cycles. What they do is compare different women. That makes this sort of study close to hopeless. There’s just too much variation from person to person. What they’re doing is finding patterns in noise. If you look hard enough—and they do—you’ll find statistically significant patterns. Along with that they have a flexible theory that can explain just about anything.

To conclude, I’m not saying I think the authors “cheated” or “fished for statistical significance.” I have no idea how they did their analysis, but they very well could’ve made various data coding and data analysis decisions after seeing the data, and in addition that have a lot of choices in how to interpret their results in light of their theories. I think it’s hopeless. They might as well be reading tea leaves.

The reporter thanked me and wrote that he was still trying to figure out whether to write about the paper.

This made me think a bit: it’s not an easy question. I’m negative on the paper but others might be positive. Beyond that there’s a selection issue. Suppose you happen to be convinced that the article is worthless, and so you decide not to run the story. But somewhere else there is a reporter who swallows the press release, hook, line, and sinker. This other reporter would of course run a big story. Hence the selection bias that the stories that do get published are likely to repeat the hype. Which in turn gives a motivation for researchers and public relations people to do the hype in the first place. On the other hand, if you do run a skeptical story, then you’re continuing to give press for this silly study, giving it more attention that it doesn’t deserve (in my opinion).

This is a reporter’s dilemma that echoes the discussion I was having with Jeff Leek about when to shoot down what we believe to be bad work and when to ignore it. If lots of other people are paying attention to the paper, then I think a journalist is doing a service by shooting it down. But if the paper is basically being ignored (or perhaps just being treated as a “politically incorrect” oddity), then there’s no point in pulling it up from obscurity just to go on about what’s wrong with it.

The problem, as I see it, is when a claim presented with (essentially) no evidence is taken as truth and then treated as a stylized fact. And the norms of scientific publication, as well as the norms of science journalism, push toward this. If you act too uncertain in your scientific report, I think it becomes harder to get it published in a top journal (after all, they want to present “discoveries,” not “speculations”). And science journalism often seems to follow the researcher-as-Galileo mold.

The post Selection bias in the reporting of shaky research appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Selection bias in the reporting of shaky research appeared first on All About Statistics.

Please comment on the article here: Analysis with Programming

The post RUG-Philippines Meetup: Markov Switching Models in R appeared first on All About Statistics.

Final day of our workshop Advances in Scalable Bayesian Computation already, since tomorrow morning is an open research time ½ day! Another “perfect day in paradise”, with the Banff Centre campus covered by a fine snow blanket, still falling…, and making work in an office of BIRS a dream-like moment.

Still looking for a daily theme, parallelisation could be the right candidate, even though other talks this week went into parallelisation issues, incl. Steve’s talk yesterday. Indeed, Anthony Lee gave a talk this morning on interactive sequential Monte Carlo, where he motivated the setting by a formal parallel structure. Then, Darren Wilkinson surveyed the parallelisation issues in Monte Carlo, MCMC, SMC and ABC settings, before arguing in favour of a functional language called Scala. (Neat entries to those topics can be found on Darren’s blog.) And in the afternoon session, Sylvia Frühwirth-Schnatter exposed her approach to the (embarrassingly) parallel problem, in the spirit of Steve’s , David Dunson’s and Scott’s (a paper posted on the day I arrived in Chamonix and hence I missed!). There was plenty to learn from that talk (do not miss the Yin-Yang moment at 25 mn!), but it also helped me to break a difficulty I had with the consensus Bayes representation for two weeks (more on that later!). And, even though Marc Suchard mostly talked about flu and trees in a very pleasant and broad talk, he also had a slide on parallelisation to fit the theme! Although unrelated with parallelism,  Nicolas Chopin’s talk was on sequential quasi-Monte Carlo algorithms: while I had heard previous versions of this talk in Chamonix and BigMC, I found it full of exciting stuff. And it clearly got the room truly puzzled by this possibility, in a positive way! Similarly, Alex Lenkoski spoke about extreme rain events in Norway with no trace of parallelism, but the general idea behind the examples was to question the notion of the calibrated Bayesian (with possible connections with the cut models).

This has been a wonderful week and I am sure the participants got as much as I did from the talks and the informal exchanges. Thanks to BIRS for the sponsorship and the superb organisation of the week (and to the Banff Centre for providing such a paradisical environment). I feel very privileged to have benefited from this support, even though I deadly hope to be back in Banff within a few years.

Please comment on the article here: Xi'an's Og » R

The post Advances in scalable Bayesian computation [day #4] appeared first on All About Statistics.

Russ Roberts had this to say about the proposal to replacing the calculus requirement with statistics for students.

Statistics is in many ways much more useful for most students than calculus. The problem is, to teach it well is extraordinarily difficult. It’s very easy to teach a horrible statistics class where you spit back the definitions of mean and median. But you become dangerous because you think you know something about data when in fact it’s kind of subtle.

A little knowledge is a dangerous thing, more so for statistics than calculus.

This reminds me of a quote by Stephen Senn:

Statistics: A subject which most statisticians find difficult but in which nearly all physicians are expert.

Related: Elementary statistics book recommendation

Please comment on the article here: The Endeavour » Statistics

The post On replacing calculus with statistics appeared first on All About Statistics.

This is a short piece I wrote for the next issue of the Oracle newsletter produced by the International Institute of Forecasters.

Special section topics

We continue to publish special sections on selected topics. Because of the change in the way regular papers are now handled, we do not publish whole special issues any more. Rather, each issue has regular papers at the front, and if there are any special sections they appear at the back.

Lately, we had a special section in the last issue of 2013 on Flash indicators edited by Dean Croushore, Esther Ruiz and Miriam Scaglione. Most of the papers in this section were part of the IIF workshop on Flash Indicators held in Verbier, Switzerland, in January 2011.

The second issue for 2014 contains a special section on Energy forecasting edited by Tao Hong, Pierre Pinson and Shu Fan. This section concerns the Global Energy Forecasting Competition 2012. Hundreds of participants contributed to the competition (hosted on Kaggle), aiming to forecast hierarchical load data or wind power data. Those who contributed the most accurate forecasts were invited to describe their methods in detail for this special section.

We have an expanded energy competition in planning, this time focusing on probabilistic forecasting. Winning entries will again be invited to describe their methods in a special section of the journal. See www.gefcom.org for further details.

Other upcoming special sections include one on forecasting business cycles and one on predicting rare events (probably both appearing in the third issue of 2014).

Changes to the IJF editorial board

There have been several changes to the editorial board over the last few months. Asger Lunde and James Taylor, two of our associate editors, have stepped down. Asger was on the board for three years and was particularly helpful for his expertise in financial forecasting. James was part of the IJF board for ten years and handled a large number of papers in several areas including exponential smoothing, energy forecasting, call centre forecasting, and financial markets. Big shoes to fill! We are grateful to them both for their excellent service to the journal and the forecasting community.

Two new associate editors have joined the board in the last few months: David Rapach and Ana Galvao. David is at Saint Louis University, USA, and is a consultant at Merrill Lynch and a visiting scholar at the Federal Reserve Bank, St Louis. He will be handling papers in financial forecasting. Ana is Associate Professor of Economic Modelling and Forecasting at the Warwick Business School, UK. She will be primarily be handling papers on real-time forecasting and non-linear time series. We welcome them both to the board.

In addition, we have appointed Nikolaos Kourentzes (University of Lancaster, UK) as book review editor, taking over from Monica Adya who remains on the board as an associate editor. Nikos works in business forecasting, forecasting support systems and data mining. Thanks to Nikos for taking on this important role on the board, and we look forward to many interesting book reviews.

Review articles

A new initiative we have begun is the publication of regular review articles. We hope to publish survey articles of areas of forecasting to assist new researchers in the field and to help encourage new research on important topics. The first one should be ready later this year, so look out for it! If any reader has suggestions for topics where a review article would be helpful, please let me know.

Please comment on the article here: Hyndsight

The post IJF news appeared first on All About Statistics.

My latest update to The Popularity of Data Analysis Software is an attempt to use blog counts to estimate the popularity of analytics software. While I was able to greatly broaden the coverage of packages when studying job data, I made very little progress on the blog measure, adding new coverage for only Python and updating the previous counts for Stata. I post the results here mostly as a request for input from people who may know of more sources of blog lists than I have found so far.

I’ve also updated the jobs data slightly both in the main article and in the background one, How to Search for Analytics Jobs. While the changes to the search algorithm are greatly simplified, but worth reading only by people who are doing their own searches. Rather than jump through hoops to estimate total jobs for each software, I only count those for the main set of search terms.  The relative results from the new search algorithm are nearly identical to the previous, more complex one (r = .99).

Here’s the update on blogs:

Blogs

On Internet blogs, people write about software that interests them, showing how to solve problems and interpreting events in the field. Blog posts contain a great deal of information about their topic, and although it’s not as time consuming as a book to write, maintaining a blog certainly requires effort. Therefore, the number of bloggers writing about analytics software has potential as a measure of popularity or market share. Unfortunately, counting the number of relevant blogs is often a difficult task. General purpose software such as Java, Python, the C language variants and MATLAB have many more bloggers writing about general programming topics than just analytics. But separating them out isn’t easy. The name of a blog and the title of its latest post may not give you a clue that it routinely includes articles on analytics.

Another problem arises from the fact that what some companies would write up as a newsletter, others would do as a set of blogs, where several people in the company each contribute their own blog, but they’re also combined into a single company blog. Statsoft and Minitab offer examples of this. What’s really interesting is not company employees who are assigned to write blogs, but rather volunteers who freely provide their time.

In a few lucky cases, lists of such blogs are maintained, usually by blog consolidators, who combine many blogs into large “metablogs.” All I have to do is find such lists and count the blogs. I don’t attempt to extract the few vendor employees that I know are blended into such lists. However, I skip those lists that are exclusively employee-based (or very close to it). The results are shown in Table 1.

         Number
Software of Blogs Source
R         452     R-Bloggers.com
Python     60     SciPy.org
SAS        40     PROC-X.com, sasCommunity.org Planet
Stata      11     Stata-Bloggers.com

Table 1. Number of blogs devoted to each software package on March 5, 2014, and the source of the data.

R’s 452 blogs is quite an impressive number. For Python, I could only find that list of 60 that were devoted to the SciPy subroutine library. Some of those are likely cover topics besides analytics, but to determine which never cover the topic would be quite time consuming. The 40 blogs about SAS is still an impressive figure given that Stata was the only other software that even garnered a list anywhere. That list is at the vendor itself, StataCorp, but it consists of non-employees except for one.

While searching for lists of blogs on other software, I did find individual blogs that at least occasionally covered a particular topic. However, keeping this list up to date is far too time consuming given the relative ease with which other popularity measures are collected.

If you know of other lists of relevant blogs, please let me know and I’ll add them. If you’re a software vendor CEO reading this, and your company does not build a metablog or at least maintain a list of your bloggers, I recommend taking advantage of this important source of free publicity.

Please comment on the article here: r4stats.com

The post Analytics Software Popularity Update: Counting Blogs, Simplifying Job Searches appeared first on All About Statistics.

Evidence that the gender gap is increasing is strong.  The p-value of the slope of the fitted curve is less than 1e-6.

2) I notice that the number of schools and the number of students participating in the Freshman Survey has been falling for several years.  I wonder if the mix of schools represented in the survey is changing over time, and what effect this might have on the trends I am watching.  The percentage of "Nones" is different across different kinds of institutions (colleges, universities, public, private, etc.)  If participation rates are changing among these groups, that would affect the results.

3) Obviously college students are not representative of the general population.  Data from other sources indicate that the same trends are happening in the general population, but I haven't been able to make a quantitative comparison between college students and others.  Data from other sources also indicate that college graduates are slightly more likely to attend religious services, and to report a religious preference, than the general population.

Please comment on the article here: Probably Overthinking It

The post Freshman hordes slightly more godless than ever appeared first on All About Statistics.

I had a brief email exchange with Jeff Leek regarding our recent discussions of replication, criticism, and the self-correcting process of science.

Jeff writes:

(1) I can see the problem with serious, evidence-based criticisms not being published in the same journal (and linked to) studies that are shown to be incorrect. I have been mostly seeing these sorts of things show up in blogs. But I’m not sure that is a bad thing. I think people read blogs more than they read the literature. I wonder if this means that blogs will eventually be a sort of “shadow literature”?

(2) I think there is a ton of bad literature out there, just like there is a ton of bad stuff on Google. If we focus too much on the bad stuff we will be paralyzed. I still manage to find good papers despite all the bad papers.

(3) I think one positive solution to this problem is to incentivize/publish referee reports and give people credit for a good referee report just like they get credit for a good paper. Then, hopefully the criticisms will be directly published with the paper, plus it will improve peer review.

A key decision point is what to do when we encounter bad research that gets publicity. Should we hype it up (the “Psychological Science” strategy), slam it (which is often what I do), ignore it (Jeff’s suggestion), or do further research to contextualize it (as Dan Kahan sometimes does)?

OK, I’m not planning to take that last option any time soon: research requires work, and I have enough work to do already. And we’re not in the business of hype here (unless the topic is Stan). So let’s talk about the other two options: slamming bad research or ignoring it. Slamming can be fun but it can carry an unpleasant whiff of vigilantism. So maybe ignoring the bad stuff is the better option. As I wrote earlier:

Ultimately, though, I don’t know if the approach of “the critics” (including myself) is the right one. What if, every time someone pointed me to a bad paper, I were to just ignore it and instead post on something good? Maybe that would be better. The good news blog, just like the happy newspaper that only prints stories of firemen who rescue cats stuck in trees and cures for cancer. But . . . the only trouble is that newspapers, even serious newspapers, can have low standards for reporting “cures for cancer” etc. For example, here’s the Washington Post and here’s the New York Times. Unfortunately, these major news organizations seem often to follow the “if it’s published in a top journal, it must be correct” rule.

Still and all, maybe it would be best for me, Ivan Oransky, Uri Simonsohn, and all the rest of us to just turn the other cheek, ignore the bad stuff and just resolutely focus on good news. It would be a reasonable choice, I think, and I would fully respect someone who were to blog just on stuff that he or she likes.

Why, then?

Why, then, do I spend time criticizing research mistakes and misconduct, given that it could even be counterproductive by drawing attention to sorry efforts that otherwise might be more quickly forgotten?

The easiest answer is education. When certain mistakes are made over and over, I can make a contribution by naming, exploring, and understanding the error (as in this famous example or, indeed, many of the items on the lexicon).

Beyond this, exploring errors can be a useful research direction. For example, our criticism in 2007 of the notorious beauty-and-sex-ratio study led in 2009 to a more general exploration of the issue of statistical significance, which in turn led to a currently-in-the-revise-and-resubmit-stage article on a new approach to design analysis.

Similarly, the anti-plagiarism rants of Thomas Basbøll and myself led to a paper on the connection between plagiarism and ideas of statistical evidence, and another paper storytelling as model checking. So, for me, criticism can open doors to new research.

But it’s not just about research

One more thing, and it’s a biggie. People talk about the self-correcting nature of the scientific process. But this self-correction only happens if people do the correction. And, in the meantime, bad ideas can have consequences.

The most extreme example was the infamous Excel error by Reinhardt and Rogoff, which may well have influenced government macroeconomic policy. In a culture of open data and open criticism, the problem might well have been caught. Recall that the paper was published in 2009, its errors came to light in 2013, but as early as 2010, Dean Baker was publicly asking for the data.

Scientific errors and misrepresentations can also have indirect influences. Consider …, where Stephen Jay Gould notoriously… And evolutionary psychology continues to be a fertile area for pseudoscience. Just the other day, Tyler Cowen posted, on a paper called “Money, Status, and the Ovulatory Cycle,” which he labeled as the “politically incorrect paper of the month.”

The trouble is that the first two authors are Kristina Durante, Vladas Griskevicius, and I can’t really believe anything that comes out of that research team, given they earlier published the ridiculous claim that among women in relationships, 40% in the ovulation period supported Romney, compared to 23% in the non-fertile part of their cycle. (For more on this issue, see section 5 of this paper.)

Does publication and publicity of ridiculous research cause problems (besides wasting researchers’ time)? Maybe so. Two malign effects that I can certainly imagine coming from this sort of work are (a) a reinforcing of gender stereotypes, and (b) a cynical attitude about voting and political participation. Some stereotypes reflect reality, I’m sure of that—and I’m with Steven Pinker on not wanting to stop people from working in controversial areas. But I don’t think anything is gained from the sort of noise-mining that allows researchers to find whatever they want. At this point we as statisticians can contribute usefully be stepping in and saying: Hey, this stuff is bogus! There ain’t no 24% vote swings. If you think it’s important to demonstrate that people are affected in unexpected ways by hormones, then fine, do it. But do some actual scientific research. Finding “p less than 0.05″ patterns in a non representative between-subjects study doesn’t cut it, if your goal is to estimate within-person effects.

What about meeeeeeeee?

Should I be spending time on this? That’s another question. All sorts of things are worth doing by somebody but not necessarily by me. Maybe I’d be doing more for humanity by working on Stan, or studying public opinion trends in more detail, or working harder on pharmacokinetic modeling, or figuring out survey weighting, or go into cancer research. Ir maybe I should chuck it all and do direct services with poor people, or get a million-dollar job, make a ton of money, and then give it all away. Lots of possibilities. For this, all I can say is that these little investigations can be interesting and fruitful for my general understanding of statistics (see the items under the heading “Why then” above). But, sure, too much criticism would be too much.

“Bumblers and pointers”

A few months ago after I published an article criticizing some low-quality published research, I received the following email:

There are two kinds of people in science: bumblers and pointers. Bumblers are the people who get up every morning and make mistakes, trying to find truth but mainly tripping over their own feet, occasionally getting it right but typically getting it wrong. Pointers are the people who stand on the sidelines, point at them, and say “You bumbled, you bumbled.” These are our only choices in life.

The sad thing is, this email came from a psychology professor! Pretty sad to think that he thought those were our two choices in life. I hope he doesn’t teach this to his students. I like to do both, indeed at the same time: When I do research (“bumble”), I aim criticism at myself, poking holes in everything I do (“pointing”). And when I criticize (“pointing”), I do so in the spirit of trying to find truth (“bumbling”).

If you’re a researcher and think you can do only one or the other of these two things, you’re really missing out.

The post How much time (if any) should we spend criticizing research that’s fraudulent, crappy, or just plain pointless? appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post How much time (if any) should we spend criticizing research that’s fraudulent, crappy, or just plain pointless? appeared first on All About Statistics.

Please comment on the article here: Entsophy

The post The Secret to Entropy’s role in Statistics appeared first on All About Statistics.