Ed Vul writes:

In the course of tinkering with someone else’s hairy dataset with a great many candidate explanatory variables (some of which are largely orthogonal factors, but the ones of most interest are competing “binning” schemes of the same latent elements). I wondered about the following “model selection” strategy, which you may have alluded to in your multiple comparisons paper:

Include all plausible factors/interactions as random intercept effects (i.e., (1|A), (1|A:B) in lmer parlance [that’s stan_lmer() now — ed.]). Since there are many competing, non-orthogonal binning schemes included all at once, the model would be overdetermined (singular) if they were all included as fixed effects [he means “non-varying coefficients.” Or maybe he means “varying coefficients estimated without regularization.” The “fixed”/”random” terminology is unclear. — ed.]. However, as random effects [“varying coefficients estimated using regularization” — ed.], we can rely on partial pooling and shrinkage to sort out among them, such that variance along factors that are not well supported by the data (or are explained away by other factors) shrinks to zero. This happens quite decisively in lmer, but a “regularizing” exponential prior on the variances in a fully bayesian model would achieve something similar, I think [easy to do in stan or rstanarm — ed.]. (A more sophisticated approach would be to put a common, pooling prior on the variances for all the individual factors…)

This approach seems to yield sensible results, but I am a bit concerned because I have never seen it used by others, so I am probably missing something. It may just be that it is rarely computationally practical to include all candidate factors/binning-schemes in such an “overcomplete” model. Or perhaps there is a compelling reason why explanatory variables of substantive interest should be treated as fixed, rather than random effects? Is there a fundamental problem with this approach that I am not thinking of? Or is this a well-known technique that I have simply never heard of?

My reply (in addition to the editorial comments inserted above):

This sounds fine. I think, though, you may be overestimating what lmer will do (perhaps my fault given that I featured lmer in my multilevel modeling book). The variance estimate from lmer can be noisy. But, sure, yes, partial pooling is the way to go, I think. Once you’ve fit the model I don’t really see the need to shrink small coefficients all the way to zero, but I guess you can if you want. Easiest I think is to fit the model in Stan (or rstanarm if you want to use lmer-style notation).

Also, the whole fixed/random thing is no big deal. You can allow any and all coefficients to vary; it’s just that if you don’t have a lot of data then it can be a good idea to put informative priors on some of the group-level variance parameters to control the amount of partial pooling. Putting in all these priors might sound kinda weird but that’s just cos we don’t have a lot of experience with such models. Once we have more examples of these (and once they’re in the new edition of my book with Jennifer), then it will be no big deal for you and others to follow this plan in your own models.

Informative priors on group-level variances makes a lot more sense than making a priori decisions to do no pooling, partial pooling, or complete pooling.

The post Partial pooling with informative priors on the hierarchical variance parameters: The next frontier in multilevel modeling appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Partial pooling with informative priors on the hierarchical variance parameters: The next frontier in multilevel modeling appeared first on All About Statistics.

The article "Fisher's transformation of the correlation coefficient" featured a Monte Carlo simulation that generated sample correlations from bivariate normal data. The simulation used three steps:

1. Simulate B samples of size N from a bivariate normal distribution with correlation ρ.
2. Use PROC CORR to compute the sample correlation matrix for each of the B samples.
3. Use the DATA step to extract the off-diagonal elements from the correlation matrices.

After the three steps, you obtain a distribution of B sample correlation coefficients that approximates the sampling distribution of the Pearson correlation coefficient for bivariate normal data.

There is a simpler way to simulate the correlation estimates: You can directly simulate from the Wishart distribution. Each draw from the Wishart distribution is a sample covariance matrix for a multivariate normal sample of size N. If you convert that covariance matrix to a correlation matrix, you can immediately extract the off-diagonal elements, as shown in the following SAS/IML statements:

%let rho = 0.8;           /* correlation for bivariate normal distribution */
%let N = 20;              /* sample size */
%let NumSamples = 2500;   /* number of simulated samples */
 
/* generate sample correlation coefficients by using Wishart distribution */
proc iml;
call randseed(12345);
NumSamples = &NumSamples;
DF = &N - 1;              /* X ~ N obs from MVN(0, Sigma) */
Sigma = {1     &rho,      /* covariance for MVN samples */
         &rho   1  };
S = RandWishart(NumSamples, DF, Sigma); /* each row is 2x2 matrix */
Corr = j(NumSamples, 1);  /* allocate vector for correlation estimates */
do i = 1 to nrow(S);      /* convert to correlation; extract off-diagonal */
   Corr[i] = cov2corr( shape(S[i,], 2, 2) )[1,2];
end;

You can create a comparative histogram of the sample correlation coefficients. In the following graph, the histogram at the top of the panel displays the distribution of the simulated correlation coefficients from the three-step method. The bottom histogram displays the distribution of correlations coefficients that are generated from the Wishart distribution.

Visually, the histograms appear to be similar. You can use PROC NPAR1WAY to run various hypothesis tests that compare the distributions; all tests support the hypothesis that these two distributions are equivalent.

If you'd like to see the complete analysis, you can download the SAS program that runs both simulations and compares the resulting distributions.

Although the Wishart distribution is more efficient for this simulation, recall that the Wishart distribution assumes that the underlying data distribution is multivariate normal. In contrast, the three-step simulation is more general. It can be used to generate correlation coefficients for any data distribution. So although the three-step simulation is not necessary for multivariate normal data, it is still an important technique to store in your simulation toolbox.

The post Simulate correlations by using the Wishart distribution appeared first on The DO Loop.

Please comment on the article here: The DO Loop

The post Simulate correlations by using the Wishart distribution appeared first on All About Statistics.

I usually stick to the InfoVis track at VIS, and that was also the case this time. The papers in this part of my report from VIS 2017 cover text, small visualizations like sparklines and glyphs, tools for showing story structure, and multi-device interaction with data.

Word Clouds

Tag or word clouds are second only to pie charts in the kind of scorn you get for using them. And yet, they were a topic at InfoVis this year, with three papers dealing with their creation and readability.

EdWordle: Consistency-preserving Word Cloud Editingby Yunhai Wang, Xiaowei Chu, Chen Bao, Lifeng Zhu, Oliver Deussen, Baoquan Chen, and Michael Sedlmair presents a fairly straightforward, but also quite neat system to edit word clouds. The EdWordle tool allows you to add and remove words, move words around, or trigger a relayout that neatly animates. This lets you make more semantic word clouds by moving related words closer to each other (like when they’re part of a phrase).

Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries by Cristian Felix, Enrico Bertini, and Steven Franconeri goes the opposite direction, asking how well people can read them. They compared the classic text size with bar length, color, and bubble size. They measured time and error and found that the more precise codings (like bars) took longer to read than the ones that performed more poorly (like font size). Their materials are available.

Perceptual Biases in Font Size as a Data Encoding by Eric Carlson Alexander, Chih-Ching Chang, Mariana Shimabukuro, Steve Franconeri, Christopher Collins, and Michael Gleicher reports on 14 studies(!) on how the reading of word clouds depends on various factors such as word length, the kinds of characters involved, ascenders and descenders, etc. They found evidence for many of the assumed (but largely unproven) effects, but also that people really aren’t that bad at reading word clouds to begin with. Clearly, a word cloud will not be used for high-precision tasks, but they’re fine for comparing when there is a reasonable difference.

Sparklines and Glyphs

An Exploratory Study of Word-Scale Graphics in Data-Rich Text Documents, Pascal Goffin, Jeremy Boy, Wesley Willett, Petra Isenberg also mentions words, but in a very different context. It’s about small graphics better known as sparklines. They let designers play with Wikipedia articles to see what kinds of small graphics they’d want to include. The result is quite interesting, with a large number of different types and designs, and not limited to just numerical data. A list of designs from the study is available.

A Systematic Review of Experimental Studies on Data Glyphs by Johannes Fuchs, Petra Isenberg, Anastasia Bezerianos, and Daniel A. Keim looks at the many different types of glyphs (and their varying definitions). They compared a large number of papers going back to 1957, and put together a nice overview of what we know, where there are gaps, and where there are contradictions in the literature.

Brushing, Multiple Views, and Interaction

While there is a lot of talk of multiple views and interaction, there is not enough research to figure out how to actually use those. These three papers pushed the boundaries on what we know about this aspect of visualization.

MyBrush: Brushing and Linking with Personal Agency by Philipp Koytek, Charles Perin, Jo Vermeulen, Elisabeth André, and Sheelagh Carpendale takes apart the concept of brushing and linking to let users create their own brushing system by picking the type of source, link, and target they prefer. They show that people found that quite useful and were able to create a wide variety of tools depending on what they were after. Their source code is available and you can play with their implementation in your browser.

Keeping Multiple Views Consistent: Constraints, Validations, and Exceptions in Visualization Authoring by Zening Qu and Jessica Hullman looks at how some of the usual rules for scaling charts (like including zero, scaling to the max) can be adapted for multiple-view dashboards. They studied a wide range of different scenarios, with common measures using the same or different encodings, etc. They came up with a number of interesting results that appear to be quite useful and directly applicable.

VisTiles: Coordinating and Combining Co-located Mobile Devices for Visual Data Exploration by Ricardo Langner, Tom Horak, and Raimund Dachselt shows a number of ideas for using multiple mobile devices (in particular tables) to expand the way interaction with data works. They have a large set of interesting demos, and their source code is also available for you to try it out yourself.

Stories and Interaction

Visualizing Nonlinear Narratives with Story Curves by Nam Wook Kim, Benjamin Bach, Hyejin Im, Sasha Schriber, Markus Gross, and Hanspeter Pfister presents a very interesting tool to explore the narrative structure of books and movies, in particular flash-backs and flash-forwards. You can run it in your browser and play with a good selection of movies with interesting structure (Memento, Eternal Sunshine of the Spotless Mind, etc.). There’s also lots of interaction to find particular kinds of features, which they tested with literary and film scholars. Source code and more materials are available from their project page.

I should also mention a talk I unfortunately missed, Timelines Revisited: A Design Space and Considerations for Expressive Storytelling by Matthew Brehmer, Bongshin Lee, Benjamin Bach, Nathalie Henry Riche, and Tamara Munzner. They look at a variety of ways to show timelines and sequences, and how to use them to tell stories. Their materials are available online as well.

See also:

• Best Papers and Other Awards
• Keynote and Capstone

Please comment on the article here: eagereyes

The post IEEE VIS 2017: Word Clouds, Stories, Interaction appeared first on All About Statistics.

Two case studies:

Splines in Stan, by Milad Kharratzadeh.

Spatial Models in Stan: Intrinsic Auto-Regressive Models for Areal Data, by Mitzi Morris.

This is great. Thanks, Mitzi! Thanks, Milad!

The post Splines in Stan; Spatial Models in Stan ! appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Splines in Stan; Spatial Models in Stan ! appeared first on All About Statistics.

Please comment on the article here: Econometrics Beat: Dave Giles' Blog

The post November Reading appeared first on All About Statistics.

Demis Glasford does research in social psychology and asks:

I was wondering if you had ever considered publishing a top ten ‘do’s/don’ts’ for those of us that are committed to doing better science, but don’t necessarily have the time to devote to all of these issues [of statistics and research methods].

Obviously, there is a lot of nuance in both methods and stats for any particular project. So, I’m not asking you for a ‘one size fits all’, but more of a 5 or 10 factor checklist as a framework for those of us committed to doing better work, but worried we may not have the expertise or time to follow-through on these commitments. Sort of a—whatever you do, at least do x, y, and z.

I looked up Glasford on the internet and found this description of his research:

The focus of much of my work is on three interrelated streams of research questions that are concerned with understanding: how people make decisions about what to do when faced with injustice; what compels people to join and stay involved in political protest that can benefit their own group, as well as groups they do not belong to; and when, why, and what helps individuals from groups of differing power improve relations with one another.

Wow—this sounds important. I should talk with this guy.

In the meantime, do I have a checklist of 10 items? I’ve given advice to psychology researchers from time to time but I don’t have a convenient list of 10 things.

But we should have such a list! Can you make some suggestions in comments? Also, if anyone out there is in contact with any leading social psychologists, maybe we could get their thoughts too? There’s a lot I disagree with in the writings of, say, Susan “terrorists” Fiske or Daniel “shameless little bullies” Gilbert or Mark “Evilicious” Hauser or all the other people you’re sick of hearing about on this blog—but, say what you want about these people, they’ve thought a lot about psychology research and I’d be interested in what their top 10 tips would be. Not tips on how to get published in PNAS or wherever, but tips on doing better science.

Unfortunately I don’t expect we’ll hear from the above people (I’d be happy to be surprised on that end, though!), so in the meantime I’d love to hear your thoughts.

OK, I’ll start with items #1, 2, and 3 on the top-10 list, in decreasing order of importance:

1. Learning from data involves three stages of extrapolation: from sample to population, from treatment group to control group, and from measurement to the underlying construct of interest. Worry about all three of these, but especially about measurement, as this tends to be taken for granted in statistics discussions.

2. Variance is as important as bias. To put it another way, take a look at your (estimated) bias and your standard error. Whichever of these is higher, that’s what you should be concerned about.

3. Measurement error and variation are concerns even if your estimate is more than 2 standard errors from zero. Indeed, if variation or measurement error are high, then you learn almost nothing from an estimate even if it happens to be “statistically significant.”

OK, none of the above are so pithy, and I’m open to the idea of other items bumping these down the list.

It’s your turn.

The post Please contribute to this list of the top 10 do’s and don’ts for doing better science appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Please contribute to this list of the top 10 do’s and don’ts for doing better science appeared first on All About Statistics.

Please comment on the article here: Junk Charts

The post Let’s not mix these polarized voters as the medians run away from one another appeared first on All About Statistics.

Part 2 of my IEEE VIS 2017 report covers the keynote and capstone talks at the conference. They were not just at opposite ends of the conference, but also about as different as they can be in terms of content and delivery.

Keynote: Jacqueline H. Chen, Analytics Inspired Visualization: A Holistic In-situ Scientific Workflow at Extreme Scale

Jackie Chen is a highly regarded scientist and she gave an interesting, if overly detailed, talk about combustion research. She talked about the challenges they face with the enormous amounts of data their simulations produce. The traditional workflow of running simulations (which can take weeks), storing all the data, and then doing analysis is clearly slow and not very flexible.

Chen described a newer approach where they integrate the analysis and visualization much earlier and thus create a tighter loop. They also use good ideas from visualization, like multiple linked views that show different aspects of the data (temperature, mixing rate, etc) and similar. In addition to fairly directly rendering the data, they also create topological abstractions to see how certain patterns move over time, cause local extinctions of a flame, etc.

Science keynotes can be hit or miss, and this one was more of the latter, unfortunately. Chen is a good speaker and she had a lot to talk about – a bit too much, even. I really didn't need to know quite that much about combustion research, and would have liked to see more about how they use visualization  – but people doing volume rendering or other scientific applications might have gotten more out of it.

Capstone: Giorgia Lupi, Data Humanism – The Revolution will be Visualized

The capstone talk on Friday was the opposite of the keynote in many respects. Giorgia Lupi’s talk was fun and engaging, and got her a lot of applause not just at the end, but also a few times during the talk. She covered many of her projects, including Friends in Space, World POTUS, and of course Dear Data (also previously covered in this fine publication).

Among her latest projects are a show about fashion at MoMA, for which she created a data sketch – except there was no data. She collected her own from the curator’s research notes and talking to her, and then turned that into a wall-sized drawing.

Perhaps the most stunning is her collaboration with musician Kaki King, however. Giorgia and Kaki collected data similar to Dear Data, which Kaki turned into a song, then Georgia collected data about the score and how Kaki plays it and visualized that. The result is this stunning video:

What struck me during the talk, even though I should have figured this out a long time ago, is how much of Giorgia’s process is about the collection of the data, not just its representation. Each of her pieces is also very specific to the data and its context, which is almost the exact opposite of visualization techniques that tend to be general and independent of context. She talked about the value of designing visualizations that are specific to their topic, and I think there’s a lot of truth in that.

Giorgia has written up many of the ideas she talked about in her piece on what she calls Data Humanism. It was a great talk and I think it gave everybody in the audience a lot to think about.

Seminal capstone on #DataHumanism from the amazing @giorgialupi at #IEEEVIS. Sometimes the world changes in good ways. #LuckyToBeHere https://t.co/sfJhkx2zg2

— giCentre (@giCentre) October 6, 2017

See also:

• Best Papers and Other Awards
• Word Clouds, Stories, Interaction

Please comment on the article here: eagereyes

The post IEEE VIS 2017: Keynote and Capstone appeared first on All About Statistics.

Today is my second day of triaging Github issues of the rmarkdown repo. I just came across rstudio/rmarkdown#980, an issue that is almost 8 months old. I didn’t immediately start working on it, but took a look at the cross-references first. Then I saw Karl Broman cross-referenced this issue from ramnathv/htmlwidgets#265, so I clicked the link and went over there.

Problem solved. I didn’t need to do anything, and happily closed the rmarkdown issue. Had Karl not referenced this issue in his htmlwidgets pull request, I’d probably have to redo what he had done (debug the problem and figure out the fix independently), hence waste my time.

Thanks, Karl, for the “free ride”. You are so cool.

Please comment on the article here: Yihui's Blog on Yihui Xie | 谢益辉

The post Thanks, Karl, for Cross-referencing on Github! appeared first on All About Statistics.

Please comment on the article here: Big Data, Plainly Spoken (aka Numbers Rule Your World)

The post Analysts must reckon with the fake data menace appeared first on All About Statistics.