C. S. Peirce: 10 Sept, 1839-19 April, 1914

Yesterday was C.S. Peirce’s birthday. He’s one of my all time heroes. You should read him: he’s a treasure chest on essentially any topic. I only recently discovered a passage where Popper calls Peirce one of the greatest philosophical thinkers ever (I don’t have it handy). If Popper had taken a few more pages from Peirce, he would have seen how to solve many of the problems in his work on scientific inference, probability, and severe testing. I’ll blog the main sections of a (2005) paper of mine over the next few days. It’s written for a very general philosophical audience; the statistical parts are pretty informal. I first posted it in 2013. Happy (slightly belated) Birthday Peirce.

Peircean Induction and the Error-Correcting Thesis
Deborah G. Mayo
Transactions of the Charles S. Peirce Society: A Quarterly Journal in American Philosophy, Volume 41, Number 2, 2005, pp. 299-319

Peirce’s philosophy of inductive inference in science is based on the idea that what permits us to make progress in science, what allows our knowledge to grow, is the fact that science uses methods that are self-correcting or error-correcting:

Induction is the experimental testing of a theory. The justification of it is that, although the conclusion at any stage of the investigation may be more or less erroneous, yet the further application of the same method must correct the error. (5.145)

Inductive methods—understood as methods of experimental testing—are justified to the extent that they are error-correcting methods. We may call this Peirce’s error-correcting or self-correcting thesis (SCT):

Self-Correcting Thesis SCT: methods for inductive inference in science are error correcting; the justification for inductive methods of experimental testing in science is that they are self-correcting.

Peirce’s SCT has been a source of fascination and frustration. By and large, critics and followers alike have denied that Peirce can sustain his SCT as a way to justify scientific induction: “No part of Peirce’s philosophy of science has been more severely criticized, even by his most sympathetic commentators, than this attempted validation of inductive methodology on the basis of its purported self-correctiveness” (Rescher 1978, p. 20).

In this paper I shall revisit the Peircean SCT: properly interpreted, I will argue, Peirce’s SCT not only serves its intended purpose, it also provides the basis for justifying (frequentist) statistical methods in science. While on the one hand, contemporary statistical methods increase the mathematical rigor and generality of Peirce’s SCT, on the other, Peirce provides something current statistical methodology lacks: an account of inductive inference and a philosophy of experiment that links the justification for statistical tests to a more general rationale for scientific induction. Combining the mathematical contributions of modern statistics with the inductive philosophy of Peirce, sets the stage for developing an adequate justification for contemporary inductive statistical methodology.

2. Probabilities are assigned to procedures not hypotheses

Peirce’s philosophy of experimental testing shares a number of key features with the contemporary (Neyman and Pearson) Statistical Theory: statistical methods provide, not means for assigning degrees of probability, evidential support, or confirmation to hypotheses, but procedures for testing (and estimation), whose rationale is their predesignated high frequencies of leading to correct results in some hypothetical long-run. A Neyman and Pearson (NP) statistical test, for example, instructs us “To decide whether a hypothesis, H, of a given type be rejected or not, calculate a specified character, x₀, of the observed facts; if x> x₀ reject H; if x< x₀ accept H.” Although the outputs of N-P tests do not assign hypotheses degrees of probability, “it may often be proved that if we behave according to such a rule … we shall reject H when it is true not more, say, than once in a hundred times, and in addition we may have evidence that we shall reject H sufficiently often when it is false” (Neyman and Pearson, 1933, p.142).[i]

The relative frequencies of erroneous rejections and erroneous acceptances in an actual or hypothetical long run sequence of applications of tests are error probabilities; we may call the statistical tools based on error probabilities, error statistical tools. In describing his theory of inference, Peirce could be describing that of the error-statistician:

The theory here proposed does not assign any probability to the inductive or hypothetic conclusion, in the sense of undertaking to say how frequently that conclusion would be found true. It does not propose to look through all the possible universes, and say in what proportion of them a certain uniformity occurs; such a proceeding, were it possible, would be quite idle. The theory here presented only says how frequently, in this universe, the special form of induction or hypothesis would lead us right. The probability given by this theory is in every way different—in meaning, numerical value, and form—from that of those who would apply to ampliative inference the doctrine of inverse chances. (2.748)

The doctrine of “inverse chances” alludes to assigning (posterior) probabilities in hypotheses by applying the definition of conditional probability (Bayes’s theorem)—a computation requires starting out with a (prior or “antecedent”) probability assignment to an exhaustive set of hypotheses:

If these antecedent probabilities were solid statistical facts, like those upon which the insurance business rests, the ordinary precepts and practice [of inverse probability] would be sound. But they are not and cannot be statistical facts. What is the antecedent probability that matter should be composed of atoms? Can we take statistics of a multitude of different universes? (2.777)

For Peircean induction, as in the N-P testing model, the conclusion or inference concerns a hypothesis that either is or is not true in this one universe; thus, assigning a frequentist probability to a particular conclusion, other than the trivial ones of 1 or 0, for Peirce, makes sense only “if universes were as plentiful as blackberries” (2.684). Thus the Bayesian inverse probability calculation seems forced to rely on subjective probabilities for computing inverse inferences, but “subjective probabilities” Peirce charges “express nothing but the conformity of a new suggestion to our prepossessions, and these are the source of most of the errors into which man falls, and of all the worse of them” (2.777).

Hearing Pierce contrast his view of induction with the more popular Bayesian account of his day (the Conceptualists), one could be listening to an error statistician arguing against the contemporary Bayesian (subjective or other)—with one important difference. Today’s error statistician seems to grant too readily that the only justification for N-P test rules is their ability to ensure we will rarely take erroneous actions with respect to hypotheses in the long run of applications. This so called inductive behavior rationale seems to supply no adequate answer to the question of what is learned in any particular application about the process underlying the data. Peirce, by contrast, was very clear that what is really wanted in inductive inference in science is the ability to control error probabilities of test procedures, i.e., “the trustworthiness of the proceeding”. Moreover it is only by a faulty analogy with deductive inference, Peirce explains, that many suppose that inductive (synthetic) inference should supply a probability to the conclusion: “… in the case of analytic inference we know the probability of our conclusion (if the premises are true), but in the case of synthetic inferences we only know the degree of trustworthiness of our proceeding (“The Probability of Induction” 2.693).

Knowing the “trustworthiness of our inductive proceeding”, I will argue, enables determining the test’s probative capacity, how reliably it detects errors, and the severity of the test a hypothesis withstands. Deliberately making use of known flaws and fallacies in reasoning with limited and uncertain data, tests may be constructed that are highly trustworthy probes in detecting and discriminating errors in particular cases. This, in turn, enables inferring which inferences about the process giving rise to the data are and are not warranted: an inductive inference to hypothesis H is warranted to the extent that with high probability the test would have detected a specific flaw or departure from what H asserts, and yet it did not.

3. So why is justifying Peirce’s SCT thought to be so problematic?

You can read Section 3 here. (it’s not necessary for understanding the rest).

4. Peircean induction as severe testing

… [I]nduction, for Peirce, is a matter of subjecting hypotheses to “the test of experiment” (7.182).

The process of testing it will consist, not in examining the facts, in order to see how well they accord with the hypothesis, but on the contrary in examining such of the probable consequences of the hypothesis … which would be very unlikely or surprising in case the hypothesis were not true. (7.231)

When, however, we find that prediction after prediction, notwithstanding a preference for putting the most unlikely ones to the test, is verified by experiment,…we begin to accord to the hypothesis a standing among scientific results.

This sort of inference it is, from experiments testing predictions based on a hypothesis, that is alone properly entitled to be called induction. (7.206)

While these and other passages are redolent of Popper, Peirce differs from Popper in crucial ways. Peirce, unlike Popper, is primarily interested not in falsifying claims but in the positive pieces of information provided by tests, with “the corrections called for by the experiment” and with the hypotheses, modified or not, that manage to pass severe tests. For Popper, even if a hypothesis is highly corroborated (by his lights), he regards this as at most a report of the hypothesis’ past performance and denies it affords positive evidence for its correctness or reliability. Further, Popper denies that he could vouch for the reliability of the method he recommends as “most rational”—conjecture and refutation. Indeed, Popper’s requirements for a highly corroborated hypothesis are not sufficient for ensuring severity in Peirce’s sense (Mayo 1996, 2003, 2005). Where Popper recoils from even speaking of warranted inductions, Peirce conceives of a proper inductive inference as what had passed a severe test—one which would, with high probability, have detected an error if present.

In Peirce’s inductive philosophy, we have evidence for inductively inferring a claim or hypothesis H when not only does H “accord with” the data x; but also, so good an accordance would very probably not have resulted, were H not true. In other words, we may inductively infer H when it has withstood a test of experiment that it would not have withstood, or withstood so well, were H not true (or were a specific flaw present). This can be encapsulated in the following severity requirement for an experimental test procedure, ET, and data set x.

Hypothesis H passes a severe test with x iff (firstly) x accords with H and (secondly) the experimental test procedure ET would, with very high probability, have signaled the presence of an error were there a discordancy between what H asserts and what is correct (i.e., were H false).

The test would “have signaled an error” by having produced results less accordant with H than what the test yielded. Thus, we may inductively infer H when (and only when) H has withstood a test with high error detecting capacity, the higher this probative capacity, the more severely H has passed. What is assessed (quantitatively or qualitatively) is not the amount of support for H but the probative capacity of the test of experiment ET (with regard to those errors that an inference to H is declaring to be absent)……….

You can read the rest of Section 4 here here

5. The path from qualitative to quantitative induction

In my understanding of Peircean induction, the difference between qualitative and quantitative induction is really a matter of degree, according to whether their trustworthiness or severity is quantitatively or only qualitatively ascertainable. This reading not only neatly organizes Peirce’s typologies of the various types of induction, it underwrites the manner in which, within a given classification, Peirce further subdivides inductions by their “strength”.

(I) First-Order, Rudimentary or Crude Induction

Consider Peirce’s First Order of induction: the lowest, most rudimentary form that he dubs, the “pooh-pooh argument”. It is essentially an argument from ignorance: Lacking evidence for the falsity of some hypothesis or claim H, provisionally adopt H. In this very weakest sort of induction, crude induction, the most that can be said is that a hypothesis would eventually be falsified if false. (It may correct itself—but with a bang!) It “is as weak an inference as any that I would not positively condemn” (8.237). While uneliminable in ordinary life, Peirce denies that rudimentary induction is to be included as scientific induction. Without some reason to think evidence of H‘s falsity would probably have been detected, were H false, finding no evidence against H is poor inductive evidence for H. H has passed only a highly unreliable error probe.

(II) Second Order (Qualitative) Induction

It is only with what Peirce calls “the Second Order” of induction that we arrive at a genuine test, and thereby scientific induction. Within second order inductions, a stronger and a weaker type exist, corresponding neatly to viewing strength as the severity of a testing procedure.

The weaker of these is where the predictions that are fulfilled are merely of the continuance in future experience of the same phenomena which originally suggested and recommended the hypothesis… (7.116)

The other variety of the argument … is where [results] lead to new predictions being based upon the hypothesis of an entirely different kind from those originally contemplated and these new predictions are equally found to be verified. (7.117)

The weaker type occurs where the predictions, though fulfilled, lack novelty; whereas, the stronger type reflects a more stringent hurdle having been satisfied: the hypothesis has had “novel” predictive success, and thereby higher severity. (For a discussion of the relationship between types of novelty and severity see Mayo 1991, 1996). Note that within a second order induction the assessment of strength is qualitative, e.g., very strong, weak, very weak.

The strength of any argument of the Second Order depends upon how much the confirmation of the prediction runs counter to what our expectation would have been without the hypothesis. It is entirely a question of how much; and yet there is no measurable quantity. For when such measure is possible the argument … becomes an induction of the Third Order [statistical induction]. (7.115)

It is upon these and like passages that I base my reading of Peirce. A qualitative induction, i.e., a test whose severity is qualitatively determined, becomes a quantitative induction when the severity is quantitatively determined; when an objective error probability can be given.

(III) Third Order, Statistical (Quantitative) Induction

We enter the Third Order of statistical or quantitative induction when it is possible to quantify “how much” the prediction runs counter to what our expectation would have been without the hypothesis. In his discussions of such quantifications, Peirce anticipates to a striking degree later developments of statistical testing and confidence interval estimation (Hacking 1980, Mayo 1993, 1996). Since this is not the place to describe his statistical contributions, I move to more modern methods to make the qualitative-quantitative contrast.

6. Quantitative and qualitative induction: significance test reasoning

Quantitative Severity

A statistical significance test illustrates an inductive inference justified by a quantitative severity assessment. The significance test procedure has the following components: (1) a null hypothesis H₀, which is an assertion about the distribution of the sample X = (X₁, …, X_n), a set of random variables, and (2) a function of the sample, d(x), the test statistic, which reflects the difference between the data x = (x₁, …, x_n), and null hypothesis H₀. The observed value of d(X) is written d(x). The larger the value of d(x) the further the outcome is from what is expected under H₀, with respect to the particular question being asked. We can imagine that null hypothesis H₀ is

H₀: there are no increased cancer risks associated with hormone replacement therapy (HRT) in women who have taken them for 10 years.

Let d(x) measure the increased risk of cancer in n women, half of which were randomly assigned to HRT. H₀ asserts, in effect, that it is an error to take as genuine any positive value of d(x)—any observed difference is claimed to be “due to chance”. The test computes (3) the p-value, which is the probability of a difference larger than d(x), under the assumption that H0 is true:

p-value = Prob(d(X) > d(x)); H₀).

If this probability is very small, the data are taken as evidence that

H*: cancer risks are higher in women treated with HRT

The reasoning is a statistical version of modes tollens.

If the hypothesis H₀ is correct then, with high probability, 1- p, the data would not be statistically significant at level p.

x is statistically significant at level p.

Therefore, x is evidence of a discrepancy from H₀, in the direction of an alternative hypothesis H.

(i.e., H* severely passes, where the severity is 1 minus the p-value)[iii]

For example, the results of recent, large, randomized treatment-control studies showing statistically significant increased risks (at the 0.001 level) give strong evidence that HRT, taken for over 5 years, increases the chance of breast cancer, the severity being 0.999. If a particular conclusion is wrong, subsequent severe (or highly powerful) tests will with high probability detect it. In particular, if we are wrong to reject H₀ (and H₀ is actually true), we would find we were rarely able to get so statistically significant a result to recur, and in this way we would discover our original error.

It is true that the observed conformity of the facts to the requirements of the hypothesis may have been fortuitous. But if so, we have only to persist in this same method of research and we shall gradually be brought around to the truth. (7.115)

The correction is not a matter of getting higher and higher probabilities, it is a matter of finding out whether the agreement is fortuitous; whether it is generated about as often as would be expected were the agreement of the chance variety.

[I will post Part 2 tomorrow, then part 3; you can find the rest of section 6 here.]

REFERENCES:

Hacking, I. 1980 “The Theory of Probable Inference: Neyman, Peirce and Braithwaite”, pp. 141-160 in D. H. Mellor (ed.), Science, Belief and Behavior: Essays in Honour of R.B. Braithwaite. Cambridge: Cambridge University Press.

Laudan, L. 1981 Science and Hypothesis: Historical Essays on Scientific Methodology. Dordrecht: D. Reidel.

Levi, I. 1980 “Induction as Self Correcting According to Peirce”, pp. 127-140 in D. H. Mellor (ed.), Science, Belief and Behavior: Essays in Honor of R.B. Braithwaite. Cambridge: Cambridge University Press.

Mayo, D. 1991 “Novel Evidence and Severe Tests”, Philosophy of Science, 58: 523-552.

———- 1993 “The Test of Experiment: C. S. Peirce and E. S. Pearson”, pp. 161-174 in E. C. Moore (ed.), Charles S. Peirce and the Philosophy of Science. Tuscaloosa: University of Alabama Press.

——— 1996 Error and the Growth of Experimental Knowledge, The University of Chicago Press, Chicago.

———–2003 “Severe Testing as a Guide for Inductive Learning”, in H. Kyburg (ed.), Probability Is the Very Guide in Life. Chicago: Open Court Press, pp. 89-117.

———- 2005 “Evidence as Passing Severe Tests: Highly Probed vs. Highly Proved” in P. Achinstein (ed.), Scientific Evidence, Johns Hopkins University Press.

Mayo, D. and Kruse, M. 2001 “Principles of Inference and Their Consequences,” pp. 381-403 in Foundations of Bayesianism, D. Cornfield and J. Williamson (eds.), Dordrecht: Kluwer Academic Publishers.

Mayo, D. and Spanos, A. 2004 “Methodology in Practice: Statistical Misspecification Testing” Philosophy of Science, Vol. II, PSA 2002, pp. 1007-1025.

———- (2006). “Severe Testing as a Basic Concept in a Neyman-Pearson Theory of Induction”, The British Journal of Philosophy of Science 57: 323-357.

Mayo, D. and Cox, D.R. 2006 “The Theory of Statistics as the ‘Frequentist’s’ Theory of Inductive Inference”, Institute of Mathematical Statistics (IMS) Lecture Notes-Monograph Series, Contributions to the Second Lehmann Symposium, 2005.

Neyman, J. and Pearson, E.S. 1933 “On the Problem of the Most Efficient Tests of Statistical Hypotheses”, in Philosophical Transactions of the Royal Society, A: 231, 289-337, as reprinted in J. Neyman and E.S. Pearson (1967), pp. 140-185.

———- 1967 Joint Statistical Papers, Berkeley: University of California Press.

Niiniluoto, I. 1984 Is Science Progressive? Dordrecht: D. Reidel.

Peirce, C. S. Collected Papers: Vols. I-VI, C. Hartshorne and P. Weiss (eds.) (1931-1935). Vols. VII-VIII, A. Burks (ed.) (1958), Cambridge: Harvard University Press.

Popper, K. 1962 Conjectures and Refutations: the Growth of Scientific Knowledge, Basic Books, New York.

Rescher, N.  1978 Peirce’s Philosophy of Science: Critical Studies in His Theory of Induction and Scientific Method, Notre Dame: University of Notre Dame Press.

Please comment on the article here: Error Statistics Philosophy » Statistics

The post Peircean Induction and the Error-Correcting Thesis (Part I) appeared first on All About Statistics.

We recently discussed an article by Isabel Scott and Nicholas Pound entitled, “Menstrual Cycle Phase Does Not Predict Political Conservatism,” in which Scott and Pound definitively shot down some research that was so ridiculous it never even deserved the dignity of being shot down. The trouble is, the original article, “The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle,” had been published in Psychological Science, a journal which is psychology’s version of Science and Nature and PPNAS, a “tabloid” that goes for short sloppy papers with headline-worthy claims. So, Scott and Pound went to the trouble of trying and failing to replicate; as they reported:

We found no evidence of a relationship between estimated cyclical fertility changes and conservatism, and no evidence of an interaction between relationship status and cyclical fertility in determining political attitudes.

The thing that bugged me when Scott and Pound wrote:

Our results are therefore difficult to reconcile with those of Durante et al, particularly since we attempted the analyses using a range of approaches and exclusion criteria, including tests similar to those used by Durante et al, and our results were similar under all of them.

As I wrote earlier: Huh? Why “difficult to reconcile”? The reconciliation seems obvious to me: There’s no evidence of anything going on here. Durante et al. had a small noisy dataset and went all garden-of-forking-paths on it. And they found a statistically significant comparison in one of their interactions. No news here.

Scott and Pound continued:

Lack of statistical power does not seem a likely explanation for the discrepancy between our results and those reported in Durante et al, since even after the most restrictive exclusion criteria were applied, we retained a sample large enough to detect a moderate effect . . .

Again, I feel like they’re missing the elephant in the room: “Lack of statistical power” is exactly what was going on with the original study by Durante et al.: They were estimating tiny effects in a very noisy way. It was a kangaroo situation.

Anyway, I suspect (but am not sure) that Scott and Pound agree with me that Durante et al. were chasing noise and that there’s no problem at all what had been found in that earlier study (bits of statistical significance in the garden of forking paths) with what was found in the replication (no pattern).

And I suspect (but am not sure) that Scott and Pound wrote the way they did in order to make a sort of minimalist argument. Instead of saying that the earlier study is consistent with pure noise, and their replication is also consistent with that, they make a weaker, innocent-sounding statement that the two studies are “difficult reconcile,” leaving the rest of us to read between the lines.

And so here’s my question: When is it appropriate to make a minimalist argument, and when is it appropriate to say what you really think?

My earlier post had a comment from Mb, who wrote:

By not being able to explain the discrepancy with problems of the replication study or “theoretically interesting” measurement differences, they are showing that the non-replication is likely due to low power etc of the original study. It is a rhetorical device to convince those skeptical of the replication.

I replied that this makes sense. I just wonder when such rhetorical devices are a good idea. The question is whether it makes sense to say what you really think, or whether it’s better to understate to make a more bulletproof argument. This has come up occasionally in blog comments: I’ll say XYZ and a comment will say I should’ve just said XY or even just X because that would make my case stronger. My usual reply is that I’m not trying to make a case, I’m just trying to share my understanding of the problem. But I know that’s not the only game.

P.S. Just to clarify: I think strategic communication and honesty are both valid approaches. I’m not saying my no-filter approach is best, nor do I think it’s correct to say that savviness is always the right way to go either. It’s probably good that there are people like me who speak our minds, and people like Scott and Pound who are more careful. (Or of course maybe Scott and Pound don’t agree with me on this at all; I’m just imputing to them my attitude on the scientific questions here.)

The post Being polite vs. saying what we really think appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Being polite vs. saying what we really think appeared first on All About Statistics.

Please comment on the article here: Junk Charts

The post Tufte soundbites appeared first on All About Statistics.

Paul Alper pointed me to this news article about where the bacteria and fungi hang out on airplanes. This is a topic that doesn’t interest me at all, but then I noticed this, at the very end of the article:

Note: A previous version of this article cited the National Science Foundation rather than the National Sanitation Foundation. The post has been updated.

That’s just beautiful. We should definitely try to get a National Sanitation Foundation grant for Stan. I’m sure there are many trash-related applications for which we could make a real difference.

The post Let’s apply for some of that sweet, sweet National Sanitation Foundation funding appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Let’s apply for some of that sweet, sweet National Sanitation Foundation funding appeared first on All About Statistics.

Please comment on the article here: Junk Charts

The post Light entertainment: what is truth? appeared first on All About Statistics.

Peter Dodds, Lewis Mitchell, Andrew Reagan, and Christopher Danforth write:

We introduce, formalize, and explore what we believe are fundamental climatological and seasonal markers: the Summer and Winter Teletherm—the on-average hottest and coldest days of the year. We measure the Teletherms using 25 and 50 year averaging windows for 1218 stations in the contiguous United States and find strong, sometimes dramatic, shifts in Teletherm date, temperature, and extent. For climate change, Teletherm dynamics have profound implications for ecological, agricultural, and social systems, and we observe clear regional differences, as well as accelerations and reversals in the Teletherms.

Of course, the hottest and coldest days of the year are not themselves so important, but I think the idea is to have a convenient one-number summary for each season that allows us to track changes over time.

One reason this is important is because one effect of climate change is to mess up synchrony across interacting species (for example, flowers and bugs, if you’ll forgive my lack of specific biological knowledge).

Dodds et al. put a lot of work into this project. For some reason they don’t want to display simple time series plots of how the teletherms move forward and backward over the decades, but they do have this sort of cool plot showing geographic variation in time trends.

You’ll have to read their article to understand how to interpret this graph. But that’s ok, not every graph can be completely self-contained.

They also have lots more graphs and discussion here.

The post Meet Teletherm, the hot new climate change statistic! appeared first on Statistical Modeling, Causal Inference, and Social Science.

Please comment on the article here: Statistical Modeling, Causal Inference, and Social Science

The post Meet Teletherm, the hot new climate change statistic! appeared first on All About Statistics.

Please comment on the article here: No Hesitations

The post The Econ Ph.D. Placement Network appeared first on All About Statistics.

A friend who teaches courses about statistical regression asked me how to create a graph in SAS that illustrates an important concept: the conditional distribution of the response variable. The basic idea is to draw a scatter plot with a regression line, then overlay several probability distributions along the line, thereby illustrating the model assumptions about the distribution of the response for each value of the explanatory variable.

For example, the image at left illustrates the theoretical assumptions for ordinary linear regression (OLS) with homoscedastic errors. The central line represents the conditional mean of the response. The observed responses are assumed to be of the form y_i = β₀ + β₁x_I + ε_i where ε_i ∼ N(0, σ) for some value of σ. The normal distributions all have the same variance and are centered on the regression line. Pictures like this are used by instructors and textbook authors to illustrate the assumptions of linear regression. You can create similar pictures for variations such as heteroscedastic errors or generalized linear models.

This article outlines how to create the graph in SAS by using the SGPLOT procedure. The data are from a blog post by Arthur Charpentier, "GLM, non-linearity and heteroscedasticity," as are some of the ideas. Charpentier also inspired a related post by Markus Gesmann, "Visualising theoretical distributions of GLMs." Thanks to Randy Tobias and Stephen Mistler for commenting on an early draft of this post.

Fitting and scoring a regression model

Charpentier uses a small data set that contains the stopping distance (in feet) when driving a car at a certain speed (in mph). The OLS regression model assumes that distance (the response) is a linear function of speed and that the observed distances follow the previous formula.

In order to make the program as clear as possible, I have used "X" as the name of the explanatory variable and "Y" as the name of the response variable. I have used macro variables in a few places in order to clarify the logic of the program. However, to prevent the program from appearing too complicated, I did not fully automate the process of generating the graph.

The following statements define the data and use PROC SQL to create macro variables that contain the minimum and maximum value of the explanatory variable. Then a DATA step creates 101 equally spaced points in the interval [min(x), max(x)], which will be used to score the regression model:

/* data and ideas from http://freakonometrics.hypotheses.org/9593 */
/* Stopping distance (ft) for a car traveling at certain speeds (mph) */
data MyData(label="cars data from R");
input x y @@;
logY = log(y);
label x = "Speed (mph)"  y = "Distance (feet)" logY = "log(Distance)";
datalines;
4   2  4 10  7  4  7  22  8 16  9 10 10 18 10 26 10  34 11 17 
11 28 12 14 12 20 12  24 12 28 13 26 13 34 13 34 13  46 14 26 
14 36 14 60 14 80 15  20 15 26 15 54 16 32 16 40 17  32 17 40 
17 50 18 42 18 56 18  76 18 84 19 36 19 46 19 68 20  32 20 48 
20 52 20 56 20 64 22  66 23 54 24 70 24 92 24 93 24 120 25 85 
;
 
/* 1. Create scoring data set */
/* 1a. Put min and max into macro variables */
proc sql noprint;
  select min(x), max(x) into :min_x, :max_x 
  from MyData;
quit;
 
/* 1b. Create scoring data set */
data ScoreX;
do x = &min_x to &max_x by (&max_x-&min_x)/100;  /* min(X) to max(X) */
   output;
end;
run;

The next statements call PROC GENMOD to fit an OLS model. (I could have used PROC GLM or REG for this model, but in a future blog post I want to explore generalized linear models.) The procedure outputs the parameter estimates, as shown, and use the STORE statement to save the OLS model to a SAS item store. The PLM procedure is used to score the model at the points in the ScoreX data set. (For more about PROC PLM, see Tobias and Cai (2010).) The fitted values are saved to the SAS data set Model:

/************************************************/
/* Model 1: linear regression of Y              */
/************************************************/
title "Linear Regression of Y";
/* Create regression model. Save model to item store. Score with PROC PLM. */
proc genmod data=MyData;
  model y = x / dist=normal link=identity;
  ods select ParameterEstimates;
  store work.YModel / label='Normal distribution for Y; identity link';
run;
 
proc plm restore=YModel noprint;
  score data=ScoreX out=Model pred=Fit; /* score model; save fitted values */
run;

The estimate for the intercept is b₀ = -17.58 and the estimate for the linear coefficient is b₁ = 3.93. The Scale value is an MLE estimate for the standard deviation of the error distribution. These estimates will be used to plot a sequence of normal distributions along a fitted response curve that is overlaid on the scatter plot. If you use PROC GLM to fit the model, the RMSE statistic is a (slightly different) estimate for the scale parameter.

Plotting conditional distributions

The next step is to create data for a sequence of normal probability distributions that are spaced along the X axis and have standard deviation σ=15.07. Charpentier plots the distributions in three dimensions, but many authors simply let each distribution "fall on its side" and map the density scale into the coordinates of the data. This requires scaling the density to transform it into the data coordinate system. In the following DATA step, the maximum height of the density is half the distance between the distributions. The distributions are centered on the regression line, and extend three standard deviations in either direction. The PDF function is used to compute the density distribution, centered at the conditional mean, for the response variable.

The following code supports a link function, such as a log link. For this analysis, which uses the identity link, the ILINK macro variable is set to blank. However, for a generalized linear model with a log link, you can set the inverse link function to EXP.

/* specify parameters for model */
%let std = 15.3796;
%let b0 = -17.5791;
%let b1 =   3.9324;
%let ILINK = ;            /* blank for identity link; for log link use EXP */
 
/* create values to plot the assumed response distribution */
data NormalResponse(keep = s t distrib);
/* compute scale factor for height of density distributions */
ds = (&max_x - &min_x) / 6;          /* spacing for 5 distributions */
maxHt = ds / 2;                      /* max ht is 1/2 width */
maxPdf = pdf("Normal", 0, 0, &std);  /* max density for normal distrib */
ScaleFactor = maxHt / maxPDF;        /* scale density into data coords */
/* place distribs horizontally along X scale */
do s = (&min_x+ds) to (&max_x-ds) by ds;
   eta = &b0 + &b1*s;                /* linear predictor */
   pred = &ILINK(eta);               /* apply inverse link function */
   /* compute distribution of Y | X. Assume +/-3 std dev is appropriate */
   do t = pred - 3*&std to pred + 3*&std by 6*&std/50; 
      distrib = s + ScaleFactor * pdf("Normal", t, pred, &std); 
      output;
   end;
end;
run;

The last step is to combine the original data, the regression line, and the probability distributions into a single data set. You can then use PROC SGPLOT to create the graph shown at the top of the article. The data are plotted by using the SCATTER statement, the regression line is overlaid by using the SERIES statement, and the BAND statement is used to plot the sequence of semi-transparent probability distributions along the regression line:

/* merge data, fit, and error distribution information */
data All;  set MyData Model NormalResponse;  run;
 
title "Conditional Distribution of Response in Regression Model";
title2 "y = &b0 + &b1*x + eps, eps ~ N(0, &std)";
title3 "Normal Distribution, Identity Link";
proc sgplot data=All noautolegend;
  scatter x=x y=y;
  series x=x y=Fit;
  band y=t lower=s upper=distrib / group=s
       transparency=0.4 fillattrs=GraphData1;
  xaxis grid;
  yaxis grid;
run;
title;

The program could be shorter if I use the SAS/IML language, but I wanted to make the program available to as many SAS users as possible, so I used only Base SAS and procedures in SAS/STAT. The graph at the beginning of this article illustrates the basic assumptions of ordinary linear regression. Namely, for a given value of X, the OLS model assumes that each observed Y value is an independent random draw from a normal distribution centered at the predicted mean. The homoscedastic assumption is that the width of those probability distributions is constant.

In a future blog post I will draw a similar picture for a generalized linear model that has a log link function.

The post Plot the conditional distribution of the response in a linear regression model appeared first on The DO Loop.

Please comment on the article here: The DO Loop

The post Plot the conditional distribution of the response in a linear regression model appeared first on All About Statistics.

There were a lot of posts, recently, related to those topics, starting with Noah Smith ‘s piece entitled “Economics has a Math Problem” and more recently “Econometrics, Math, and Machine Learning…what?” by Matt Bogard. I don’t have (yet) a clear mind on those issues, but there are still a few thoughts that I wanted to share. I did not really want to, but I’ve been asked, on Twitter, and I thought it might be good to write them down, to clarify some ideas I have, but also (probably, hopefully) to get interesting feedbacks.

About the general context, I am all the more interested by those questions that I usually have them in interviews. I have a PhD in applied mathematics (on statistical models, but to be honest, it was more on extreme value theory and copula, so basically, all I did was more applied probability than statistics). Then I got a position in an Economics departement in France, and then in a Mathematics department, in Canada. Each time people ask me if I am a statitician. Or an economist. Or an actuary…. Furthermore, I’ve been teaching statistics and econometrics based on standard textbooks, but I have always frustrated about (strong) limitations of standard results. Like, what should we do if residuals are not Gaussian, and if we have a small number of observations. Like, what should be do if the relationship between the dependent variable and some covariates is clearly not linear… Most of my courses had a focus on computational aspects, yes. And more recently, I have been doing also some training for actuaries on “data science” (an important buzzword in the industry, but let’s call that “freakonometrics“), and a lot of them expected an emphasis on machine learning techniques. They had the feeling that consultant selling machine learning techniques could outperform their actuarial models, usually based on standard econometric techniques. And they wanted to understand what was really behind those words like ‘random forrest‘, ‘SVM‘, ‘neural networks‘, ‘gradient boosting‘, etc. So I have been working on those topics recently, even if I still don’t claim any kind of expertise. But I am curious, and I have been reading a lot of things, asking a lot of questions to colleagues, so maybe I can share some thoughts here.

• Machine learning vs. mathematical statistics

In machine learning, we have a dataset, i.e. a collection of observations , and some loss function, . And the goal is to get a function , defined on the same space as the ‘s, taking values in the same space as the ‘s, which could be a good predictor for the ‘s in the sense that we should solve

for some functional space  (which somehow describes the kind of predictor you’re looking for, like linear functions, polynomial functions, stepwise functions, etc). A popular case is the quadratic loss function, but we’ll get back on that point soon… And the problem should probably be formalized differently, because the function should be defined on a training sample, but the loss should be minimal on a testing sample.  Anyway, in machine learning, we want to minimize a loss, based on data. Everything is about data, and a loss function.

In mathematical statistics, we have a dataset, i.e. a collection of observations ,. The first step is to assume that there is a probability space  such that observations can be seen as realisation of i.i.d. random variables . More specifically, here, we assume that conditionnally on ‘s, ‘s are i.i.d. with distribution . Then we solve something like

which is a maximum likelihood problem, i.e.

Here we assume that there is simple relationship between parameter  in the parametric family, and the mean, , like, for some (known) functional . Then, another assumption is that we believe that the mean can be a good ‘predictor’. So here, we set

A popular example is probably the Gaussian distribution, where the mean is the first component of the parameter. But here again, we’ll get back on that point soon.

To summarize, the main difference I see here is that machine learning starts in a non-probabilistic world. There is no assumption of a random sample, of a possible causal relationship (implicity admitted with the conditional distribution mentioned above). That’s why in mathematical statistics we can get confidence intervals (it is possible to say that with some probability, the predicted values is bounded by some quantities since there is, underneath, a probabilistic model). And we should not get a confidence interval in a machine learning context. But that’s possible to get a probabilitic interpretation of most techniques, as explained by Kevin Murphy in Machine Learning: a Probabilistic Perspective.

• The paradox of econometrics

The paradox of econometrics is that it is taught as a machine learning problem. All econometric course starts with Gauss-Markov theorem. The OLS estimator is defined based on a (quadratic) loss function, not based on some distribution on the residuals. We want to solve the following problem, that is the starting point of econometrics,

This is a machine learning formulation, with a quadratic loss function. We like that formulation because of the probabilistic interpretation we can get,

or, if we consider the empirical formulation

OLS are related with the expected value. Which – again – is a natural predictor. But that is not a statistical way of defining econometrics (the way I described in the previous section). If we want to do it properly, we should go for a GLM approach. To be more specific, we assume that we have i.i.d. observations (i.i.d. given the covariates), and that

Then we end_up solving the same problem as before, namely

but here, we have a statistical definition of the econometric problem. Of course the two are related, since from our maximum-likelihood approach is translated, from a computational point of view, as solving the first order condition of an optimization problem. From a computational point of view, there is no probabilitistic distribution: we can use a Poisson regression even when the dependent variable is not an integer! So obviously, econometric models can not be described only as mathematical statistics ones.

• On nonparametric econometrics, or the meaning of ‘linear’

While I was giving my econometrics class a few years ago, I usually starting with a simple example. In economc theory, one can find “relationships” between agregated variables. For instance, with Philips curve, we want to visualize the relationship between inflation and unemployment. An economic theory might tell you that if unemployment decreases, inflation increases. From that economic theory, we might try to fit an econometric model, describe that “relationship” But usually, econometric models are based on a linear relationship. This is no a priori reason to have a linear relationship. I mean, even if you look at data, I challenge you to find something linear here

The Cowles Commission, which initiated Econometrics (by founding the Econometric Society, and the prestigious journal Econometrica) had the postulate that an econometric model should be based on an economic model. That’s how you get SEM (Structural Equation Model) like Klein model, in the 1950’s,

That is a standard (linear) econometric model, with parameters  and , and some linear relationship amoung them. Nonparametric econometric models started, assuming that models might be non linear.

I my econometric classes, I was introducing splines, but also local regression. Which is very natural when you think about it : if the goal is to get a good estimation for , shouldn’t you look in the neighbourhood of . We do not want necessarily a good global model, but a model good in that neighbourhood.

With nonparametric models, we start to have numerical problems, since problems are not as simple as linear ones. So the first goal is to get an efficient algorithm to solve it. So here we start to have connexions with machine learning. The main difference I see is simple. In econometrics, seen as a mathematical statistics problem, we seek asymptotic results, nice probabilistic interpretations. In econometrics, seen as a machine learning problem, we focus more on the algorithm. We do not care about the output or the interpretability, we want a good algorithm. See for instance gradient boosting techniques against spline regression (in a recent post)

The blue line is a simple (linear) spline regression model, and the red line is a boosted spline algorithm (it is a stepwise procedure). The blue line is simple, easy to understand. The red line, after 200 iterations is a sum of 200 functions. It is much more difficult to interpret. But who cares ?

• On meta (or tuning) parameters

The other difference between machine learning and statistics (in the context of nonparametric models) is the choice of the tuning parameters (or meta-parameters) that show-up. For instance the bandwith in the context of kernel regression (which is the size of the neighborhood in local regression) or kernel density estimation.

In traditional econometrics, we use plug-in methods. For instance Silverman’s rule of thumb

This was obtained under the assumption that observations were normaly distributed. A lot of rule-of-thumb techniques have been obtained under the assumption of a normal distribution. In machine learning, again, we just use data, we do not need a probabilistic model. It is more natural to use cross-validation techniques. That is the standard way of getting that optimal bandwidth parameter.

• Boostrap

Bootstrap is an interesting tool that appears both in statistics and in machine learning. I am a real fan of bootstrap techniques. I usually see it as a non-parametric monte carlo algorithm. We use the inverse transform sampling technique, with the empirical cumulative distribution function. Use  where

Monte carlo techniques can be used in a probabilistic space, so it is natural to use them in mathematical statistics. I use them a lot in my courses when I define statistical tests : tests are based on statistics that have a known distributions when  is true. But “known” is a very general word. I mean, if we can draw from that distribution, that’s enough, right. So monte carlo techniques are important tools in mathematical statistics. An necessary in econometrics. In most textbooks, results in econometrics either rely on the Gaussian assumption (that is hardly satisfied in real life), or on asymptotic results (and then you need a large number of observations). With a small number of observations, boostrap is necessary.

But bootstrap is also natural in machine learning. For instance bagging. It is claimed to be a machine learning technique, “bagging, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms” (as explained on Wikipedia). Actually, that is what we use if we use bootstrap in econometrics: we generate possible samples by resampling, and then you aggregate the predictions. Simple isn’t it.

• To go further

I know, all of that is way too simplistic. But clearly, machine learning, statistics and econometrics will live together in the future, and there will be a lot of overlaps. They will learn from each other. Probabilistic interpretation of machine learning tools will be more important, but so will be computational aspects of statistics. But it started a long time ago, with nonparametric approaches, and the branch of computational statistics. And a lot of interesting posts and articles have been published on those topics, starting with “Statistical Modeling: The Two Cultures” by Leo Breiman.

Please comment on the article here: Freakonometrics » Statistics

The post Some thoughts on Economics, Mathematics, Econometrics, Statistics, Machine Learning, etc appeared first on All About Statistics.

A combinatorics Le Monde mathematical puzzle:

How many distinct integers between 0 and 16 can one pick so that all positive differences are distinct?

If k is the number of distinct integers, the number of positive differences is

1+2+…+(k-1) = k(k-1)/2,

which cannot exceed 16, because it is a subset of {1,2,…,16}, meaning k cannot exceed 6 if all differences are distinct. From there, picking k integers at random makes it easy to check for the condition:

k=6
N=16
x=sort(sample(0:N,k-1))
y=outer(x[-1],x[-k],"-")
while (max(duplicated(y[!upper.tri(y)]))==1){
  x=sort(sample(0:N,k-1))
  y=outer(x[-1],x[-k],"-")}

which quickly returns for k=5

> x
[1] 0 1 7 12 15

as a solution. And is still running for k=6, meaning there is apparently no solution for k=6. Now, reading the puzzle solution of Le Monde today, on September 09, I discovered that the authors proposed a sequence of length 7, (0,1,2,4,5,7,11,16), which does not work since 1-0=2-1… and proved that 8 is an impossible value by quite a convoluted argument. Did I misread again?!

In the earlier version of the R code posted today, I used

...y[lower.tri]...

which does not include the diagonal, instead of the proper

...y[!upper.tri(y)]..

a mistake that led to a wrong solution for k=6, as pointed out by Stephan.

Please comment on the article here: Xi'an's Og » R

The post Le Monde puzzle [#928] appeared first on All About Statistics.