Joel Bernstein writes:
I just finished reading your oped article on reproducibility in science. As an experimental scientist – more precisely a chemical crystallographer – I have had to deal with this kind of situation a number of times, and at least two examples may serve as the possible exceptions to your rules.
One of the beauties of x-ray crystallography is the internal self-consistent confirmation of the result of an x-ray crystal structure determination. X-ray crystal structure determinations are today used (increasing required by journals) as essentially absolute proof of the molecular structure of a new compound. With the result of a determination there is no doubt that the crystal from which the data were obtained contained molecules with the composition and structure that can be represented by a three dimensional ball and stick model. There occasionally are some problems with measurements and refinements of structures, but they are a very, very small fraction of the mass of crystal structures (now nearing one million in the Cambridge Structural Database, the archive for such results) and those problems are almost always readily recognizable by a skilled practitioner.
Having said that, in my own experience now approaching 50 years I have had two situations in which a chemical reaction yielded a totally unexpected result with the production of one single crystal. In both cases the structure of the single crystal was not only unexpected but the structures were polymorphs of materials for which other, totally difference structures were already known. Thus reactions “gone awry” yielded unpredicted and unexpected results, and the veracity of those results is essentially incontestable, and has not been challenged in any way by referees of the resulting publication.
However – and here is where I come to the point of your oped piece – while I have total confidence in the results of those experiments I do have serious doubts about our ability to reproduce the experiments that led to the formation of those crystals. In the interest of full disclosure we have admitted this “failing” in our papers, but that does not detract from the significance of the results of the crystal structure determination. Thus, there may be cases, at least in chemistry, where the conditions of the experiment may be difficult – but certainly not impossible! – to reproduce, but the result is still valid.
Bernstein adds:
I am reviewing some old work and just recalled a 1972 paper we wrote on a crystal structure determination of one single crystal that took an entire year to grow (while the chemist was away on sabbatical). Until then the material had resisted all attempts to crystallize it. I am attaching a copy of that paper (on muconic acid). Our best example is the second polymorph of tetrathiafulvalene, which we published in 1994. I am attaching reprint of that paper as well. The third example is ammonium hydrogen succinate which actually is in press, so I don’t have a reprint to send you (yet!).
I don’t know anything about crystallography except that James Watson didn’t think his colleague had good fashion sense, but I can respect the general idea of hard-to-replicate results, as these arise all the time in observational social science, where you can’t just spin the wheel and get data on N more elections, or recessions, or wars, or whatever.