Roy Baumeister, a social and personality psychologist of great standing, recently commented on the replication debate in psychology and on the much-discussed non-replications of some of his own studies.1 Baumeister is worried about the current push for more replications and more statistical power. He foresees a social psychology that is obsessed with large samples and precise procedures, at the detriment of the kind of small-scale creative work that social psychologists have excelled at and that has brought the field so much success.
When Baumeister’s comment appeared online it was greeted with howls of derision. One passage in particular drew sarcastic remarks. Looking back at his early days in research, Baumeister writes: “Getting a significant result with n = 10 often required having an intuitive flair for how to set up the most conducive situation and produce a highly impactful procedure. Flair, intuition, and related skills matter much less with n = 50.”2
Baumeister adds that in his own lab, not all his graduate students and postdocs are able to replicate his results. Some lack the required flair and must find another job. In the new social psychology, however, these “bad experimenters” (as he calls them) might build a career publishing their failed attempts to replicate the work of other, more competent researchers.3
Among the critics, ‘flair’ has since become a byword for the self-delusion among some of the old guard in social psychology. There is indeed much that I disagree with in Baumeister’s comment, but one thing that I don’t have a problem with is the notion of flair in science.
I think there’s nothing wrong with acknowledging that flair has a role in science, if by flair Baumeister means that doing research requires expertise, and that some of this expertise may be implicit – not written down in textbooks or in the method section of reports, not explicitly taught. In fact, this was one of the findings of sociologists of science when they began to study the work of laboratory scientists in the 1970s: scientists do not always work by the book, much of lab work consists of ‘tinkering’, and the success or failure of an experiment often turns out to depend on little things that the experimenter doesn’t realise she does differently than other experimenters.
Jacob Jolij, a cognitive neuroscientist at my department, agrees that flair matters in science. Regarding his own work with Transcranial Magnetic Stimulation, for example, he writes: “There’s a lot of tacit knowledge required with regard to subject handling, coil positioning, and stimulus design to get good masking effects.”4 The same is probably true of ‘social’ manipulations, he adds, and therefore we should not be too hasty to accept a negative result from a replication. Lack of competence or ‘flair’ might really play a role in the failure.
Like Jacob, I don’t think the concept of flair as such is problematic. The problem is not that scientists like Baumeister take a seemingly frivolous concept like flair seriously, the problem is that they don’t take it seriously enough.
First of all, it is usually trotted out only when replications fail, as an ad hoc explanation, or rather as a way of blaming the replicating researcher for the failure. Of course, it is possible that the replicator is to blame, but that’s a hypothesis that needs to be tested by investigating what the missing bit of expertise was. And why should the replicator be the (only) one to shoulder that responsibility? If he or she is an otherwise competent researcher (a category with fuzzy borders, of course), his or her replication failure may well point to an unknown aspect of the phenomenon under study. Such things should interest the original researcher just as much as the replicators. Thus, failed replications are potentially of great value: they may point to the role of tacit expertise in producing a certain experimental effect.
More generally, if tacit expertise is an important part of the skills of scientists, we should study it to get to know it better. Together with Eric Rietzschel and Jonna Brenninkmeijer I have started a research project asking psychologists which skills and practices they find important in doing research, but are not found in the handbooks or in the method sections. From our first few pilot interviews it is already clear that researchers have no trouble coming up with examples of such ‘unofficial’ forms of expertise.
There are more questions to be answered. For one thing: how are these skills transferred, and can that process be improved? How should we train our students? Should flair be in the methods handbooks, for example? That, in turn, raises the question whether this tacit expertise can always be made explicit. Baumeister is not clear on this point, but I get the impression he believes that it cannot.
That would raise all sorts of thorny issues, for example with regard to replication – it would mean you can never fully spell out how to replicate an experiment. I won’t go into that here, but it underlines the fact that we need to do better than Baumeister’s handwaving reference to “flair intuition and related skills”, or a facile dismissal of flair as unscientific. We should take the role of tacit expertise in psychology and science in general more seriously.
[This post was earlier published on mindwise]
1 Baumeister, R. (2016). Charting the future of social psychology on stormy seas: Winners, losers, and recommendations. Journal of Experimental Social Psychology, 66, 153–158.
2 Baumeister (2016, p.156)
3 Baumeister (2016, p.156)