I have a few pet peeves. One of them is how “peer review” is perceived by far too many people as the gold standard certification of scientific authority. Any academic who’s been through the peer review process many times (as I have) knows that the process is full of potholes and is sometimes subverted by unethical behavior on the part of editors and reviewers.
The reason I bring this up is because of a brewing scandal in my own discipline, economics. There has been online discussion about this for over a month, but I’ve delayed this post both because I’ve been traveling too much and because I was hoping for a resolution before I wrote anything down. But as a junior economist recently told me: “The relative silence by senior economists regarding the editorial handling of this paper has been deafening.” So here it goes.
The facts seem easy to summarize.
- Two young economists, Petra Persson and Maya Rossin-Slater, wrote a paper entitled “Family Ruptures, Stress, and the Mental Health of the Next Generation.” They submitted the paper for publication to the American Economic Review (AER), the premier publication of the American Economic Association.
- The paper was handled by AER co-editor Hilary Hoynes, an economist at UC Berkeley. All of the available information indicates that the paper went through normal reviewing procedures. Hoynes sent out the paper to four referees that she specifically selected to give her advice on whether the paper was sufficiently important and original to be published in the AER.
- After the referees wrote their reviews and the authors addressed the various issues raised by the reviewers, Hoynes accepted the paper for publication.
Let me emphasize an important point for readers from outside the profession: Landing a paper in the AER can be a career-changing moment for a young economist. The probability of acceptance has been about 8 percent in the past few years, so that the professional signal given by a paper in the AER has a very high value in terms of job prospects, grants, income, and prestige.
And this is where things begin to get interesting. Over a month ago, an anonymous post appeared at Economics Job Market Rumors (EJMR), a popular internet forum frequented (I am guessing) by many young economists. This post noted that Family Ruptures resembled a paper already published in the medical journal Psychosomatic Medicine (the two papers apparently used much of the same data to address similar questions). I am not familiar enough with the medical literature to make an informed evaluation on the originality or methodological contribution of Family Ruptures, but that’s not the source of the scandal anyway.
The initial EJMR post created a bit of a firestorm, attracting many additional posts (almost 4,400 as of this writing). Some of those posts looked carefully at the earlier literature. Others began to focus on the process by which the paper got into the AER and on how the elite economists who run that journal reacted once certain facts became known.
And here’s where things get really interesting. It turns out that Hilary Hoynes, the co-editor at the AER, happens to be a current coauthor of one of the junior economists who wrote the Family Ruptures paper. (Page 12 of her CV dated October 13, 2015, indicates Hoynes was working with one of the coauthors while the review process was ongoing). This is a big no-no. The editor, by selectively picking which referees will review the paper, has a lot of influence over how the “peer review process” turns out. A good editor has a feel for how particular economists will react to particular kinds of work, so that by choosing the right reviewers the editor can “nudge” the final assessment in a particular direction. The conflict of interest is so large and so obvious that the AER has written guidelines about this:
There are several rules that affect assignment of manuscripts. Coeditors are generally not assigned manuscripts authored by an individual at his or her institution, by an individual with whom the Coeditor has been a recent coauthor, by an individual who has a close professional or personal relationship with the Coeditor, or by an individual who has served as a graduate student advisor or advisee of the Coeditor.
To make matters worse, after the barrage of posts at EJMR pointed out that there existed at least one paper in the medical literature that resembled the now-forthcoming AER paper, Hoynes (and perhaps other AER editors) attempted to resolve the problem by allowing the authors to add footnotes and a new section to the Family Ruptures paper. These post-acceptance revisions were apparently added sequentially in different rounds. Despite the additions and despite the new information, the paper was never again sent to the four referees to determine if the nature of the contribution had changed in light of the new information. Instead, the to-be-published version of Family Ruptures contains added-on passages with “Consumer Warnings”-like notes that stick out like a sore thumb.
(As an aside, EJMR has been referred to as a cesspool by some commentators. Retraction Watch published an article about the brouhaha last month, and quotes Hoynes dismissing EJMR because it is “unmoderated” and “not a legitimate source of information.” Unfortunately, she does not address how this unmoderated forum of illegitimate nonsense led to revisions in an already-accepted paper at the AER. Retraction Watch also quotes Janet Currie, a Princeton economist who was the doctoral advisor of one of the young coauthors. Currie takes an even easier approach to dismiss EJMR: sexism. I personally find the forum refreshing. There’s still hope for mankind when many of the posts written by a bunch of over-educated young social scientists illustrate a throwing off of the shackles of political correctness and reflect mundane concerns that more normal human beings share: prestige, sex, money, landing a job, sex, professional misconduct, gossip, sex, and putting down “reg monkeys,” a subspecies of economists that cares little about conceptual issues and lives simply to run regressions.)
I want to make sure that readers do not get the wrong impression. I feel quite a bit of empathy for the young authors of Family Ruptures. They were ill-served by editorial decisions. Had Hoynes recused herself from shepherding the paper through the process at the AER, the worst that could have happened is that the paper would have been rejected; the two economists would have resubmitted the paper elsewhere; the paper would have eventually found a friendlier reception at a lower-tier journal; and the paper would have been published and henceforth forgotten, a fate shared by practically all academic papers. From the authors’ perspective, I think that outcome is far preferable to the situation today.
Dan Hamermesh, who I have known since my graduate school days, has written a number of very interesting and widely read papers on professional etiquette. I emailed him a few days ago to find out what his thinking was on this matter. He gave me permission to quote his reply in full:
1. Hoynes should have recused herself ab initio. I don’t know if the AER has written rules for Co-Editors, but it shouldn’t need them on something that seems so blatant. It’s obvious to me that editors should have a Caesar’s wife attitude, so that if there might even be the slightest hint of favoritism, they should recuse themselves. She failed on this.
2. Having discovered the problem with what is essentially a dual publication, Hoynes should have bowed out and asked the Editor-in-Chief to handle everything henceforth, including the decision whether or not to reject the paper in light of the new information. She apparently failed on this one too.
3. Given these two failures, what to do?? I’m not sure what the answer is on that, given the implicit promise of an acceptance to these young authors.
These are sensible points. I would add another one. What’s the bigger scandal? The fact that Hoynes did not recuse herself from handling the paper initially? Or the fact that the AER’s leadership has not really addressed the ethical lapse involved? As is much too common these days, when important people do something wrong, heads no longer roll. Would anyone be surprised if any day now the people involved issue a generic non-apology apology telling everyone that it’s time to move on? As someone else famously said: What difference, at this point, does it make?
Let me get back to the pet peeve that motivated this long rant. Next time you hear about “professional consensus in peer-reviewed research,” do as I do: Roll your eyes. Who knows what went into the making of that particular sausage? And this warning applies ten-fold for peer-reviewed research in any politically charged subject.
Having gone through the peer-review process countless times (and having been an editor myself) really opened my eyes to the role that human frailties play in the advancement of science. One of my early papers on immigration, “Self-Selection and the Earnings of Immigrants,” was published in the American Economic Review in 1987. One of the referees who reviewed that paper wanted to make sure that the study never saw the light of day and wrote the most negative review imaginable. This is how that report began:
This is a pretentious paper with a “self-selected” recollection of the literature. The wealth of literature on migration models, migration self-selection and migrant adjustment are (incorrectly) dismissed as not being based on income-maximizing principles. The author tells us that this model will set it all straight!
And it went downhill from there. Fortunately for me, another reviewer wrote an extremely positive assessment and the AER’s editor at the time sided with the other reviewer. That article is now my second-most cited paper and has 2,660 citations in Google Scholar.
The point is that many human emotions, including nepotism, professional jealousies, methodological disagreements, and ideological biases go into the peer review process. It would be refreshing if we interpreted the “peer-reviewed” sign of approval as the flawed signal that it is, particularly in areas where there seems to be a larger narrative that must be served. The peer-review process may well be the worst way of advancing scientific knowledge–except for all the others.
Disclosures: I don’t believe I have ever met either of the young authors of the Family Ruptures paper. I have known Hilary Hoynes for a long time. Both my memory and a quick check of my records tell me that she has never been the editor of any of the papers I have submitted to any journal. The facts stated in this post summarize what I have learned from the online debate and from exchanges with some economists. I will gladly correct any factual errors brought to my attention.
