Thursday, May 7, 2020

Covid and economics publishing

The pandemic is dramatically illustrating one area in which the epidemiologists are beating the economists about 100-1: publishing. Scientific publications are reviewed and posted in days, contributing in real time to the policy debate.

Economists are writing papers in a similar flurry. They are writing really good, thoughtful, well done papers that are useful to the policy debate. See the NBER website for example, or SSRN. See my last post and previous one for several great examples.

But when will these papers be peer reviewed? Where will they be published?


Not every new paper is right, and the review and comment process improves economic papers a lot.

Economics publishing is stuck in a leisurely 19th century process. It typically takes several years from completing a project to its online publication, and often another year to print. And "completing" already includes  pre-submission vetting through conferences, correspondence, seminars and working papers. I often wait a year to hear the first review of my papers. Even the best journals try for 6 weeks.  Several rounds of revisions are almost universal. It is typical to be rejected, shop the paper to several journals who reject it (after 6 moths to a year), hear the same comments from different referees, until you find an editor who likes the paper. (Almost all of my papers have been rejected at least once. My record is 7 times.)

As a result, academic journals long ago ceased to be avenues for communication. They are archives, and certifiers for tenure committees that don't trust themselves to read papers. Individual websites, and working paper series such as NBER and SSRN have taken over the communication function. But not every paper is right, and that means little of our communication takes place after any peer review.

This state of affairs has a supply effect. I cheer my colleagues writing these papers. But where in heaven's name are they going to publish the papers? When, a year from now, the first reviews come back, will the editors still find the papers  interesting and relevant? How will you fill in the increasingly mandatory section on "why this is important" and "policy implications" a year, or two years from now? The referees will demand a literature review. How will you cover the hundreds of papers sure to come out in the next few months? (Economics does not demand review only up to the date the working paper is first posted.)

Fe example,  my last post could well become a paper. It needs a lot of work -- refining the model, exploring economic costs and benefits, modeling the externality or heterogeneity helps, (surely yes), comparing it with data, and so forth. But should I put in that work, at least a month off of doing everything else, and then a further two to three months of revisions, submissions, resubmissions, and so forth for a period of years? No thank you.

My model, like most of these papers, is not a deep methodological improvement. It is a where-are-we-now paper, and a quantified attempt to peer through the mist of the next few months. Where-are-we-now papers are useful! They are 99% of academic research really. If there was a chance of publishing it the way scientists do, I might well do so. But submit it in July and still be waiting in May 2021 to hear a first review? Face a good bet that the editor thinks this is interesting economic history, but not a methodological or factual development of enduring interest? No thanks.

This is not news. There is a lot of soul-searching going on in our journals about how to become more relevant. Economics articles are quantitative essays more than scientific reports, so they often need a bit more digestion. Peer review itself is imperfect -- there is lots of nitpicking but often basic mistakes go unnoticed. (I've long been a fan of open reviewing to broaden the base of input on papers. Often a conference discussant or other reader will know a lot about a paper, but the journal editor and selected referees don't know about it. Universally, editors ask for new referees thus wasting the efforts of the often dozens of people who have reviewed the paper before. The idea is going nowhere.) Economists believe in markets, but not for papers. Why do we not figure out a market for submissions so that papers can get better matched to journals to start with, rather than one by one embark on a 6 month process and move from rejection to rejection. Why not simultaneous submissions?

But the I hope the model of the scientists inspires our journal editors to action.

Update

Three hours after posting, I find the market has worked. CEPR is launching a real time journal on Covid economics.  Tim Taylor coverage. (HT marginal revolution.) From Tim,
The first issue of the journal, with six papers, appeared April 5. The 14th issue (not a typo) of the journal appeared today, May 6, with these eight papers:  
OPTIMAL LOCKDOWN
Fernando Alvarez, David Argente and Francesco Lippi 
POLICY INCENTIVES
Roberto Chang and Andrés Velasco 
MISSING EMERGENCIES
Jorge Alé-Chilet, Juan Pablo Atal and Patricio Domínguez 
HEALTH VS. WEALTH?
Peter Zhixian Lin and Christopher M. Meissner 
MARKETS GET COVID
Mariano Massimiliano Croce, Paolo Farroni and Isabella Wolfskeil 
HEALTH VS. GDP: NO TRADE-OFF
Sangmin Aum, Sang Yoon (Tim) Lee and Yongseok Shin 
SOCIAL DISTANCING AROUND THE WORLD
Gonzalo Castex, Evgenia Dechter and Miguel Lorca 
WHO CAN WORK AT HOME?
Isaure Delaporte and Werner Peña
Academic incentives remain though. The online journals seem to have largely failed. Will hiring, tenure and salary review committees, deans and provosts, view these as publications or as op-eds and blogs?

Even in economics, barriers slowness and inefficiency that we tolerate in normal times is unconscionable in a crisis.

Update 2: Thanks to a comment below, here is the first published paper on Covid-19. Maybe I will be wrong. That would be great. Can journals increase speed and not decrease quality? Yes, but how? In WWII the P-51 was produced in 102 days. But a lot of terrible planes and other projects got built too.

Update 3: A thoughtful twitter stream from Greg Kaplan JPE editor on how to handle Covid-19 papers

2 comments:

  1. When will the peer review be all public at least? I've been very pleased to see recently open review does exist and is used by some.

    Where is that econ paper showing peer review is the best (or even at all good) mechanism for whatever is its purpose?
    If it's official career decisions, why not do request to investigate a person's research to a random sample of ppl in the field, as that's pretty much what pr is doing anyway except one paper at a time. Could be as slow as pr, but at least it would be lag for career stuff which isn't fast anyway, not for research publication.
    If it's research vetting: open to public rating and comment system? open q re who can comment and what weight they get and whether anonymous etc.

    I'm curious about the latter and similar ideas in particular because it also addresses another q which JC doesn't raise: accessibility and usability by research consumers. For econ research in particular the audience is enormous.
    The fact that wrong papers are usually not retracted and it's (as far as I know) impossible for external consumer to figure out whether smth is well-known to be wrong is a disgrace. Given the amount of papers published, I'd say it's crucial for a research consumer to have reliable ratings/know what's considered good/novel/important/most relevant etc by experts. Yet there is none of that, especially for current stuff? Longer term journal stamps of approval are kinda useful, if less so than citation count. Shorter-term there is only authors reputation? And informal mentions on blogs and twitter which are unsystematic and relatively hard to quickly dig up?



    Isn't this stuff what those progress studies ppl supposed to be doing? Maybe somebody knowledgeable can link to their tentative answers on this or related qs.

    ReplyDelete
  2. Maybe I'm drinking the Kool-aid, but I think the slow publication process in econ is part of why quality improves through peer review (you want a 2-week review? You get what you pay for). In contrast, peer review in medicine and natural science is superficial (I've refereed for them a little) and it seems to only detects obvious mistakes. ''Peer-reviewed'' articles in medicine and natural science are like econ research-center working papers (obviously wrong papers are not accepted, but everything else is OK). If we dramatically speed up the publication process, we lose quality (how much?). Anyway, the best outcome is journal competition: let journals experiment! If speed is better than quality, competition will reveal that.

    ReplyDelete

Comments are welcome. Keep it short, polite, and on topic.

Thanks to a few abusers I am now moderating comments. I welcome thoughtful disagreement. I will block comments with insulting or abusive language. I'm also blocking totally inane comments. Try to make some sense. I am much more likely to allow critical comments if you have the honesty and courage to use your real name.