Thursday, May 7, 2020

Covid and economics publishing

The pandemic is dramatically illustrating one area in which the epidemiologists are beating the economists about 100-1: publishing. Scientific publications are reviewed and posted in days, contributing in real time to the policy debate.

Economists are writing papers in a similar flurry. They are writing really good, thoughtful, well done papers that are useful to the policy debate. See the NBER website for example, or SSRN. See my last post and previous one for several great examples.

But when will these papers be peer reviewed? Where will they be published?


Not every new paper is right, and the review and comment process improves economic papers a lot.

Economics publishing is stuck in a leisurely 19th century process. It typically takes several years from completing a project to its online publication, and often another year to print. And "completing" already includes  pre-submission vetting through conferences, correspondence, seminars and working papers. I often wait a year to hear the first review of my papers. Even the best journals try for 6 weeks.  Several rounds of revisions are almost universal. It is typical to be rejected, shop the paper to several journals who reject it (after 6 moths to a year), hear the same comments from different referees, until you find an editor who likes the paper. (Almost all of my papers have been rejected at least once. My record is 7 times.)

As a result, academic journals long ago ceased to be avenues for communication. They are archives, and certifiers for tenure committees that don't trust themselves to read papers. Individual websites, and working paper series such as NBER and SSRN have taken over the communication function. But not every paper is right, and that means little of our communication takes place after any peer review.

This state of affairs has a supply effect. I cheer my colleagues writing these papers. But where in heaven's name are they going to publish the papers? When, a year from now, the first reviews come back, will the editors still find the papers  interesting and relevant? How will you fill in the increasingly mandatory section on "why this is important" and "policy implications" a year, or two years from now? The referees will demand a literature review. How will you cover the hundreds of papers sure to come out in the next few months? (Economics does not demand review only up to the date the working paper is first posted.)

Fe example,  my last post could well become a paper. It needs a lot of work -- refining the model, exploring economic costs and benefits, modeling the externality or heterogeneity helps, (surely yes), comparing it with data, and so forth. But should I put in that work, at least a month off of doing everything else, and then a further two to three months of revisions, submissions, resubmissions, and so forth for a period of years? No thank you.

My model, like most of these papers, is not a deep methodological improvement. It is a where-are-we-now paper, and a quantified attempt to peer through the mist of the next few months. Where-are-we-now papers are useful! They are 99% of academic research really. If there was a chance of publishing it the way scientists do, I might well do so. But submit it in July and still be waiting in May 2021 to hear a first review? Face a good bet that the editor thinks this is interesting economic history, but not a methodological or factual development of enduring interest? No thanks.

This is not news. There is a lot of soul-searching going on in our journals about how to become more relevant. Economics articles are quantitative essays more than scientific reports, so they often need a bit more digestion. Peer review itself is imperfect -- there is lots of nitpicking but often basic mistakes go unnoticed. (I've long been a fan of open reviewing to broaden the base of input on papers. Often a conference discussant or other reader will know a lot about a paper, but the journal editor and selected referees don't know about it. Universally, editors ask for new referees thus wasting the efforts of the often dozens of people who have reviewed the paper before. The idea is going nowhere.) Economists believe in markets, but not for papers. Why do we not figure out a market for submissions so that papers can get better matched to journals to start with, rather than one by one embark on a 6 month process and move from rejection to rejection. Why not simultaneous submissions?

But the I hope the model of the scientists inspires our journal editors to action.

Update

Three hours after posting, I find the market has worked. CEPR is launching a real time journal on Covid economics.  Tim Taylor coverage. (HT marginal revolution.) From Tim,
The first issue of the journal, with six papers, appeared April 5. The 14th issue (not a typo) of the journal appeared today, May 6, with these eight papers:  
OPTIMAL LOCKDOWN
Fernando Alvarez, David Argente and Francesco Lippi 
POLICY INCENTIVES
Roberto Chang and Andrés Velasco 
MISSING EMERGENCIES
Jorge Alé-Chilet, Juan Pablo Atal and Patricio Domínguez 
HEALTH VS. WEALTH?
Peter Zhixian Lin and Christopher M. Meissner 
MARKETS GET COVID
Mariano Massimiliano Croce, Paolo Farroni and Isabella Wolfskeil 
HEALTH VS. GDP: NO TRADE-OFF
Sangmin Aum, Sang Yoon (Tim) Lee and Yongseok Shin 
SOCIAL DISTANCING AROUND THE WORLD
Gonzalo Castex, Evgenia Dechter and Miguel Lorca 
WHO CAN WORK AT HOME?
Isaure Delaporte and Werner Peña
Academic incentives remain though. The online journals seem to have largely failed. Will hiring, tenure and salary review committees, deans and provosts, view these as publications or as op-eds and blogs?

Even in economics, barriers slowness and inefficiency that we tolerate in normal times is unconscionable in a crisis.

Update 2: Thanks to a comment below, here is the first published paper on Covid-19. Maybe I will be wrong. That would be great. Can journals increase speed and not decrease quality? Yes, but how? In WWII the P-51 was produced in 102 days. But a lot of terrible planes and other projects got built too.

Update 3: A thoughtful twitter stream from Greg Kaplan JPE editor on how to handle Covid-19 papers

4 comments:

  1. When will the peer review be all public at least? I've been very pleased to see recently open review does exist and is used by some.

    Where is that econ paper showing peer review is the best (or even at all good) mechanism for whatever is its purpose?
    If it's official career decisions, why not do request to investigate a person's research to a random sample of ppl in the field, as that's pretty much what pr is doing anyway except one paper at a time. Could be as slow as pr, but at least it would be lag for career stuff which isn't fast anyway, not for research publication.
    If it's research vetting: open to public rating and comment system? open q re who can comment and what weight they get and whether anonymous etc.

    I'm curious about the latter and similar ideas in particular because it also addresses another q which JC doesn't raise: accessibility and usability by research consumers. For econ research in particular the audience is enormous.
    The fact that wrong papers are usually not retracted and it's (as far as I know) impossible for external consumer to figure out whether smth is well-known to be wrong is a disgrace. Given the amount of papers published, I'd say it's crucial for a research consumer to have reliable ratings/know what's considered good/novel/important/most relevant etc by experts. Yet there is none of that, especially for current stuff? Longer term journal stamps of approval are kinda useful, if less so than citation count. Shorter-term there is only authors reputation? And informal mentions on blogs and twitter which are unsystematic and relatively hard to quickly dig up?



    Isn't this stuff what those progress studies ppl supposed to be doing? Maybe somebody knowledgeable can link to their tentative answers on this or related qs.

    ReplyDelete
  2. Maybe I'm drinking the Kool-aid, but I think the slow publication process in econ is part of why quality improves through peer review (you want a 2-week review? You get what you pay for). In contrast, peer review in medicine and natural science is superficial (I've refereed for them a little) and it seems to only detects obvious mistakes. ''Peer-reviewed'' articles in medicine and natural science are like econ research-center working papers (obviously wrong papers are not accepted, but everything else is OK). If we dramatically speed up the publication process, we lose quality (how much?). Anyway, the best outcome is journal competition: let journals experiment! If speed is better than quality, competition will reveal that.

    ReplyDelete
  3. What about this new scandal with the lancet's study database..

    Governments and WHO changed Covid-19 policy based on suspect data from tiny US company

    https://www.google.com/amp/s/amp.theguardian.com/science/2020/may/28/questions-raised-over-hydroxychloroquine-study-which-caused-who-to-halt-trials-for-covid-19

    Their credibility has gone.

    ReplyDelete
  4. People say that the COVID death rate is 0.40% ! This was circulated in several news channels also !

    Me Thinks the death rate is beyond 10%,on aggregate count,and for some nations it is way beyond.

    As per - https://www.worldometers.info/coronavirus/#countries = there are 7.5 million cases and 420000 dead.Simple numerics place it at proximating 6%.

    Wrong me says ! dindooohindoo

    India,Brazil,Russia have seen a sharp rise in cases,in the last 30 days.40% of their cases came in the last 30 days,and for India,it will worsen exponentially.If you see the kill data of the RIB in the BRICS - it has increased sharply,in the last 30 days (which proves my thesis)

    People dying today,were in the quasi morgue (hospitals) 30-60 days ago.Let us take it,at 30 days.

    So we rewind to 30 days ago,and exclude the jump in RIB of BRICS,in the last 30 days. So we have say 4.5 million cases and the kill quant is 420,000

    Rate proximates 10% ! But that is also wrong,as the infected are NOT solely on RTPCR mode.Many nations cannot afford it and are doing antibody tests.An antibody positive may be RTPCR negative,and the vice versa is less likely. If you exclude these specimens from the infected tally,the % rises further.

    Also have to exclude the recovered cases - as those with immunity will recover in 30 days - AS THE VIRUS was DESIGNED THAT WAY.Unlike HIV and Cancer - where patients are NOT likely to recover- on a generic mode.But those who recover from COVID -WILL (in part) come back again,and then die.That will double count the infected cases.Hence,we exclude the recovered cases (which are 4 million,as per site stated above).

    These Johnnies who recouped,may have been infected,say 15 days ago - and if you rewind to 15 days ago,and deduct the spike in the RIBs of the BRICS - you will have an infected base of,say 6 million.If you remove the recovered (4 million),and then ratio it,to the dead of 420000 - then you have a
    kill ratio of 21% !

    Cannot compare the dead to the entire population - as of this instant - as it would include billions of aged,morbid and asymptomatics - who will get infected very soon.

    If we take a 1 year horizon - then post the 1 year - you could take the global population - as that by that time,the virus would have had enough time,to spread,evolve and mutate (across the latitudes and seasons).At that stage,a ratio w.r.t the population,would be a meaningful statistic - to benchmark intra and inter se,with other diseases.By that time the death rate will mature and the complete breakdown of the health infrastructure will be apparent (to explain the future geometric rise)

    ReplyDelete

Comments are welcome. Keep it short, polite, and on topic.

Thanks to a few abusers I am now moderating comments. I welcome thoughtful disagreement. I will block comments with insulting or abusive language. I'm also blocking totally inane comments. Try to make some sense. I am much more likely to allow critical comments if you have the honesty and courage to use your real name.