Tuesday, July 24, 2018

Shorter Papers

Ben Leubsdorf at the WSJ does a great job of covering the discussion within economics over too-long papers, picky editing and refereeing, and other issues.

Defensive writing is certainly part of the issue
“If you want to publish a paper in a top journal, even if you think you have one key insight that can be conveyed succinctly, the referees are not going to take it,” Ms. Finkelstein said.
I think Amy would want to clarify this means referees at other journals. Editors are also to blame. We must remember, referees do not take or reject papers, referees advise editors, and it is always the editor's job to make publication decisions.
From an early stage of an academic career, “it becomes pretty clear that you need to check off a pretty long list of items to really convince people that the way you’re interpreting your results is indeed the right way to do it,” Mr. Bazzi said.
..... When you’re trying to anticipate possible criticisms on a controversial topic like the minimum wage, and situate your research in the deep existing literature on the subject, it “quickly adds up to a long paper,” said University of Massachusetts-Amherst economist Arindrajit Dube,....
Mr. Dube said that paper is now in the process of being revised ahead of publication—including acting on a request to make it shorter.
However, journals don't encourage length, and there is some sense to the current equilibrium.  You write a paper with lots of defensive "what if this what if that." You send it to journals. My typical paper is rejected at 2-3 journals, so by the time it's published I have 6 to 12 reports.  My referees are typically thoughtful and diligent, and the paper grows in addressing all of their what-abouts too. Since I haven't been doing detailed empirical work lately, the requests are not nearly as extensive as those authors receive. Then we finally arrive at publication, and the editor says "now cut it down to 40 pages. You can stuff all that into an internet appendix if you like." Which nobody reads.

This isn't necessarily a bad equilibrium.


Way too many papers are published in the first place. Way too many of them have pretty fatal flaws: didn't the (12!) referees notice the huge selection bias? Simply less reviewing isn't necessarily the answer -- though journals are pretty wantonly using free referee time. And it seems to me that everyone else's papers get better as they go through the process. As a reader, short papers that have been through the wringer are better!

(On the other hand, I did once send a paper that was less than 10 pages to a finance journal only to be told that the journal does not print short papers. The editor invited me to expand the topic and send in the usual 60 pages.)

There is also some selection bias in the remembered history. Yes, there are a few great short papers, and papers that have trouble getting published, have been immensely influential. (In addition to the mentioned papers by Samuelson and Nash, Akerlof's lemons, Bob Shiller's volatility tests -- an AER papers and proceedings with one sexy graph, that would never have been published as a regular paper -- Bob Lucas' Critique stand out in my  mind.)  But there are many other terrible short papers, and I have seen many papers that were confused messes in the first draft get a lot better. The process, laborious as it is, does add value.

The whole discussion raises the question, just what are journals for anyway? Journals started as the internet of the 1700s -- a way to communicate results, using this great technology of printing and paper. The combination of the internet of the 1990s and increasing publication lags mean journals do not have that function anymore. "Publishing" is now a moment of setting a particular work in stone.

I think journals are trying hard to make papers perfect on publication, especially given the replication crisis in social sciences, and the number of prominent economics papers that have fallen apart under scrutiny. There is a vision that it should not be "published" unless it's "right" so that anything "published" is reliably true. I think we need to give up on that hope! Publication is the start of a conversation. Most papers are forgettable. The ones that matter will have others tear them apart. Yes, papers should be not full of obvious problems, but the 70% or more of weight that is what about this and what about that is probably not that useful.

Writers could do a lot to help the whole process. Writers: Get to the point! Why not write the short paper that Amy Finkelstein describes and then tack on all the extra stuff? Don't force the average reader to go through a lot of junk to get to the results.
Massachusetts Institute of Technology professor David Autor compared a 94-page working paper about the minimum wage to “being bludgeoned to death with a Nerf bat” and started a Twitter hashtag, #ThePaperIsTooDamnedLong.
Well, if the paper (I have not read, I'm inferring) had gotten to the point -- the 10 page paper followed by the what-ifs, then most readers would not have had to read 83 pages.

Current papers tend to be structured:

  • Motivation -- why you should care. Useless as it's about a result you don't understand yet. 
  • What everyone else is doing on the subject and what's wrong with it. Generally impossible to follow unless you already know it already.
  • Review of literature, what other people are doing on the topic. Ditto
  • Preview of results. But without what you did to get the results, usually useless. 
  • If empirical: Data sources and transformations. Plots of data. Preliminary analysis. Cute facts. Simple results. Finally, around table 4, page 35, and an hour and 10 minutes into the seminar presentation, the main result. Then extended what if this what if that. 
  • If theoretical: (good papers) A simple example that shows the basic idea, then the main result. (most papers) An incredibly general setup. Then simplifying assumptions, and finally the main result. Then the simple example. 
  • Conclusion outlining the researcher's plans for the rest of career
  • Long appendix. 

Often this has gone on so long that the actual meat of the paper is stuffed in the appendix. Theory papers don't include proofs or derivations any more! Empirical papers don't include an actual complete specification of the specification. This is the paper. It's what you did. Throw the rest out!

Writers could help a lot by focusing laser-like on what this paper does, in the first 10 pages. Then stuff the rest in the back. Motivate it after we know what it is. Tell us how it fits with what everyone else is doing after we know what it is. (Or leave that to historians). And so forth. All it takes from journals and referees is a little tolerance for unusual organization. That is not easy.

Journals are changing. I think Glenn Ellison deserves a lot of credit for starting the modern wave of self-examination in The Slowdown of the Economics Publishing Process. (Publishing this paper was also one of my proudest moments as JPE editor.) Since then, journals have all been moving to process papers more quickly, to see themselves as making publish or not decisions, rather than endless refinement; a one round and then yes or no ethic is taking hold, desk rejections of hopeless papers are increasing, and editors are becoming much more decisive.

The process is ongoing. Open refereeing and discussion forum are hot topics that I hope will bear fruit sooner rather than later. We are wasting a lot of time on a faux judicial process. The insights of people who have read and thought about the paper, such as conference discussants, are lost in the editorial process; first round referee points are also lost, and a huge amount of effort is duplicated. Achieving such an open forum with a modicum of civility is an important challenge.

Hmm. A post about too long papers has gone on far too long.

6 comments:

  1. Oh MY goodness...So much to say on this (some which is not very popular with the MIT crowd and some not popular with Grumpy). But I will refrain.
    I will say this: I liked the WSJ article, but unfortunately it fails to mention that *there are* journals (well known ones) that *do* publish short papers on purpose:
    1) Economics Letters (EL), which has been around for ages - Amy Finkelstein forgets this when she proudly talks about her new endeavor in the AER: Insights. Eric Maskin (very famous theorist) was a long-time editor (I think) of EL. 2) Economics Bulletin, a web-based short paper journal (founded by my advisor, John Conley), and this journal has been around for several years now.
    It would have been nice that the WSJ mention these facts. Which for me, begs the question, why do we need AER:Insights anyway?

    Finally, I think there is value in longer papers. There is value in justifying why exactly what you are writing really matters. There is value in describing the evolution of an idea (like a little history of economic thought of a topic), and thus, trying to justify why your contribution makes a real difference.

    Now... on referees, editors, etc, that is a fight for another day. I have hyper-strong feelings about this. And this is not the moment, and perhaps not the forum... (because I know Grumpy has strong feelings as well... and Grumpy is the gatekeeper of his blog...:-) :-)).

    ReplyDelete
  2. I largely agree that many papers are longer than needed today and the reasons that is the case. To the question of what are journals for I would say now journals exist to determine who gets tenure. There seems to be a disconnect between the number of papers we think should be published, the number of tenured faculty departments desire, and tenure standards. It shouldn't be a particularly hard equation to solve, but the parts never seem to add up.

    ReplyDelete
  3. Hi John, thanks for the post!

    Since you already discuss what should be included in papers, it would be interesting to have your thoughts on a similar topic: Multiple testing in empirical papers. When I read (very well published) empirical papers, I often see enough degrees of freedom that I guess any PhD student with mediocre talent can get significant results, even when the effect is nothing more than white noise. Just considering the sample selection (time frame, S&P 100,500,1500, exclude financial service companies?), the estimation of abnormal returns (different risk free rates, FF3 model, FFC4 model, different estimation window, use log returns?), and the hundreds of possible controls that can be included or ignored gives an endless number of (more or less) reasonable data points. Obviously some result and 5 selected robustness checks are significant. Why does nobody ask for a histogram of t-values including all these degrees of freedom? Or control for multiple testing as other fields, such as Biology, already do since a couple of years?

    Maybe you find the time someday to write a couple of words on it, I think it would be very interesting.

    ReplyDelete
  4. Trump must read Grumpy. In his own words, today in the Oval Office with European Commission President:

    "If we could have no tariffs and no barriers and no subsidies, the United States would be extremely pleased. We want it to be a level playing field for our farmers, for our manufacturers, for everybody. Whether it’s with the European Union or others, it has to be reciprocal in nature at a minimum, and we’re working on that, and I think we’re making tremendous strides."

    ReplyDelete
  5. Frederic MenningerJuly 26, 2018 at 1:13 AM

    I am not a fan of avoiding real names in posts and just realized my own comment concerning multiple testing is labeled with unknown. Sorry.
    Frederic

    ReplyDelete
  6. your papers are too long

    ReplyDelete

Comments are welcome. Keep it short, polite, and on topic.

Thanks to a few abusers I am now moderating comments. I welcome thoughtful disagreement. I will block comments with insulting or abusive language. I'm also blocking totally inane comments. Try to make some sense. I am much more likely to allow critical comments if you have the honesty and courage to use your real name.