"A referee report is not a mind-dump about the paper."

That is my favorite line in a recent Journal Economic Perspectives paper on how to write good referee reports. Some of my personal thoughts on the recommendations: 

1. Most importantly, I am so happy to get any guidance on how to write a referee report.  Yes, we can talk about the not-so-ideal incentives referees may have when writing reports, but I think a bigger problem is that we, the referees, are not exactly sure what editors would like to see in referee reports. Until now, most of my thoughts on this have come from reading reports on my own papers, but if everyone just does this, norms determine everything. 
2. I do like to see a good, healthy number of robustness tests in a paper, and I kind of like extensions. I think these additional analyses should be done. The question then is how many need to be done by the original authors vs. other researchers in other papers published in less prestigious journals. I don't know the answer to that. 
3. The most important piece of practical advice in the paper is to separate comments by whether they are essential or suggested. I think all authors want to make their paper the best it can be, and it's nice for referees to share their impressions on how to do that. That said, the authors have often thought about the issues for many years while referees have thought about them for..a week? a day? Given this, I don't think the best way to produce good research is for the authors to mindlessly follow the whims of referees---even though that is the often the easiest and least risky approach to getting a paper published. I think it's fine for referees to share their whims (sometimes whims result in great ideas) as long as they are clearly labeled as such.  
4. More useful practical advice: Comments should be numbered and the whole thing should be 2-3 pages.
5. The hardest part of refereeing is making a call on whether the topic of the paper is "sufficiently broad interest" or whether the paper has"made a sufficient leap over the existing literature." I would have love to have more guidance on how to make those calls, but I guess in the end, it's just about our opinion. Ah, but this brings me to my favorite piece of advice for my graduate students writing papers: Do take the time in your introduction to explain the contribution of your paper, making it very clear how the paper contributes to the existing literature. 

Ok, time to get back to writing a referee report. Or a paper.