Mission Creep of Peer review

Composer / musicologist Kyle Gann on Peer Review

A quotation is an ornament to a piece of writing when the quoted phrase is so striking and memorable that the author couldn’t have come up with anything as evocative himself. But if I can state an idea clearly (and little academic writing is as readable as mine), why would it carry more authority if put into a sentence I stole from another writer? If what I say is false, and its falsity has been demonstrated in a previous publication, then I should be told to do my homework. But if what I say is demonstrably true, what does it matter whether someone else has said it before? We are not medieval monks, that we fear to record the fact in front of us unless we can find a citation for it in Aristotle.

Crowdsourcing review

Would crowdsourcing peer review work? Probably not, at least in fields I know anything about. What you need for a peer review is the guy (the gal). The gal or guy who knows the lay of the land, who can compare the article with existing literature. Crowd sourcing works best with lowest common denominator tasks. If I were to ask someone to read an article or chapter for me, it would not just be any old person. Of course, you can also have a reading by someone who knows nothing about your particular topic. Then the reading serves a different purpose, because you have to be more clear about your assumptions to convince that kind of reader.

The reader slightly out of your field can be good, to bring another perspective. So I might be a good reader for your paper on French poetry, say. I wouldn't have any hobby-horses about the subject matter.

But generally you want the guy.

Peer review

This article by Rebecca Schumann is really, really bad. Peer review works really well, actually. Of course, anyone can collect horror stories, but even these must be taken with a grain of salt. Some stories amount to: my article was rejected. Curiously, the comments to her article just skewer it, pointing to its multiple inaccuracies. There are 271 comments and most are critical of her argument. She is losing a lot of credibility because academics know she is full of shit. One comment compared her perspective to that of the typical grad student gripe session. How true.

I've probably done an average of 6 reviews a year since 1990 or so. I've also been on the other side of the process for a quarter century. I know I'm good at it because the editors who use me keep asking me to do it again.

In my view, what is more important than peer reviewers is a good editor. This editor will

*Redact or simply not use reviews that are unhelpful.

*Stop using reviewers who are dilatory or too hostile in tone.

Problem solved!

Rebecca's suggestions make no sense because they are addressed to a misconceived notion of what the problem is. She states that someone should earn the right to submit to a journal by doing peer reviews. But a good editor will only ask established scholars to do reviews. Her notion that most reviews are done by grad students or recently minted PhDs is simply not true.

I haven't agreed with all my peer reviews. I have had a few articles rejected. Usually a grown-up will just learn from those experiences and move on. It is probable that a reviewer has saved me from the embarrassment of publishing something that was not yet ready to be published.

My recent experience suggests that articles being submitted are worse than ever. This is likely to increase the number of horror stories, since someone clueless enough to submit utter crap might be clueless enough not to know that a reviewer is saving him / her from embarrassment.

***

One question is whether it is legit to say that an article should cite me. I have mixed feelings. Sometimes I have felt that an article should cite me, but haven't said so. Instead, I'll give a list of other scholars that also should have been cited. Sometimes, I'll include myself in a list of names that the article should have cited. Sometimes, I feel that if have written on the topic, in a way relevant to the article, and my name is on the editorial board of the journal, the writer should have known that the article would be sent to me.

***

I have had people cite me and disagree. That makes it very hard for me to reject an article, because I would never want to reject someone because they disagree with me.

L'enfer, c'est l'écriture des autres

I swear, reviewing articles is a fool's errand. Having to read articles with a very naive hermeneutical position, after having refined my own so much in the past month, can be tortuous.

I have revised Jean-Paul Sartre's aphorism, "hell is other people" to say that hell is other people's writing. Not the best position for a professor to take, I understand!

Annals of Peer Review

[Details changed to protect anonymity]

I reviewed an article a few years ago that was almost all summary. It spoke with a firm voice of authority, giving a very good and readable summary of a non-fiction book by a Peruvian novelist. The article, however, was rejected because it didn't have a critical argument apart from the summary. The quotes from the novelist were lengthy, but the author of the article didn't really analyze them. Normally, I like a lack of signposting, but in this case there was no introduction, no conclusion, no sense that there were three (or four) main points to be made. The summary just went seamlessly through the book and explained all of it. We never found out what the author of the article really thought about anything. There was no independent critical voice or separation between the the voice of author and critic. We never found out why the non-fiction work was important, or why a non-specialist author should care about any of it.

There are many ways to go wrong. Unhappy articles come in many forms and shapes. The complete lack of effort to frame an article is very unusual, but it can also happen.

Do You Know Who I Am?

I've received some invitations to be a referee in my email recently. "Hey, we're starting a new on-line journal, if you want to submit articles or be a referee for us... If you want to be a referee, send us your cv."

My reaction, always, is to delete the message. Why? I have enough peer-reviewing to do already. I can publish, myself, in better journals. Being a member of an editorial board or being asked to referee a certain article in my speciality is fine, but I don't need to send you my cv to referee for you! I am an established, senior scholar in my field at an R1 institution and if you are contacting me at all to do peer reviews you should know who I am.*

The journal in question who most recently contacted me requires that their referees simply have a PhD and be college faculty. (Any idiot can get a PhD.) There is no effort to get the most qualified referees, merely a mass email sent, I presume, to many, many people. This particular journal did not seem to be scam or one that required a submission fee. That is all the more unfortunate. If someone is starting a legit journal, they should contact distinguished scholars for the editorial board, not a random group of no-name referees.

___

*"Do you know who I am?" is always an assholic thing to say. I apologize for sounding like an asshole in this post. Even if you feel the urge to say that, don't say it. Ever, Even if the person who's irritated you deserves this response, you will still be more of an asshole than that person, who is simply dumb, rude, or ignorant, but is not the arrogant jerk that you are.

Peer Review and Hypocrisy

Since I don't subject myself to peer review that much, there is a certain hypocrisy in [people like me] advocating it. Once I got to the point where I could publish enough by invitation, and promotion was not an issue, I didn't do nearly as much submission where I would be read blindly. Even when I publish in peer-review journals, it is still by invitation. I get suggestions, but am never rejected, and generally I am able to circumvent the worst aspects of the system. Yet I still want my junior colleagues to go through this process. Fish infamously argued that senior people should be able to enjoy the benefits of their stature to publish more easily, and in fact we do, simply because any senior person with a substantial rep can simply not go through blind peer review any more.

The PMLA was the worst, in that I got a good but not thrilling article published there, but they would reject all the much more brilliant work I submitted. It seemed a system geared toward the canonical and boring or the politically edgy, with not as much room for the stuff in the middle. I got my PMLA on my cv, so I can't complain, but why couldn't it have been something better?

Peer Review and Blindness

Increasingly, it's been hard for me to review articles anonymously because I often know or suspect the identity of the author. In one case, the author had sent me the unpublished ms. a few months before. In another, I recognized the distinctive style and approach. In another, the author had not sufficiently anonymized his article. I was looking for an excuse not to do it anyway so I saw how the writer referred to his own article as his own. In yet another, I recognized the same style and some of the same quotes from another article I had reviewed recently for another journal. I didn't recuse myself because I didn't know the actual name, just that it was the same author. If you used google and really tried to find out who wrote something, you probably could in a lot of cases.

So in smallish subfields anonymity is really hard to maintain. I'd have to really write an article in a diligent way to make it truly anonymous at this point in my own project, so it's not really worth it. I get enough invitations that I have no room in my schedule to send out any other articles. Also, I'd rather only publish in special, monographic issues of journals that are likely to be read.

So basically I know or could find out the identities of many I'm asked to review, and I don't need to go through the process on the other side either.

Indulgencia

One journal I review for uses a variation on a quote from Larra: La indulgencia mal entendida es la muerte de la inteligencia." The original is "La indulgencia mal entendida es la muerte del arte." (A badly understood indulgence is the death of intelligence / art.).

In other words, you can't be too complacent, too generous. The journal is telling its reviewers to be tough, not to be too nice just for the sake of being a good guy or gal.

Peer Review Regrets

I've never regretted accepting an article I've accepted. I've never gone back and said I should have rejected that article.

I've never regretted a revise-and-resubmit. in almost all cases, the author has taken my suggestions and published the article. In almost all cases the author has seemed grateful or at least graceful.

The only question, then, is whether I have regretted an outright rejection, a case where the article might have been more easily fixable than I gave it credit for. There is no way to really make this judgment, because it's a subjective measure of where the threshold lies. I've come to realize the system of peer review is rather flawed, and that I've thought of it as fine simply because I assume everyone does it like I do, with prompt, fair, helpful, and thorough reviews. Unfortunately, that's not the case.

A Suggestion for Peer Review

From John Holbo. I think it has some merit, but would be impractical for the reasons I point out in my comment to his post.

How I Do A Peer Review

1. Usually, articles are sent to me as word documents, so I open up the document and begin to "track changes." I Correct some obvious typos and put comments directly into the document. The author of the article will never see these comments or corrections, but I find it helpful for myself. I also take "mental notes," sub-verbalizing what I think. What I'm doing is writing a draft the reader's report in my head. At this point I basically know whether it's an acceptance, a review and resubmit, or a rejection.

2. I get a good night's sleep.

3. The next day I draft a reader's report. I use my marginal comments and my mental notes as the basis of that. I outline the strengths of the paper, the exact points that need to be improved. I might make specific comments on style, organization, and argumentation. When I have the reader's report done, I make sure that it corresponds with my judgment of whether it's a resubmit, rejection, or acceptance. For example, I might have thought it's a resubmit, but reading my own report it's obvious that there are too many serious problems. Or, when I outline the strengths of the article, a reject might be promoted to a resubmit. I never change from acceptance to rejection or rejection to acceptance, though.

4. I get another good night's sleep.

5. I look at my reader's report the next day. Is this still what I think? Do I need to rewrite it to soften the criticism?

In some cases I can do 5 on the same day as 3. In other cases I do the entire report in one day, when the flaws of the article are so obvious that I can tell the editor what they are in much less time.

The most frequent kind of article I read are resubmits, followed closely by rejections and outright acceptances with only minor changes This means that I am usually help an author get hir articles into print.

Peer Reviews

My friend and colleague thought that my idea of offering a peer review service was unique. Neither of us know of anyone else doing this.

Basically, what I do is write a peer review of your article as though I were doing it for a journal. You can get this critique before you send your article to a journal, and get (often) better and more extensive comments from me than you would get from a real peer reviewer, and possibly save some time by not having to wait 2-6 months only to be told your article is a revise-and-resubmit or an outright rejection.

Invitations

Invited contributions have some advantages, especially after tenure.

(1) Rejection is not as much of an issue. I've never been invited to submit something and then rejected.

(2) You can still get feedback, but you aren't necessarily anonymous so you can cite your own work freely.

(3) It is nice to be invited. It shows your work is valued, or at least you are on the short list of people invited in a particular sub-field.

(4) Invitations are usually for special issues of journals or books that are less random than non-special issues of journals. They are more likely to be read by specialists in your own field.

(5) The article is often published in a journal that is usually refereed. Thus it essentially counts as an article in a refereed journal. (Even if you put an asterisk beside it.)

(6) It provides a stimulus for writing an article, and external incentive.

There are some drawbacks too:

(1) If you aren't careful, you can relax your own internal standards. They won't reject you, after all, so you don't have to be quite as rigorous with yourself.

(2) The level of critique you get might not be as rigorous either. You might miss out on the opportunity to improve your work.

(3) You can easily begin allow other people determine your research agenda. (I could do little else but write articles in response to invitations, if I wanted to.) You can justify it to yourself: I would have written this anyway, right?

I have no idea how many articles I have written by invitation. I could look at my cv and tell you, I guess. I'd guess it is a lot, more than half of my total number of articles. I don't regret any in particular, though I might have taken more control of my destiny at times.

***

Invitations are the elephant in the room in any discussion of peer review. If a lot of articles are published through invitations, then peer review becomes much less attractive to senior scholars like myself. If I am reasonably confident that I can write a good article without the "help" of an anonymous reviewer, who may or may not help, then I can side-step the process almost completely by publishing almost exclusively by invitation. Then the only people having to subject themselves to peer review are junior scholars too new to the field to be invited a lot, and who need the stamp of approval of peer review for tenure.

I actually got a lot of invitations even before tenure, so my case was a mixture. I did go through the process of submitting and being accepted or rejected, but very quickly I also began to get invited too. I don't remember a long period of being an unknown.

Is Peer Review Oppressive? (II)

The more subtle argument is that scholars might tailor their work to make it more acceptable to the hypothetical peer reviewer, censoring themselves, or else choose less risky, less polemical stances and research programs. They might be afraid to take on established scholars in their fields. Peer review is oppressive, then, because it subjects younger scholars to a system in which they must conform to established opinion.

My first response is that this is not a problem with peer review per se, but with any kind of system for judging scholarship, as long as it's other, more senior scholars who are doing the evaluating.

Secondly, I think that young academics want to be conformist, more or less. They want to join a community of scholars. To do so they must exercise a certain tact, paying homage to the existing consensus, insofar as it is worthy of respect (even if possibly mistaken). The argument I pose above assumes that everyone wants to be a polemicist rather than to get ahead by jumping on the bandwagon. Is there anything wrong with scholarship being gregarious?

Now since I myself am a polemicist, sometimes going against received opinion, I have to say that some of my controversial work was accepted by very good journals. Hispanic Review wanted my two most argumentative articles, and not the five or six other ones I have sent them. So if scholars are afraid of being polemical, maybe they are mistaken about what the consequences will be.

Also, it seems to me that if you want to be polemical, you have to accept the consequences. You cannot argue against everyone else and then turn around and wonder why your view is not immediately welcomed. Ultimately, originality will be more valued than agreement. If someone writes a book to refute my views, as someone did, then I have to say that at least it's my ideas that are in dispute, not someone else's. It's hard to get upset about that.

The academy sometimes rewards originality and risk-taking, and sometimes rewards mere competence and conformity to received opinion. All of us are somewhat ambivalent about how much originality we really want, so that the same person who claims that peer review can stifle creativity might turn around and stifle someone's creativity in a peer review. Oppression, though, is the wrong concept to apply here. It's more a question of ambivalence, I think.

Is Peer Review Oppressive?

I hear people complain about peer review. Here is my perspective.

First, the complaints:

*It stifles creativity and innovation, because really innovative work might get rejected if it breaks with the consensus of the field. It promotes "group-think."

*Some peer reviewers are slow, nasty, or not very good readers.

*It is too competitive. Not everyone can get into the top journals.

Now, my experience. In the cases of journals that use peer reviews and give the reports to the authors, I have had only one reviewer in my entire career who was unfair in an arbitrary way. I disagreed with some opinions of reviewers, but that's just life. Not everyone is going to agree with me all the time; how boring would that be! I never felt I was penalized for being too innovative or original. I do consider myself to be original, and have had no problems publishing tons of articles.

Journals that reject articles without giving any substantive comments might be using peer review, but they are using it wrong. Hispanic Review used to just send your article back to you in 10 days, with no comment. I've been accepted and rejected by them throughout the years.

I've had articles accepted as is with no changes requested. This is also not a good use of peer review, because no matter how good an article might be, there is always something that can be improved. In some cases, these were invited submissions. If your work is half-way decent, you will get invitations, and most of those articles are accepted and count almost the same as refereed articles, if the journal is one that is normally refereed. That's another way of getting around the oppression, but at the cost of not having peer reviewers save you from yourself.

In my experience on the other side of the ledger, I am very fast, I try to give my best impartial judgment and avoid all ethical conflicts of interest. The hardest articles to review are those closest to one's own work, where you have a particular stake in one side of a debate. I have been told by particular authors and editors that my comments are useful in revision. I recommend a fair number of "revise and resubmits" and have seen quite a few articles successfully into print.

If an article really makes me see a canonical author in a new light, I am tremendously grateful.

Rejected articles are all unhappy in the same way, to cite Tolstoy in reverse. Excellent articles make you see something distinctive in the material they treat. They tell you why Machado is different from Jiménez. Bad articles are generically bad; they go through the motions of presenting information, but they don't have a strong thesis and a convincing argument. Prose is often an issue.

As a reader of excellent scholarly journals, peer review also works for me, in that well-edited journals are actually better than internet sites masquerading as journals that publish just about everything they get. I may not love or be interested in every article, but I can usually see why they were accepted.

So peer review works for me on all three sides of the process. (The fourth side I don't know about first-hand, because I am not the editor of a journal, but my spouse is, and she makes it work for her by choosing reviewers who do a good job. Most authors do not complain.) I could have had an article wrongly rejected, and I could have wrongly rejected an article, but I don't think that has happened in more than 5% of cases. I don't think I've every mistakenly accepted an article. I've never said to myself, "I shouldn't have accepted that one."

I don't know who these original scholars are who have been oppressed by peer review. Aren't the top scholars in my field influential because of their originality? I guess there could be scholarly geniuses languishing in obscurity who are even more original and just haven't been able to get their ideas out there.

No. I don't think so.

***

In my experience, scholars who work hard enough, even without being particularly brilliant or original, are able to get published. It might not be PMLA, but it will be a legit publication.

***

The notion that peer review stifles originality is curious, in that we don't really want complete originality, but work that pushes an existing debate in a new direction. The system rewards meaningful advances based on previous knowledge. I think, in fact, that the system is right to do so. But what do I know? I'm just a part of this system myself.

Don't Do This Either

If you are submitting to a journal that reviews articles anonymously, don't use the 1st person singular to refer to another article you wrote, listing the article in the Works Cited. Then I will know who you are and have to disqualify myself.

What Not To Do

Don't send an article to a journal and leave in sub-headings like "5.1, 5.2" etc... The reviewer is going to know that this is chapter 5 of your dissertation and you simply haven't bothered to change those headings. Ouch.

Don't use the passive voice as a simple default mode. It shouldn't be your automatic, unthought-out choice.

Don't save your worst writing for the key paragraph in which you are explaining what the article is about.

Don't call your own work "groundbreaking." (Well, you can do it if you are writing jacket-copy for your own book, but don't do it in the book itself.) It just doesn't sound good.

***

You can feel good. You can also feel well (being the opposite of sick, not the adverb form of good), but you can't feel badly or horribly. That's a hyper-correction. I winced when Donald Trump corrected Cindy Lauper and she accepted the hyper-correction of "I feel bad" to "I feel badly." At this point, though, the hypercorrection is itself a colloquialism accepted by many speakers of English, so I guess you can feel "badly" if you really want to.

***

Don't use the word thusly in formal writing. Thus is already an adverb and thusly is a jocular expression that only belongs in certain context. Well, I hate it even where it does belong.

Internalized Standard

If your internal standard for what's acceptable is higher than the external standard of the field, then rejection will not be an issue for you. You'll still get rejected once in a while, by highly prestigious journals, but your average article will be accepted on the first or second try. You can't eliminate arbitrariness or unfairness in the peer review process, but you can maximize your chances. Most peer reviewers want to accept a good article.

If you are particularly good in one aspect--organization, prose style, whatever--then you can take that off the table. For example, I can confidently say I have never had an article rejected because of the way in which it is written.

By having a higher internalized standard for your own work I mean simply being more rigorous with yourself than the average peer reviewer would be. Learn to be a good judge of your own work.

Revise and Resubmit

Most articles I review that end up getting published are "revise and resubmit." I rarely just accept an article as is, and revisions might be extensive. The difference between reject and revise and resubmit is whether I see the germ of a publishable article. Usually, the writer just has to organize things a little better.