“Yeah, but good luck getting it peer-reviewed.”
(Apostle speaking to Jesus after curing of the paralytic, Cartoon by E. Flake, The New Yorker, Dec. 19 & 26, 2011, 115.)
Bonnie Wheeler, Department of English, Southern Methodist University, Dallas, TX
Did the postmedieval crowd review work? If crowd review is meant primarily to generate excitement about the “new,” then yes, crowd review worked. As a journal and book series editor, I’ve commissioned thousands of private peer reviews. I’ve written pretty extensively about peer review in the last five years. Reviewers assume an onerous task if they take the job seriously for which (in the case of most journals) they are unpaid and unacknowledged: crowd review provides a bit of acknowledgment and a welcoming professional entry and interaction. The experiment garnered a lot of good p.r. for the journal. The Authors got useful interventions, ranging from enhanced bibliographical recommendations to remarks on sharpening vocabulary and arguments, and included a few overarching recommendations about one or another essay’s logic and movement. The crowd form deliberately eschews ideas of greater and lesser “authority.” Since younger scholars often have sharp, current expertise, that’s healthy. Huzzahs should probably be omitted.
During the Shakespeare Quarterly’s earlier, two-month long open review experiment largely replicated by postmedieval, “forty-one participants (including the submitters, Editor and guest editor) posted more than 350 comments, making for a lively exchange. The journal’s open review pages on MediaCommons were accessed over 9,500 times.” (Rowe, 2011) For the postmedieval experiment, there were 33 unique commenters (excluding contributors), who submitted a total of 45 comments: more than a few, but hardly the size of the crowd that regularly appears at the Kalamazoo BABEL parties. I don’t know how many private comments were sent to any of the Authors, nor do I know how many (if any) the Editor-Enforcers knocked out of the crowd because they were Crazy or Cruel. Eight of the comments came from journal board members. The remaining twenty-five came from mostly humanist medievalists with a sprinkling of scientists. In spite of Eileen Joy’s frequent e-invitations to all to participate, my sense of the postmedieval initiative is that it provoked not a torrent but a trickle.1
Did the crowd experiment work as well as does a private peer review? No, not yet. As I noted with SQ’s Katherine Rowe and MLA’s Kathleen Fitzpatrick in our e-correspondence (Dec. 2010): “For a fully open system to thrive, we will have to learn to tolerate seeing lots of submissions publicly rejected and we will have to find ways to do so respectfully as editors, authors, and reviewers.” Are we (and our tenure and promotion committees) ready for the carnage of visible rejection? Will anything short of sweetness in open replies keep you from being invited to the cool parties?
In its current formatting, crowd review privileges the paragraph (this will surely change as will technological systems) sometimes at the cost of assessing the whole. I missed the private peer Reviewer’s obligation to provide full and detailed readings of an entire essay as well as of its lone paragraphs. Private peer Reviewers gain authority precisely when they ground their judgment in the long discourse with a broad view of our disciplines. Thus the best peer reviews I have read try to understand the essay in light of current and conventional discourse in the field. They remember that some great scholarship was written pre-1985 and often call such work to the attention of the Author. Private peer Reviewers try to assess the likelihood that the essay is sufficiently innovative to impact those conversations. They assess the logic and accuracy of the argument in parts and in sum. Some private peer Reviewers manage this in a few detailed pages, but the best reviews I’ve ever read enter into a rich dialogue with the submission, challenging the Author to divorce the judgment from one’s defensive ego and return to the work with new eyes and a newly opened mind.
I don’t care if the names of the Authors and Readers are known to each other, as they are in some journals, but I do care that private reviews are read only by the Editors and Authors: the Reader’s concern is with the essay and not with the Author’s ego or the Crowd’s disapproval. Traditional private Peer Review doesn’t give one license to be a Mean Girl/Guy, but it carries the responsibility to be blunt as well as helpful. This is a rhetorical problem for open review.
Most simply: peer review is judgment rendered and defended. In a substantial review, an Author and Editor see the Reader’s situation and biases and thus decide how to assess the assessment. A really keen review can lead a smart Author to jettison a flimsy argument or guide the revision process. Penetrating reviews may only come to you a few times in your lifetime, but I know some famed scholars who tossed entire book manuscripts and started over when they understood the clarity and accuracy of a negative review. A great review can have transformative heft.
A few years ago, as president of the Council of Editors of Learned Journals (CELJ), I was in the midst of preparing an essay and essay cluster about peer review for the Journal of Scholarly Publishing, I had terrific e-exchanges with Kathleen Fitzpatrick (now at MLA) and Katherine Rowe (editor of open review issue of Shakespeare Quarterly). My own published argument can be summarized this way: unpaid peer review of journal submissions is the last (and perhaps the only happy) vestige of the “old-boy” network. It assumes that we share a professional vocation to which we commit our expertise on a pro bono basis.2 It is a miracle that Readers give of themselves so fully and generously.
A caveat: I have found it not uncommon for departmental chairs or deans to ask me for copies of peer reviews, especially in cases when the essay has been rejected. I quickly established a policy of shredding reviews. I have never encountered a situation when an administrator has requested a review in order to buttress a positive case.
Does it really matter if you gather these flowers from a crowd or an individual reader? Preprint will become our norm in the humanities; collectively written essays will flourish; new technologies will continue to beget new modes of scholarly production. In a positive sense, peer review in some form will remain because scholarship is a constantly evolving conversation in which judgments are rendered. In a practical sense, as long as we have tenure committees and intense specializations, we will need to accrue every possible form of peer review. It is better to get trenchant private and/or public readings early in the process rather than the week before tenure review. If you go the way of crowd review, just remember you also must be wily: prime your e-pump (what else are friends for?) to get things rolling. Insist that your Editors perfect a system of e-erasure.
1 It took some time for the comment-flow on Fitzpatrick’s e-book, Planned Obsolescence: Publishing, Technology, and the Future of the Academy, to move from drops to a steady stream (emails, Fitzpatrick and Wheeler). The book had previously been accepted for hardcopy publication. Note that her e-book has 2011 print publication by NYU Press (Fitzpatrick, 2011). “Peer Review” (Chapter One): http://mediacommons.futureofthebook.org/mcpress/plannedobsolescence/one/. All the essays for the experimental SQ issue were also accepted.
2 See Wheeler (2011). Also check out the CELJ website, celj.org.
Fitzpatrick, K. 2011. Planned Obsolescence: Publishing, Technology, and the Future of the Academy. New York: New York University Press.
Rowe, K. 2011. From the Editor: Gentle Numbers. Shakespeare Quarterly Open Review: “Shakespeare and Performance.” (http://mediacommons.futureofthebook.org/mcpress/ShakespeareQuarterly_NewMedia/from-the-editor-gentle-numbers/).
Wheeler, B. 2011. The Ontology of the Scholarly Journal and the Place of Peer Review. Journal of Scholarly Publishing 42(3): 307–323.