August 30, 2009, [MD]
The electronic publishing and the Open Access movement has led to many different experiments with academic journals. One idea that I found quite interesting is "open peer review" (the Wikipedia article is a good overview). Open peer review could simply mean that the author knows the names of the reviewers, which is not very interesting to me. What is much more interesting is when the journal publishes not only the final revised and copy-edited version, but also all the intermediary steps, including the communication between the reviewers and the author. These two versions of open peer review could be combined, but they don't have to be - one could still open up the communication to the world after publication, without revealing the names of the peer reviewers.
I've always thought this was a very exciting idea, because it allows you a peek into the process of publishing, and can be very helpful when you are yourself beginning to write for publication. I think this is especially true for beginning scholars who do not receive as much support from colleagues or mentors - for example graduate students who are not at top institutions, or researchers in the developing world who want to publish in English. Publishing is a mysterious process, and opening it up to see that even the greats in the field had to go through several rounds of editing can be very confidence building. (In a similar vein, Gans and Shepherd interviewed famous economists about their histories of being rejected).
However, so far I believe it the only journals experimenting with this have been in the STEM fields, and the things they tend to comment on - statistical significance and so on - are very different from picking apart an argument in a political science or anthropology text. It would be great if some journals in the social sciences and humanities would also agree to open their peer review process.
This summer, I submitted a number of manuscripts to various journals. This is the first time I have submitted papers to real journals (I submitted one to an undergraduate journal when I was doing my BA). I got three papers accepted (one a multi-authored one), and three rejections (two papers, and one conference submission). However, I was quite inspired by the rejections, because they were all very "good" rejections: in all cases, the editor or the peer-reviewers had engaged deeply with my material, and gave insightful and (mostly) useful comments.
Comparative Education Review is the flagship journal of the Comparative International Education Society (CIES), an organization that also holds a large annual conference with several thousand attendees. It's also considered one of the most prestigious journals in the comparative education field, and I got a chance to listen to some of their editors during their last annual meeting. Although extremely selective, what impressed me was their commitment to providing very good feedback, even for papers that were not accepted. So although I did not have high hopes that they would accept my paper, I submitted it to them this summer.
I received it back, rejected by the editor, who did not think it was good enough to be forwarded to the peer reviewers. But he wrote a page long very good critique of the paper - much more than I received in my graduate school, where I got an A and a "good job"... I wrote back to thank him for the thoughtful critique, and also suggested that it would be very useful for others if they made, not necessarily all (I didn't want to be too bold), but a selection of their articles "open peer review".
To my surprise, Dr. David Post (the editor who had reviewed my paper) wrote back and said that he very much liked the idea, and that they would produce a document containing the full correspondence of past papers. I recently contacted him again to see if something had happened, and lo and behold: on their website, they currently host a 369 page document containing the communication between the author, editor and peer-reviewers concerning 35 different articles between 2003 to 2009. This is a selection that makes up about 1.5 years worth of the journal, which I suppose is based on the subset of authors and reviewers that agreed to have these documents made public.
This is how Dr. Post introduces the document on the CER website:
To provide examples of reviews and of the CER editors' responses, we selected about one-third of the articles published over the past five years, choosing a variety of styles, methods, disciplinary orientations, and geographic areas. We are grateful to the authors we selected for allowing this open access to their files, especially since (without exception) all files contain critical suggestions and request further improvement. The authors' willingness to participate speaks volumes about their security as scholars. The CER has been publishing less than 15 percent of submissions in recent years, and all the articles represented here have gone through multiple revisions, as part of the review process, before they were developed enough for acceptance.
Having a quick look through the document, I see several author names that I recognize - people who are intellectual leaders in the community. As Dr. Post says, it is very generous of them to give us some insight into not only their finished published articles, but the struggles they had with getting them published (some articles were revised 4 times).
Of course, CER has not changed to open peer review, and in that sense, the title of this blog post might be slightly misleading. This collection was based on previously published material, and this is not a policy for articles going forward, nor are there, as far as I understand, any plans for producing follow up volumes. However, I still think this is a very brave move by one of the leaders in its discipline, and perhaps other journals will be inspired to follow. We are hoping to start a article writing group at my school this year, for students who want to begin submitting papers to collaborate, share ideas and critique each others work, and I think using this document can be very valuable for that project.
Thank you, CER!
Stian Håklev August 30, 2009 Toronto, Canada comments powered by Disqus