Research Commons

Connect. Collaborate. Contribute.

Author: Maureen Walsh (Head, Publishing and Repository Services, University Libraries)

Peer Review Week 2017: Peer Review in an Online Publication

This is the second of a three post series focused on transparency in peer review hosted by Publishing and Repository Services and the Research Commons to celebrate the 3rd annual Peer Review Week. The first post reflected on editing the open peer review journal, Empirical Musicology Review. Our second in the series is a guest post from Theresa Delgadillo, the lead editor of Latinx Talk. Published by The Ohio State University Libraries, Latinx Talk is an interdisciplinary, moderated, and peer-reviewed forum for ideas, research, and creativity that fosters critical dialogues.

Peer Review in an Online Publication
By: Theresa Delgadillo

As the lead editor of an online academic forum for most of the past six years, I have been involved in an ongoing discussion about the meaning, function, and process of peer review with scholars who are also working in new formats. The publication that I have been involved in growing is Mujeres Talk, online from 2011 to 2017 and now transitioned to our new online academic forum Latinx Talk, which will debut in September 2017. Mujeres Talk was, and Latinx Talk will be, published by The Ohio State University Libraries. Latinx Talk will carry forward key aspects of the goals of Mujeres Talk, which includes publishing peer-reviewed short-form research, informed commentary, creative work, reports from the field, and guidance on academic professionalization.

For its first year of existence, Mujeres Talk was strictly an editor-reviewed site, and I provided review and requested revisions for all submissions. In 2012, I recruited a group of women to join me in an editorial collective, and in the process of sharing the work of reviewing submissions, we recognized that we were actually engaged in a form of peer review. Over the next few years, we worked to codify our review process into what we called a modified form of peer review, and that model, with some tweaks, became our standard. But we didn’t get there without some controversy, primarily from scholars involved in print journals.

When we began referring to our site as one that offered a modified form of peer review, I heard from a number of scholars who were not shy about how they viewed that claim. “Peer review is blind. How can you say you do peer review?” “Peer review only applies to journals.” “By saying that you do peer review, you’re going to mislead junior scholars into thinking that what they write for your site counts in promotion and tenure when it doesn’t.” What did we mean by “modified form of peer review”? Because two priorities of our site were, and remain, to publish short articles on current research and commentary informed by research, we felt it important to ensure that submissions in these categories sufficiently engaged with relevant and current research. Every submission to the site is reviewed by two members of our editorial collective, and reviewer’s comments are returned to authors with suggestions for revision. Our guidelines for review are published on our web site and include assessing whether a submission offers an original and unique perspective, sufficiently cites and discusses relevant scholarship, is appropriate for our site, and meets our length requirements. Because we are an online forum, we are constantly working to provide feedback to authors on how to write for an audience that includes both academics and non-academics. But our review process was not blind. Reviewers were aware of author names and authors knew who had reviewed their piece when they received our comments. We adopted what might be better termed “open peer review,” in part, because we found this the most economical and efficient way to review for the length, forms, and context in which we published – quick reviews, more advice on how to write in this format in the earlier years, and no staff. We often asked scholars to engage with or cite relevant scholarship. We rarely accepted submissions that did not require revisions, but key exceptions were invited essays from experienced senior scholars.

Print journals maintain expansive Editorial Boards so that there are enough potential reviewers from as wide a range as possible of expertise to review any submissions a journal might receive. In our first six years, we maintained what might be better described as a working group of editors whose research and teaching was centered in our interdisciplinary field, and so we managed to offer strong reviews to submissions from a variety of fields while also pinch hitting on moderating comments, or managing the site. I am not sure how our practice of providing non-anonymous reviews to authors mattered to them, but my impression, based on the professionalism of authors who responded to reviews and revised, is that authors seemed less likely to take a review personally when they knew who the reviewers were. I can think of one or two notable exceptions, but it’s also likely that those exceptions would have taken it personally in any case. This is strictly an impression, and even if I asked a few authors, I couldn’t offer more than anecdotal evidence in support of this impression, but both are based on six years of experience and interactions with authors on over 140 submissions. As a contributor to the forum, I have valued and appreciated my colleague’s edits, corrections, directions, and suggestions on anything that I’ve written for Mujeres Talk.

Authors who published on Mujeres Talk gained a wider audience for their work (between 200 and 1200 page views per post) and gained publication experience, while also contributing to building a public and national network of scholars in Chicana/o and Latina/o Studies. We encouraged authors to include this digital publication on their CVs, while also cautioning them that a short form publication would weigh less in promotion and tenure than a longer critical essay. In these six years, I have received one inquiry from a Chair of a Department asking for clarification on whether we were really a peer-reviewed publication. I explained our open peer review process, which has also always been described on our site. But that inquiry was a sign of what’s at stake.

While our editorial process and policies departed from those of print journals, they did so because we were working in a distinct context, yet this didn’t obviate the need for peer review. I believe that some of that early critique was driven by a number of factors: the perception of a short-form digital format as a threat rather than a complement to print journals, the belief that Latina/o Studies couldn’t sustain more than two or three journals, and competitiveness, but an underlying part of that critique was also the shared experience of difficulty in getting recognition for the scholarship and research of Chicanas and Latinas, and the fear of departing from well-known and respected forms. I have experienced that difficulty quite intimately in my years in the academy, and have witnessed how easily we recognize the innovation of our peers when they are white and male, yet how difficult we find it to recognize the innovation of our peers when they are women of color. While our work on Mujeres Talk and Latinx Talk gets categorized as “outreach,” similar work by others is categorized as “public humanities” or “digital humanities scholarship,” with the attendant disparities in funding and institutionalization for our respective projects. Nonetheless, we have made a contribution to new forms of digital scholarly publications, widening the possibilities for where Latina/o Studies work has impact, and working alongside peers on similar sites. And we are working on gaining funding and resources.

About two years ago, Susy Zepeda, one of our editorial board members proposed that we return to a practice we had intermittently followed in earlier years by sharing our reviewer’s comments with each other as a way of developing a shared editing ethos and of training newer members to the group in how to do peer review. We adopted this practice anew, and we all benefitted from it, especially since the composition of the editorial group had changed with time as some cycled off and new members joined. In an attempt to streamline our editorial process, we tried to review in succession so that the second reviewer would receive both the essay and the first reviewer comments, but this turned out to take too much coordination in the short turn-around times and lack of resources that we work with on Mujeres Talk and now Latinx Talk. However, just this past spring, I was asked to provide a blind peer review for a print journal, and a short while after completing the task, I received a copy of the other reviewer’s comments, though the second reviewer was not identified. This had never happened before. I inquired and the journal editor explained that this was their practice, to share reviewer comments with each other. Though I appreciated the other reviewer’s comments, that reviewer had had access to my comments before doing their own review, and this is how I was able to recognize the drawbacks of trying to do reviews in succession. It may be a good way of helping to teach people how to do reviews, but it lessens the independence of the second review, and thereby deprives the author of greater insight. In our case, a better way to teach new editors how to peer review is to share examples of competent and useful peer reviews, which we do when new editors join our Editorial Board.

Peer reviewing articles and manuscripts is not something anyone ever taught me, but a task I learned in fits and starts when I joined the profession. I learned, slowly, how to do this by reading reviews of my own work; by listening to scholars commenting on reviews they received, reviews they gave, reviews they heard about — sometimes with an eye toward critiquing someone they suspected of giving them a bad review (not always an accurate assumption). Each of our editors brought their own academic experiences of the review process to our shared work, yet because we publish work in shorter formats, work with quicker turn-around times from submission to publication, and meet regularly to coordinate this work, we find ourselves building more of a shared ethos and practice in peer review and editing than might exist at print publications.

Most recently, and as we prepare to launch our new site, we experimented with re-instituting blind review by removing the names of authors from some reviews and removing the names of reviewers in comments to authors. My initial impression was that this didn’t make much difference. I felt that our reviewers were as attentive to rigor as they were to supporting authors who submit to our site as they had always been, but our Editorial Board has expressed a preference for not knowing author names and a desire to institute partial blind review as a way of adhering to academic norms that would make the work of our site, and those who publish on it, more legible in academia.

I confess that my experiment was driven in part by the nagging sense that we might gain greater academic standing if we could claim that we were a publication governed by blind peer review. Those earlier voices of opposition – “peer review must be blind to be peer review” — and more recent experiences with how my work is viewed within my own department – where this work has been viewed as “service” rather than “research” (and “service” weighs less in evaluating merit raises, not to mention all kinds of other ways) both combined to lead me to experiment with blind peer review in this format. I thought that this transition period from Mujeres Talk to Latinx Talk might be a useful time for the experiment. My own view is that for this forum, for this format, we gain more by employing an open peer review process in which we enact, with authors, a shared commitment to make more Latina/o Studies knowledge known among wider audiences – a commitment that calls us all to act fairly with respect to our work and the work of others – as well as constant practice in diplomatically and fairly responding to the ideas of others. I believe our new review process, though partially blind, continues to embrace an ethos of shared respect, commitment, generosity, and responsibility while recognizing that this ethos only exists to the degree that we attentively enact it every day.

For more on Peer Review Week, including available webinars, visit the Peer Review Week activities page: https://peerreviewweek.wordpress.com/activities/

To learn more about the Libraries Publishing Program, visit our website: go.osu.edu/librarypublishing

Peer Review Week 2017: Reflections on Editing an Open Peer Review Journal

Today marks the beginning of the 3rd annual Peer Review Week. To celebrate, Publishing and Repository Services and the Research Commons are hosting a three-post series focused on transparency in peer review. Our first in the series is a guest post from Daniel Shanahan and Daniel Müllensiefen, the editors of Empirical Musicology Review (EMR). EMR is an Open Access and open peer review journal published by The Ohio State University Libraries.

Reflections on Editing an Open Peer Review Journal
By: Daniel Shanahan and Daniel Müllensiefen (Editors, Empirical Musicology Review)

Science, as Thomas Kuhn writes, is not an immoveable object – it is a constantly evolving community of scholars engaged in a dialogue. As commonly accepted theories are supplanted by newer findings, the focus, discussion, and scientific paradigms change. In any field, researchers create knowledge and discover findings only as a result of a long discourse in which studies are reviewed, picked apart, replicated –or not– and built upon. Such discourse requires both a community in which scholars can trust that their research will be discussed fairly and openly, while also rigorous and with high standards. Finding the sweet spot between both a curated and an open environment can be difficult, however, and most peer review journals have tended to opt for an anonymously curated environment, sometimes at the expense of open discourse.

There are obviously a number of benefits that come with such anonymity. Reviewers, particularly those that might be untenured or on the job market, can be honest, often speaking truth to power in a very needed way. The downside, however, is that reviewers can sometimes be unhelpful. As anyone who has read any website’s comments section will attest, anonymity can frequently breed a certain degree of unconstructive negativity. A further downside of the anonymous peer review system is the difficulty of attracting expert reviewers. On average, a scholar might be asked to review at least 5-10 articles a year, with that number increasing as they become more established in their respective field.

It therefore makes sense that many of the most senior and well-established researchers will turn down peer reviewing requests. Peer reviewing can be a difficult and thankless task, requiring careful reading of the submission, as well as many of the main sources from which the submission draws, and thoughtful reviews take time – several hours and sometimes even days -to write. In almost all cases reviewers are not compensated in any way for writing reviews and there are hardly any other benefits either: anonymous and unpublished reviews cannot enter academic CVs and they do not help promotion or achieving tenure status. Unfortunately, this means that journal editors are often left scrambling for reviewers. In a worst-case scenario, this will lead to reviewers that are not an expert in the particular area they are reviewing; they are, after all, just filling in for those that couldn’t do it. The ideal situation would be one in which established and respective scholars would feel incentivized to review and engage with works in a meaningful and constructive way.

EMR issue cover image

Empirical Musicology Review (http://emusicology.org) was founded as an online journal in 2006 by David Butler and David Huron with this in mind. By adopting an open peer review model, as pioneered in journals such as Behavioral and Brain Sciences, reviews would be in the form of an invited and published commentary provided by multiple experts in the field. This would not only encourage writers to contribute (they would get a publication), but it would also – hopefully – ensure that reviews would be constructive and helpful. The reader would be a witness to this dialogue.

Empirical Musicology Review is now on Vol.12, and publishes four issues a year. Submissions are vetted by both editors, and upon preliminary acceptance, it is sent to a reviewer, who then writes a commentary to be published alongside the article. Perhaps the question we get most often is “but is it actually peer reviewed”? We would argue that, yes, not only is it peer reviewed, but submitting to EMR requires a certain degree of confidence in your work. If it is methodologically sound, and contributes to the larger discussion, it will likely be accepted. The finer points of the study, however, will still be picked apart, and in an extremely public forum. There are instances in which we reject articles before sending them to commentators. These are rejected for a number of reasons, including content that falls outside of the scope of the journal or research that hasn’t engaged with the previous literature in the field in a meaningful way.

The open peer review process is not always completely open. Occasionally, the reviewer will alert us to issues with a central premise of the article, such as a misinterpretation of a previous article’s findings. In these instances, we are faced with a decision: do we let the article stand as is, and allow the commentary to mention such issues, or do we discuss the issues with the author before publication? Whenever possible, we do our best to keep the original article as is, allowing for it to be discussed in a commentary. Sometimes, we will discuss these points with the author ahead of time, asking if they would like to change it.

We might therefore argue that EMR is, in fact, a quasi-open peer review journal. Before publication an article is not only reviewed by the editors, but we provide as much constructive feedback from as many sources as possible to the author. Each article is then published alongside reviews that address the methodological and philosophical points of the article, and place the article within the larger context of research in the field. Often these commentaries, suggest directions for future research, or simply point to alternative conclusions one might draw from the data.

This model has proven to be quite successful. Commentaries are frequently written by some of the biggest names in the music research community, and as an early-career researcher, having a senior scholar engage with your work can be exhilarating and extremely useful. We don’t expect this hybrid model to supplant the traditional peer review model at any point in the near future, but we hope it fosters a dialogue, and shines a light on the process.

For more on Peer Review Week, including available webinars, visit the Peer Review Week activities page: peerreviewweek.wordpress.com/activities/

To learn more about the Libraries Publishing Program, visit our website: go.osu.edu/librarypublishing