The EJS provides space for research notes like the one that appears below. Research notes should generally be less than 2000 words, short, concise and moderately well documented. Their purpose is to a) inform the scholarly community of planned or in progress research in order to b) lay preliminary intellectual claim to important research areas, and c) elicit commentary, feedback, and even collaboration. Publishing research notes in the electronic medium is a highly democratic alternative to the invisible college structure first noted by Price (1966) since the entire discipline, rather than just invisible networks of key researchers, has access to information on current activity. In addition research notes have the benefit (?) of obviating the need for travelling to conferences in order to remain current.
Mike Sosteric
Department of Sociology
University of Alberta
Edmonton, AB Canada
© 1996 Electronic Journal of Sociology
Mike.Sosteric@Ualberta.ca
...the traditional image of the scientist-as-writer, an image common both within science and without, is... a peculiarly humble and constricted one. Gone are the creativity and daring of the Titan. In their place is the self-effacing toil of the amanuensis, for, unlike poets, who are free to create the world anew as they write of it, scientists must be totally circumspect, must expunge every trace of their own personalities from their work, as they record only what is there. The scientist must view the world through some impersonal ocular device, the spectacles of objectivity, rather than through the eyes of a person. The scientist must weigh the world not in a hand but in a balance; must measure it not with a stride but with calipers. The Promethean scientist may uncover the secrets of the gods but must whisper them - like someone with an artificial larynx - in a voice devoid of emotion so as not, as the Fiesers put it for their fellow chemists, to "divert attention from the story we are trying to tell." Above all, scientists dare not run naked through the streets shrieking "Eureka!" or they will, as the Fiesers say, "violat[e] principles of good usage."
David Locke
Since the late 1950s, writers and government agencies have been commenting about the 'crises' in scholarly publication (Licklider, 1965; SATCOM, 1969, Vagianos and Kurmey, 1969). Most of those who have chosen to comment on this crises have focused on things like cost (McCarthy, 1994; Moline, 1989; Thompson, 1988), proliferation of journal titles (King, McDonald, Roderer and Wood, 1976) and concomitant fragmentation of research (Fox, 1994; Lock, 1994; Stossel, 1985), and bottlenecks in the process (Harnad, 1991). Others have focused more specifically on the peer review process. For example, some earlier challenges came from psychologists who found evidence of systematic bias in the peer review process (Abramowitz, Gomes and Abramowitz, 1975; Mahony, 1976; Simmons, 1978). Others, like Gardner (1981) and Grogan (1976) challenge the gatekeeping process of peer review by pointing out that the scientific communication process has not always proved a stopgap against ridiculous or bogus claims or by pointing out that major contributions to disciplines can be overlooked by even the most prestigious journals.
While all these are important considerations, philosophers of science have recently begun to criticize the epistemological foundations of formal scientific communication. Writers like Gross (1990), Locke (1992) and others have challenge the privileged position of 'science' as the road to truth by arguing that there is an underlying social and rhetorical dimension that needs to be considered. These authors further argue that the essential rhetorical nature of the scientific communication process is systematically underplayed or ignored, or that its actualization in the peer review process is lopsided. Yet scientists of all stripes prefer to see their disciplines as being capable of unproblematically referencing the natural or social world. Both Gross (1990) and Locke (1992) competently demolish the realist assumptions of science and replace, or more appropriately embed, the processes of scientific truth finding in a context that accepts and emphasizes the rhetorical nature of science.
The problems with peer review become evident once the fact that science has a rhetorical element is accepted. On the one hand, the traditional mode of peer review obscures the problems of reference and the rhetorical dimension of science. The rhetorical process which is at the heart of science and peer review conveniently disappears with the final publication of the manuscript. In its place is an ideal typical representation (the scientific paper) of the realist assumptions about empirical reference. All the academic world sees is a polished manuscript where the personal involvement of the researcher and reviewers has been systematically eliminated. The sanitized manuscript supports the myths of science by reinforcing the notion that science unproblematically references the natural world. "As a result of this process, there is cultivated a systematic neglect of the relationship between the claims in these reports and the process by which their truthfulness is initially certified: the move to publication systematically distorts the wholly argumentative grounding of the knowledge that peer review certifies" (Gross, 1990: p. 140).
The other problem arises when we consider Habermas's ideal speech situation This theoretical construct describes the ideal type of interpersonal interaction that should exist in a rhetorical situation. Gross (1990) draws from Habermas in his description of the ideal speech situation. Gross describes the ideal speech situation in the following terms. 1) The ideal speech situation permits each interlocutor an equal opportunity to initiate speech. 2) There is mutual understanding between interlocutors. 3) There is space for clarification. 4) All interlocutors are equally free to use of any speech act. 5) There is equal power over the exchange. As applied in the context of peer review, Gross notes that ideally "scientific peer review would permit unimpeded authorial initiative, endless rounds of give and take, [and] unchecked openness among authors, editors, and referees. (Gross, 1990: 137).
Remembering the assumption that the peer review cycle is instrinsically rhetorical, this criticism of peer review now becomes evident. Peer review as it is traditionally practiced is not an ideal speech situation. That is, it does not conform to the dictates of good rhetorical practice. There is a strong element of hierarchy and authority in the process. Except for the initial act of initiating the peer review process, author's almost never initiate speech. There is no opportunity for clarification since only one round of referee critique and editorial comment is possible. Flowing from this is the inability to ensure mutual understanding since the author can't respond directly to reviewers, nor can reviewers query authors about specific problems with the submission. That is, there is no possibility for interactive clarification. Rather, authors must "reply to every question and respond to every criticism despite the fact that, were interaction possible, some questions might not have been asked, nor some criticisms made." (p. 131). Further, "...authors are prohibited from issuing commands and inhibited from asking critical questions, while editors and referees do both freely." (p. 133).
This set of peer review transactions can be analyzed in terms of the criteria of the ideal speech situation, which purports to be a set of suppositions unavoidable if we are to create rational consensus, to detect its presence, and to judge its quality. Perfect symmetry is required: each interlocutor must reveal himself, each must have every opportunity to initiate communication and to make assertions, and each must have equal control over the exchange; in the pursuit of rational consensus under these conditions, bias is revealed, countered, and neutralized. A peer review decision made under the conditions of the ideal speech situation represents a consensus among referees, editors, and authors that a particular paper is publishable science (Gross, 1960: 130; s.a. Habermas, 1992).
Accepting the fact that there is a rhetorical element is key. Not only do we move towards an acceptance of the underlying reality of the situation, but also as Gross indicates in the above quote, we install additional checks against bias. This applies particularly to the comments of reviewers. When the author is able to openly challenge certain statements as inappropriate, faulty, or even biased, the reviewers are forced to reconsider their comments in light of information or clarification provided by the author. Whether this leads to a more objective and fair process is a key question being addressed in this research.
There are related problems that emerge as a result of the inadequate realization of the rhetorical in the peer review process. One such problem is the resulting intransigent peer review process which discourages further development of a submission. Authors faced with a process where they can only listen and not respond are less likely to make suggested revisions and more likely to seek alternate outlets which ask less questions but publish more papers of questionable quality. This is unfortunate since peer reviewers may make useful suggestions and no doubt almost all submitted papers can benefit from at least one round of peer review. However the inability of the author, who it should be remembered can also be considered an expert in the field, to respond to criticisms that may be irrelevant, misplaced, or based on a faulty understanding of key sections of the manuscripts, virtually dooms the submission to mediocrity despite the appropriate criticisms of peer reviewers. An unfortunate result is that the time, energy, and work of both authors and reviewers is lost.
These criticisms go against the idealized versions of the scientific process propounded by writers like Popper (1968), Garvey (1979) and King et. al (1981). King et. al provides a representative statement of the idealized conception of scientific communication.
Overall the reporting cycle reflects a variety of modes of presentation, including direct and indirect, formal and informal, and oral and written. As the cycle progresses, the size of the audience reached and the degree of formality increases. With publication of a journal article, the research results are certified and made available to the entire scientific community. There is redundancy in this cycle, but for the most part it appears to be a necessary part of the interactive social process leading to the general acceptance of scientific research results (King, McDonald & Roderer, 1981: 6-7).
Whether dealing with bias in the peer review process, failures in the gatekeeping process, or criticisms of the peer review process, it is clear that an idealized version of the scientific communication process is untenable. However, the question is begged. Can we move in the direction of a more adequate and responsive system? And if we do make that move, what will be the benefits? Gross discards the possibility of an ideal speech situation in peer review and argues against the possibility of even approximating an adequate process. He further argues that moving in the direction of the ideal speech situation in peer review via conventional means would put undue strains on the whole process and would also require more time of the peer reviewers who would have to be engage in a critical debate for longer periods. "A decided movement in the direction of the ideal speech situation would improve an author's position only at the expense of the professional time of editors and referees." (Gross, 1990: 138)."
However, with the advent of information technology, it has become possible to move in the direction of repairing the peer review process, at least at the rhetorical level, by introducing a form of peer review that allows online dialogue and debate between authors and reviewers. In the next section I describe the methodology of the EJS case study into interactive peer review that seeks to determine whether interactive peer review can bring progressive qualitative changes to the peer review process and if so, what the nature of those progressive changes are.
Reviews will remain semiprivate until the final decision has been made. Only members of the EJS collective, and the authors' themselves will be able to view and comment on reviews. However once the final decision has been made, the papers and the associated reviews will become public in the form of preprints and associated commentary.
The EJS collective has agreed to set a formal limit on the study of 15 papers or 6 months. Most likely it is the latter limitation that will bring the study to a close since it is unlikely that in the next 6 months we will receive 15 papers for consideration.
A mechanism for ongoing feedback in the form of a netforum newsgroups for participants will be provided to ensure adequate opportunity for comment.
In addition to the preliminary exchanges between myself and the members of the board, I will also keep transcripts of the content of the peer review process, and of the authors formal responses to peer review comments. There will also be an online space for ongoing commentary by participants in the study.
A third tier of data collection will involve a short questionnaire designed to elicit the opinions of both peer reviewers and authors of submitted manuscripts about the process of interactive peer review. At this stage, I will be concerned with the personal reaction of participants to the study. Did they find the process differed in any way from the traditional methods of peer review? Did they believe that allowing "interaction" was beneficial? And in what way? Was there a noticeable shift in the dynamics of power and authority? Was the process more or less biased? It is useful to note that some members of the EJS have already commented that opening the peer review process may not have any substantial impact on the peer review process. Rather, the appearance of a progressive an open process may do more to obfuscate the relations of power an authority than it does to demolish them! This is a key point I think, and one that the questionnaire will seek to elicit answers to.
However, I remain open about the possibility. If it is true that the implementation of technology is embedded and conditioned by social and structural factors (Noble, 1979), then it follows that the outcomes of the implementation of technology will depend not on some illusory nature of technology, but on the goals to which technology is directed and the manner in which it is implemented. This gives to technology not a deterministic, but an ambivalent and ultimately negotiable impact. Ultimately, it is to learn about the negotiation of technology and its progressive application (or potential application) to which we address ourselves in this study.
|Front Page|Table of Contents|