That, too, can be gamed, although it’s not guaranteed to work. One of the ways an editor can pick reviewers is by looking at the reference list: someone who has multiple papers cited in the manuscript is more likely to be familiar with the science topic, if not the specific research protocol. Even in a small field, no editor will know all of the relevant literature, so the authors can stack the deck by citing the papers of someone who is likely to be a sympathetic reviewer and not citing the papers of someone who is not likely to be sympathetic. This won’t work if the topic of the manuscript involves a sufficiently well-known controversy, because even the editor will notice if the introduction fails to acknowledge the existence of the controversy. But with non-controversial topics or large fields, this technique is more likely to succeed.

Of course, even if the authors successfully game the review in this fashion, it will still be a genuine review, unless somebody is successfully spoofing the reviewer database.

There should be techniques for catching spoofed e-mail addresses of suggested reviewers. A Google or MSN e-mail address should be automatically regarded with suspicion, unless the potential reviewer is identified as having a Google or Microsoft affiliation (most journals I am familiar with require authors to supply affiliations for suggested reviewers). For that matter, an address that does not match a known affiliation of the reviewer should raise a flag. Any reputable organization (and quite a few that aren’t) should have its own domain by now.

Ethan, thanks for that insight.

Also, it may be that the authors were not always aware of this game. It seems that some companies that help non-English speaking authors polish their manuscript were also helping with the submission.

That won’t solve the problem, either, because it can be defeated by the simple expedient of registering an alias with the database.

Even requiring the editor to know who the reviewer is can be problematic. It’s one thing to do that in a small field like astronomy or geophysics (the latter being my area). But even there, you run a risk of turning it into pal review–climate change, a field mentioned above, is part of geophysics. In a field as large and varied as medicine, the risk of pal review becomes overwhelming, simply because the field is too big for everybody to know everybody. Thus, as a quick perusal of Orac’s archives will demonstrate, pal review is a serious issue, resulting in obvious nonsense like homeopathy and theraputic touch showing up in peer-reviewed publications.

If you can’t evaluate whether a paper in the scientific literature is good or bad, is correct or wrong, then you should not use that paper in your research, and should not cite that paper in your publications.

The problem is that the scientific literature is being used for things that it was never intended to be used for, and which it is not suitable for. The number of citations a journal has is not a measure of the “scientific value” of articles published in that journal. The “scientific value” of a particular article can only be determined by those knowledgeable enough to understand the paper and its context in the scientific field it is in.

It would be “scientific misconduct” to pretend that the “scientific value” of a particular paper can be evaluated by where it is published. In other words, there is zero scientific bases for determining the scientific value of a paper based on where it is published.

I appreciate that the people using citation index metrics to evaluate “scientific value” are not scientists. That doesn’t make the process acceptable. It means the evaluations have even less value.

Less than zero value means the process is hurting scientific progress.

I got your point. I even agree with it. I just think that you’re being way too easy on the editors involved. IMHO an editor who doesn’t know who the referee is, is an editor who has failed to exercise due diligence. I know that Chinese names, particularly Chinese names rendered in the Roman alphabet, can be ambiguous. There are astronomers who use identical renderings of their names who have nothing to do with one another. My point is that if you haven’t taken the time to distinguish between people then you’re not doing your job as an editor.

It’s yet another argument for getting journals to use ORCID so that authors have unique identifiers, but there will always be people who haven’t gotten one yet. People have to do their jobs for the journals to work.

Full disclosure: I work in a relatively small field (astrophysics) so this may affect my outlook.

Cheers
Ethan Vishniac