Do you sometimes find yourself in a situation where people expect you to undertake a very daunting task despite your lack of prior experience?
Well, I know I do. And although it would be tempting to pull that joke - no, this isn't a cheeky metaphor for my PhD studies. At least not an intentional one.
Instead, I am alluding to something much more specific: the mysterious process of becoming a referee.
As usual, these things happen to me somewhat out of the blue, and so, two months after commencing my PhD studies, I opened my mailbox just to see that I was selected to file my first referee report! I felt really excited, but this feeling was quickly overridden by the sense of utter insecurity as I had no idea whether I was sufficiently qualified for the job. Sure, in my prior work I used similar estimation techniques as the ones in the questioned paper, but still - could I rigorously asses merits of another person's work? Even more so if it is a Professor? Clearly, the seed of doubt had been planted....
Nevertheless, I am always up for a challenge, so I agreed to write the report. The whole task was however still very obscure to me, and therefore I started gathering information about the do's and dont's of refereeing. Apart from some referee reports I had in hand, I made a good use of advices provided by JME, IAAE, and this wonderful piece by Dave Harrison. If you're in a similar position like me, I strongly recommend reading through these works carefully.
After absorbing all the recommendations, I have begun with the assessment itself. First read-through revealed that the paper has some potential, but handful of considerable flaws in empirical analysis marred the overall picture. I guess this happens way too often, but still - such an outcome was not what I was hoping for. Unlike with extremely good (or bad) papers, you cannot just write an exalted acceptance letter (or a resolute rejection pamphlet). You have to ponder whether the paper is good enough to be let through (after some minor/major changes), or whether the author would be better off starting from a scratch, or maybe switching to a more fitting journal.
Sounds hard? Well don't forget that this dilemma is further obfuscated by the fact that you cannot check validity of either the data or the codes which back up the analysis in question.
...boy, deciding between the recommendation statements is tough! I was contemplating either a revision or a weak rejection, and you can imagine that my initial sense of insecurity was now beyond belief... assessing the relevance of one's paper is one thing, but claiming it is not worth publishing? That's harsh.
Now, I am not going to disclose my eventual decision, but I will share a piece of advice which helped me to make up my mind:
If you're not sure whether the paper can make it to the publishing phase, try to imagine how much time it would take to transform it into an acceptable manuscript (provided all the results remain valid). If it is approaching the time you deem necessary to write up a similar, yet publishable paper, then it's probably better to reject. If, however, the author could be better off retaining his project and changing it here and there, suggest which changes are bound to be made and ask for a revision.
Closing thoughts? Refereeing is a daring task, especially if you want to do your job well. Given the time it takes, it is hardly a surprise that many academics turn the requests down and consider the whole business as nothing but a tiresome nuisance.
However, every coin has two sides and I believe that the refereeing can be very enlightening, especially for PhD students. Firstly, by assessing the paper you are bound to develop better sense of critical thinking. Secondly, scrutinizing the paper's subject matter will broaden your research scope. But most importantly, the whole experience will improve your own journal submissions.
After finishing the first report, I browsed through my older works and realized that I see them in an entirely different light. I could pinpoint many of the little flaws which were bound to displease prospective referees and tilt their decision towards rejection.
Obviously, I have tried my best to correct these flaws, and my fist submitted paper is currently under review...so let's see whether that refereeing gig paid off!
