Evaluating evaluators: a neglected requirement

Abstract

Evaluation was considered a permanent practice in and on universities since the 1990s, and -of course- it was prior practice at other levels, such as thesis defense processes, or the acceptance of articles for scientific journals, mechanisms that acquired greater formalization since then. With Covid-19, the "slowdown" promotes the possibility of reflection on such institutional practices. The old question about "who evaluates the evaluator" returns, because surprisingly evaluators enjoy enormous impunity in their decisions: nobody evaluates their activities, which allows all kinds of relatively open arbitrariness, in addition to the involuntary errors that always exist. Personal revenges, settling accounts between rival academic tribes, differences in theoretical position and even professional jealousies are discharged against the authors of scientific articles, and even more so against thesis students who often receive the attacks that are dedicated to their directors, because in the Thesis cases the names of the academics involved are public. It is time to propose statistical sampling modalities that systematically select some evaluations, and examine what has been done by the evaluators, with proportional penalties in cases where obvious distortions in the actions are discovered. It is surprising that to date these mechanisms for cross-checking the activity of academics have not been stipulated.