Measuring research performance – do only numbers count?

DFG launches measures for fairer evaluation. Researchers at FAU tell us what they think.

Which criteria can be used to evaluate and assess how successful a research achievement is? Researchers from all disciplines rely on their work being evaluated fairly and correctly, as this determines not only their reputation in research but also significant amounts of their funding. Ultimately, this decides on their future in research. For quite some time, the German Research Foundation (DFG) has been searching for a fair method of evaluating research performance.

Should research performance be based on quantitative data alone such as the number of publications? What role does qualitative data have to play? Do special circumstances need to be considered such as a researchers raising a child or having to earn a living alongside their research career, as in these situations the number of publications is likely to be lower?

Following the publication of a position paper in early summer, the DFG has announced a package of measures including requiring the submission of a curriculum vitae where researchers can indicate periods when special circumstances may need to be considered. Moreover, the review and evaluation of an application should have a stronger focus on the content of previous work, rather than the number of publications or the h-index.

Will these measures really lead to change or will they come to nothing? We asked researchers at FAU what they think.

Andreas Hirsch, Vice President People and Chair of Organic Chemistry II at FAU.

“The DFG initiative is most welcome. Considering quantitative publication data such as the number of publications, impact factors of journals or the h-index is not an objective measure of a researcher’s performance. There are a wide range of other qualitative factors that determine the value of a researcher’s contribution. These include academic prizes, acquisition of third-party funding, memberships in scientific academies, sustainability of research, invitations to presentations at leading conferences, expert roles, contributions to subject committees, membership in expert panels such as the DFG or ERC or mentoring young researchers on a national and international level.

The importance of individual qualitative factors differs from subject to subject. However, experience has shown that outstanding researchers are usually equally well represented in several or even most of these criteria. The initiative will certainly contribute to helping appointment committees or funding organizations to make more objective decisions.”

Kathrin Castiglione, Chair of Bioprocess Engineering

“In my view, improving equal opportunities by ensuring standardized criteria is a welcome effort. Moving away from the emphasis on abstract numbers such as the h-index to a comprehensive evaluation of individual research performance is an important step in the right direction. In my subject group Bioprocess Engineering at the DFG, applications from women still have a lower approval quota than men although the application and research proposals are not lacking in quality. Under these circumstances, standardized criteria for evaluating research careers and performance to consider special circumstances and contributions to committee work, which women are over proportionally involved in, could only be considered an improvement.”

Johannes Fürst: Head of junior research group and ERC grant recipient. Research topic: The strategy for the future for global alpine glacier prognoses

“Certainly, the possibility of narrative sections in the curriculum vitae, as well as the expansion of the permitted forms of publication, facilitates a more rapid assessment of applicants’ careers. Previously, if this information was relevant, it was often highlighted prominently in the application itself. I see very limited benefits in this step by the DFG, although it may help to emphasize supplementary information on special circumstances. In my opinion, this step will have little impact on a proposal that is compelling in terms of content. The curriculum vitae is and should continue to be considered as supplementary information.”

Matthias Braun, Head of junior research group “Ethics and Governance”

“The initiative to improve equal opportunities in research assessment is in itself very welcome. It addresses an essential topic, especially for young researchers: Which research achievements count and can be included in the evaluation process? Improving the visibility and ability to assess work that is essential for maintaining the research system but achieves little recognition is an excellent starting point in my opinion. However, it is difficult to predict what impact this measure will actually have. Essentially, it is just changing the template for the researcher’s curriculum vitae. That is a good thing as it allows individual circumstances to be considered. But we will need to see whether this is better and fairer than quantitative metrics such as the h-factor. Quite honestly, I am skeptical of the outcome and I don’t believe that quantitative and qualitative factors are mutually exclusive. Even if the evaluation has a greater focus on content and qualitative factors, reviewers still need criteria that they can base their evaluation on. Maybe this can be made more explicit.

In my opinion, a further step towards more equal opportunities would mean that in addition to the changes that have now been made in the presentation options for applicants, the assessment process itself and the communication and transparency in the transmission of the results of the assessment are evaluated and adapted.”

Imke Leicht, Head of the Office of Equality and Diversity

The introduction of the curriculum vitae template with voluntary appraisal-relevant information on career history enables an evaluation that is more closely related to individual circumstances. Here, different diversity aspects can come into play such as family care, disability or illness, social background, flight or asylum experiences that are important to consider with regard to leave, delays or special circumstances in research careers but also in terms of experiences and skills. Indicating diversity-relevant aspects in the curriculum vitae can, if desired, be taken into account accordingly in a positive way or be recognized as part of research performance. Individuals still have the choice not to state reasons for gaps in their career should they wish to do so.

With this initiative, the DFG creates space for diversity in science and heterogeneous pathways among researchers, which is reflected in the diversity of their background and life experiences. It is important here that certain circumstances are not interpreted to the disadvantage of the applicant, but in the sense of equal opportunities.

In order for this to be implemented accordingly, reviewers, board members, and committee members, for example, should urgently be sensitized to the different life situations and diversity aspects and trained in how these can be appropriately taken into account; otherwise, I see the danger that discriminatory evaluation patterns will creep in again due to (unconscious) biases. We should continue to monitor how the DFG measures are implemented.”