Evaluation Methods for R&D in European Countries
https://doi.org/10.15826/umpa.2019.04.025
Abstract
The paper is a case study of the experience of developing and using systems for evaluating the quality and effectiveness of scientific research within British, Dutch and Finnish universities and other fundamental scientific centers. The article aims at showing key organizational and methodical approaches for assessing the effectiveness of research in Western European countries against the background of the ensuing discussion about the role of quantitative and peer review methods in analyzing research results. Descriptive, historical and comparative methods are used. The assessments of quality and the state of research in the selected group of countries are found to be based mainly on the complexity and ambiguity of scientific pursuit and, consequently, on understanding that the evaluation of research effectiveness should be multidimensional and complex, that it should rest on expert quality assessment and quantitative (scientometric and bibliometric) indicators, the latter not to be dominant, but to serve as an advantage to expert judgment. The article gives new detailed and more complete (than the analogical works) analysis of the systems and criteria for assessing the quality of research together with the factors underlying their evolution in the three European countries with highly developed national scientific systems. There is shown the formation of modern evaluation systems in accordance with the needs for the development of scientific potential and with the need to strengthen its competitiveness on the international arena. This made it possible to draw valid conclusions about the essential general features of the assessment systems and about their adaptation to national characteristics, and to formulate recommendations on using the experience of Western European countries to improve the practice of evaluating research activities in Russian universities and research institutes. The results of the work may as well be of interest to researchers, university professors, postgraduates, students, and also to all those interested in topical issues of developing the science of science.
About the Authors
I. V. KirichenkoRussian Federation
Irina V. Kirichenko – PhD (Economics), Senior Researcher, Department for Science and Innovation
23 Profsouznaya str., Moscow, 117997
N. V. Shelyubskaya
Russian Federation
Natalia V. Shelyubskaya – PhD (Economics), Senior Researcher, Department for Science and Innovation
23 Profsouznaya str., Moscow, 117997
References
1. The San Francisco Declaration on Research Assessment (DORA), available at: https://sfdora.org/ (accessed: 24.05.2019). (Eng.).
2. The Leiden Manifesto for research metrics. Nature, April 2015, vol. 520, pp. 429–431., available at: https://www.researchgate.net/publication/275335177 (accessed: 24.05.2019). (Eng.). DOI: 10.1038/520429a.
3. Benedictus R., Miedema F., Ferguson M. J. Fewer numbers, better science. Nature 27 October 2016, vol. 538, iss. 7626, pp. 453–455., available at: https://www.nature.com/news/fewer-numbers-better-science-1.20858 (accessed: 24.05.2019). (Eng.). DOI:10.1038/538453a
4. Wilsdon J., et al. The Metric Tide. Report of the Independent Review of the Role of Metrics in Research Assessment and Management. July 2015, 163 p., available at: https://blogs.lse.ac.uk/impactofsocialsciences/files/2015/07/2015_metrictide.pdf (accessed: 24.05.2019). (Eng.). DOI: 10.13140/RG.2.1.4929.1363
5. Next-generation metrics: Responsible metrics and evaluation for open science. Report of the European Commission Expert Group on Altmetrics. EK, Directorate-General for Research and Innovation, 2017, 26 p., available at: https://ec.europa.eu/research/openscience/pdf/report.pdf (accessed: 24.05.2019). (Eng.). DOI:10.2777/337729
6. Sivertsen G. Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective. Palgrave Communications. Published: 15 August 2017. 3:17078, available at: https://www.nature.com/articles/palcomms201778 (accessed: 23.05.2019). (Eng.). DOI: 10.1057/palcomms.2017.78
7. Zacharewicz T., Lepori B., Reale E., Jonkers K. Performance – based research funding in EU Member States – a comparative assessment. Science and Public Policy, 2019, vol. 46, no. 1, pp. 105–115, available at: https://academic.oup.com/spp/article/46/1/105/5037253 (accessed: 23.05.2019). (Eng.). DOI.org/10.1093/scipol/scy041
8. Parshin A. N. Nauka ili bibliometriya: kto kogo? [Science or bibliometry: who will win?]. Herald of the Russian Academy of Sciences, 2018, vol. 88, no 11, pp. 982–984. DOI: 10.31857/S086958730002330-1 (In Russ.).
9. Ivanchik A. I. Osobennosti otsenki issledovatelei i issledovatel’skikh programm v gumanitarnykh naukakh [Particular properties of evaluation of researchers and research programs in the humanities]. Herald of the Russian Academy of Sciences Herald of the Russian Academy of Sciences, 2018, vol. 88, no. 11, pp. 985–991. DOI: 10.31857/S086958730002331-2 (In Russ.).
10. Grinyov A. V. Nauchnye publikatsii i naukometricheskie pokazateli kak ob»ekt nechistoplotnogo biznesa [Scientific publications and scientometric indicators as an object of unscrupulous business]. Herald of the Russian Academy of Sciences Herald of the Russian Academy of Sciences, 2018, vol. 88, no 10, pp. 908–917. DOI: 10.31857/S086958730002147-9 (In Russ.).
11. Global Innovation Index 2018: Energizing the World with Innovation. Cornell University, INSEAD, and WIPO (2018): Ithaca, Fontainebleau, and Geneva. 430 p. ISBN 979-10-95870-09-8, p. xx, available at: https://www.wipo.int/edocs/pubdocs/en/wipo_pub_gii_2018.pdf (accessed: 24.05.2019). (Eng.).
12. Gross domestic expenditure on research and development, UK: 2017. Office for National Statistics, 14 March 2019, 13 p., available at: https://www.ons.gov.uk/economy/govern-mentpublicsectorandtaxes/researchanddevelopmentexpenditure/bulletins/ukgrossdomesticexpenditureonresearchanddevelopment/2017 (accessed: 24.05.2019). (Eng.).
13. QS World University Ratings 2019, available at: https://www.topuniversities.com/university-rankings/world-university-rankings/2019 (accessed 25.06.2019). (Eng.).
14. Times Higher Education World University Rankings, available at: https://www.timeshighereducation.com/world-university-rankings/2019/world-ranking#!/page/0/length/25/sort_by/rank/sort_order/asc/cols/stats (accessed 23.06.2019). (Eng.).
15. CWUR World University Rankings 2018–2019, available at: https://cwur.org/2018–19.php (accessed 23.06.2019). (Eng.).
16. Independent report. Research Excellence Framework review. An independent review of university research funding by Lord Nicholas Stern. Published 28 July 2016. Last updated 31 December 2018. From: Department for Business, Energy & Industrial Strategy. Research Excellence Framework Review. Evidence Report. Technopolis group, October 2018, 116 p., available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/768162/research-excellence-framework-review-evidence-report.pdf (accessed: 23.05.2019). (Eng.).
17. Independent report. Research Excellence Framework review. An independent review of university research funding by Lord Nicholas Stern. Published 28 July 2016. Last updated 31 December 2018. From: Department for Business, Energy & Industrial Strategy. Research Excellence Framework (REF) review: Building on success and learning from experience. An Independent review of the Research Excellence Framework, July 2016, p. 6, available at: https://www.gov.uk/government/publications/research-excellence-framework-review (accessed: 23.05.2019). (Eng.).
18. Research Assessment Exercise 2014. Panel 10 – Social Sciences. Specific Criteria and Working Methods. (August 2013), available at: https://www.ugc.edu.hk/doc/eng/ugc/rae/psg_10_201308.pdf. (accessed: 23.05.2019). (Eng.).
19. Total R&D in the Netherlands by Sector of Performance and Source of Funds, R&D Activity Type and Type of Costs. Rathenau Institut, available at: https://www.rathenau.nl/en/science-figures/investments/how-much-does-netherlands-spend-rd/total-rd-netherlands-sector (accessed 26.05.2019). (Eng.).
20. R&D Personnel and Researchers in the Netherlands by Sector. Rathenau Institut, available at: https://www.rathenau.nl/en/science-figures/personnel/rd-personnel-netherlands/rd-personnel-and-researchers-netherlands-sector (accessed 26.05.2019). (Eng.).
21. Standaard Evaluation Protocol 2015–2021. Protocol for Research Assessments in the Netherlands. The Netherlands, KNAW, VSNU, NWO, 2014. 31 p. (Eng.).
22. Elizabeth Gadd. Research Evaluation: Things We Can Learn fron the Dutch. available at: https://thebibliomagician.wordpress.com/2018/05/31/research-evaluation-things-wecan-learn-from-the-dutch/ (accessed 26.05.2019). (Eng.).
23. Research and Development in Finland. Helsinki, Academy of Finland, 2019. 9 p. (Eng.).
24. Sivertsen G. Why has no other European country adopted the Research Excellence Framework, available at: https://blogs.lse.ac.uk/politicsandpolicy/why-has-no-other-european-country-adopted-the-research-excellence-framework/ (accessed: 23.05.2019). (Eng.).
25. Tieteen Tila 2018. Helsinki, Suomen Akatemia, 2018. 84 p.
26. Tieteen Tila 2016. Helsinki, Suomen Akatemia, 2016. 68 p.
27. The State of Scientific Research in Finland, 2012. Helsinki, Academy of Finland, 2012. 94 p. (Eng.).
28. Publication Forum has completed its review of ratings, available at: http://www.julkaisufoorumi.fi/fi/arkisto/tiedotteet/publication-forum-has-completed-its-review-ratings (accessed 26.05.2019). (Eng.).
29. Ilkka Hemmilä. Finnish Scientific Funding Favours Some Disciplines over Others. available at: https://www.tylkkari.fi/english/finnish-scientific-funding-favours-some-disciplines-over-others (accessed 26.05.2019). (Eng.).
Review
For citations:
Kirichenko I.V., Shelyubskaya N.V. Evaluation Methods for R&D in European Countries. University Management: Practice and Analysis. 2019;23(4):9-20. (In Russ.) https://doi.org/10.15826/umpa.2019.04.025