Development of bibliometric indicators of research quality
The United Kingdom, through the Higher Education Funding Council for England, has worked since 2005 to reform the evaluation of research, known as the RAE (Research Assessment Exercise). The latest study, published in a report in September 2008, sought to analyze and deploy a database containing all academic and publications submitted to evaluation committees during fiscal 2001. Commissioned by HEFCE and conducted by the Center for the Study of Science and Technology (CWTS, Leiden University, Netherlands), the authors of this study explored and tried to solve a series of technical problems in optics developing methods to produce bibliographic appropriate indicators of research quality. The objective is to advise HEFCE on choice and use of such indicators in a pilot exercise to be implemented in the near future.
The areas of research, academic publications and evaluated were classified into 8 groups. Six together the hard sciences (clinical medicine, health sciences, disciplines related to health, biology, physics and engineering and computer science) and the two remaining groups are math and social sciences and humanities.
The issues raised in this report were grouped into three sections: issues of methodology, discipline groups and their aggregation and sensitivity analysis.
1.1 Relevance of Web of Science (WoS)
To start, the authors created a database of researchers and publications from the complete list of departments for the RAE 2001, which had been provided by HEFCE. In the category of “best” publications submitted to RAE 2001 articles published in scientific journals are on average 73% of all publications submitted for evaluation. Cette proportion atteint 92 % en sciences dures et en mathématiques. In contrast, social sciences and humanities, books and book chapters represent 15 and 24% respectively.
On all publications submitted to evaluation committees, 84.1% were published in journals listed in the WoS for the hard sciences, 81.8% for mathematics and 24.9% for social sciences and humanities . In the hard sciences, the range of WoS coverage extends from 97% for clinical medicine at 70% for engineering and computer science. It is noteworthy that in computing and engineering, sub-disciplines are remarkably heterogeneous in terms of the WoS coverage: 44% in computer engineering, 86% for electrical and electronic engineering and 88% for chemical engineering.
1.2 Correlation between the impact of quote “best” publications and the classification of RAE 2001
The analysis of correlation between the impact of quote “best” publications submitted for evaluation by a university department on the one hand, and the classification of the same department received the 2001 RAE the other hand, focused on publications of the period from 1996 to 2000 and citations periods of four years of each publication. According to a global trend, citing the impact of increases in parallel with increasing rank of the department in EIR. The authors suggest however that there are exceptions. In the group of engineering and computer science for example, departments rated 2, 3a, 3b, 4 and 5 in RAE 2001 obtain quotations impacts very similar. Only those departments rated 5 * get a higher citation impact to others. This distribution is also found in group medical clinic in this case, however, the impact of citation are much higher than those observed in Group Engineering Sciences and Informatics. Therefore, the authors suggest that if the HEFCE wishes methods of peer review used for the RAE 2001 represent the baseline measurement to validate the newly developed metric indicators (for the years ahead score), the approach of relying on the citations of “best “publications of a department on the 4-year period following the publication is only partially valid in engineering and computer science and clinical medicine, because it masks the incremental intrinsic value of a department. However, it allows to distinguish the best from other departments. In most cases, the citation impact differs substantially if the authors took into account not the “best” but all publications.
1.3 Analysis of citations of publications published in journals not referenced in the WoS
This analysis indicated that the number of citations is higher for a book to an article referenced in the WoS. However, it is lower for book chapters, conference proceedings and items not listed in the WoS. In engineering and computer science, conference proceedings and items not listed in the WoS are much less cited than those listed in the WoS. It is noted that an analysis of publications not referenced in the WoS can be automated so that the need for human intervention limits the number of publications that can be incorporated into the analysis.
1.4 Collection of exhaustive lists of publication of staff submitted to RAE 2001
The report’s authors have developed a technique extremely sensitive and almost completely automated allowing them to identify all publications referenced in the WoS for a given author. This technique, which takes into account the homonyms and synonyms, showed high precision but low recall, especially for authors with only few publications referenced in the WoS.
During the analysis of a sample of publications collected in several major British universities, the authors found that in 70% of cases, the address of the institution indicated in the article matched the address of the researcher at the time of evaluation. In the subgroup of “best” publications referenced in the WoS, this proportion rose to 78%. In other words, between 20 and 30% of publications submitted to evaluation by a department are the result of work that has not been done in the department. It is therefore crucial that the evaluation takes into account these differences and that is decided, upstream of the year, what approach will be used: the past performance of a department or those active staff in the department at evaluation.
2. Combinations of disciplines and aggregation
2.1 Appointment of groups of subjects
Initially, the authors found a correlation between categories of journals listed in the WoS and the group of scientific disciplines of the RAE 2001. Each article submitted for evaluation and was assigned to a group of scientific disciplines, according to the newspaper in which he had been released. This correlation was then enlarged using co-citations and redistributing each article published in a newspaper multidisciplinary in unique categories of newspapers, on the basis of the nature of newspapers cited references. For example, an article mentioning mostly articles published in journals of astronomy would be classified in astronomy. Whatever happens, an article will often be classified in more than one category because the boundaries between disciplines are increasingly blurred.
2.2 Distribution of researchers in the groups of subjects
The authors used two methods to classify the researchers evaluated each based on recent publications: classification of the researcher in the group of disciplines in which he introduced more publications; splitting proportional to the researcher’s classification between groups of subjects in which publications are classified. The correlation of the two methods shows that only 77% of results are correlated in Engineering Sciences and Informatics, 75% in physical sciences, 68% in biology, 67% in health sciences, 65% medicine clinic, 64% in social sciences and humanities, 58% in sciences related to health and 55% in mathematics.
2.3 Aggregation of bibliometric indicators in groups of subjects
The profiles of citation impact of an institution or group of disciplines clearly reflect the distribution of staff submitted to the RAE and publications, regardless of the impact factor of journal in which the article was published. They bring a new vision of comparison between institutions and the differences that occur when changing the period analyzed. The authors developed a method of aggregation of bibliometric indicators by group of scientific disciplines. This method is based on research and publications. The steps are:
- Assign to each researcher or publication of a category of citation impact;
- Allocate a value to each impact category;
- Calculate the total value of all researchers or publications by institution and by discipline group;
- Finally, calculate the share of each individual publication or within an institution and compare the total number of individuals or items in a given group of subjects.
3. Sensitivity analysis
A framework is established to review the relevance of the results depending on a variety of methodological variations. This framework creates profiles of groups of scientific disciplines for a series of bibliometric indicators, profiles reflect the distribution of a performance indicator among institutions. These profiles are compared in pairs.
Minimal differences were observed when compared pairs of indicators are the number of articles published between 1992 and 2001 on the one hand, and those published between 1997 and 2002 on the other. In contrast, larger differences were demonstrated when comparing the total number of researchers to researchers assessed by the RAE, and the value of impact category of quote attributed to them. This difference suggests that consideration of the impact of quote allocated to researchers evaluated the influence score of the institution. The results are substantially different from those obtained when only the volume indicators is taken into account.
Comparing the number of researchers with a value of citation impact and assigned the number of articles published in the top 5% merit special attention: if both indicators are a selection of excellence, but they differ in approaches used first, and from one institution to another, on the other. It is therefore important to decide what upstream approach will be most appropriate to the desired results.
Leave a Reply