NEW DELHI: Lack of mechanisms to directly assess teaching quality, huge fluctuations in rankings, inadequate transparency in methodology and reliance on self-reported data, are among the inconsistencies in the education ministry’s ranking framework for higher education institutions, according to a research paper. Authored by former IIT-Delhi director and current vice chancellor of Birla Institute of Technology and Science (BITS) V Ramgopal Rao, the paper has been published in the ‘Current Science’ journal.It has been co-authored by Abhishek Singh, also from BITS, Pilani.
The ninth edition of the National Institutional Ranking Framework (NIRF) was announced earlier this month.
“While the NIRF rankings aim to enhance transparency and accountability, the present study has identified several inconsistencies, thus raising concerns about their reliability,” said the paper titled ‘Unpacking inconsistencies in the NIRF rankings’.
“These include huge fluctuations in the rankings, overemphasis on bibliometrics neglecting non-traditional research outputs, subjective nature of perception rankings that introduces biases, challenges in the regional diversity metric, overlooking teaching quality, inadequate transparency in methodology, questions about data integrity and limited global benchmarking,” it noted.
The academicians noted that NIRF rankings lack specific mechanisms to directly assess teaching quality, overlooking crucial aspects such as classroom observations, student evaluations and alumni feedback.
“The omission of these evaluation methods hinders a comprehensive assessment of teaching effectiveness, leading to an incomplete depiction of the educational prowess of an institution.
“Moreover, the NIRF rankings overlook the practical dimension of teaching, a crucial aspect in various disciplines,” the paper said.
The study has also flagged the variability in the positions of institutions from year to year.
“While some fluctuations may be attributed to genuine changes in performance, others might result from factors beyond the control of an institution, such as temporary variations in data reporting or interpretation errors.
“Unlike some international ranking systems, such as the QS World University Rankings, which utilise a damping mechanism to spread large, interannual swings in data, the NIRF rankings lack a similar mechanism,” it said.
The absence of a comparable mechanism in the NIRF framework raises questions about the ability of the system to correct for anomalies and errors, potentially impacting the stability and reliability of the rankings, the study said.
The paper noted that reliance on self-reported data raises pertinent questions regarding the consistency and accuracy of the information presented.
“Institutions varying in size, structure and resources may interpret and report data differently, potentially leading to disparities in the ranking outcomes. The absence of stringent mechanisms for verifying the accuracy and uniformity of the submitted data introduces an element of uncertainty into the rankings,” it said.
Without standardised reporting practices, the rankings may inadvertently favour institutions adept at presenting data in a favourable light rather than those genuinely excelling in academic parameters.
This, in turn, has profound implications on the credibility of the rankings, as their reliability depends on the accuracy and consistency of the data underpinning the evaluation process, it added.
The study noted that while the NIRF rankings have certainly proven to be a a valuable tool for assessing and comparing educational institutions in India, a cautious and discerning approach is essential.
“The identified inconsistencies underscore the need for an ongoing dialogue and refinement of the ranking framework. It is essential to acknowledge that the rankings, by their nature, subtly influence perceptions.
“Thus, the identified issues, if left unaddressed, may impact the credibility and relevance of the NIRF rankings, potentially affecting the perceptions of stakeholders such as students, parents and policymakers,” it said.
The ninth edition of the National Institutional Ranking Framework (NIRF) was announced earlier this month.
“While the NIRF rankings aim to enhance transparency and accountability, the present study has identified several inconsistencies, thus raising concerns about their reliability,” said the paper titled ‘Unpacking inconsistencies in the NIRF rankings’.
“These include huge fluctuations in the rankings, overemphasis on bibliometrics neglecting non-traditional research outputs, subjective nature of perception rankings that introduces biases, challenges in the regional diversity metric, overlooking teaching quality, inadequate transparency in methodology, questions about data integrity and limited global benchmarking,” it noted.
The academicians noted that NIRF rankings lack specific mechanisms to directly assess teaching quality, overlooking crucial aspects such as classroom observations, student evaluations and alumni feedback.
“The omission of these evaluation methods hinders a comprehensive assessment of teaching effectiveness, leading to an incomplete depiction of the educational prowess of an institution.
“Moreover, the NIRF rankings overlook the practical dimension of teaching, a crucial aspect in various disciplines,” the paper said.
The study has also flagged the variability in the positions of institutions from year to year.
“While some fluctuations may be attributed to genuine changes in performance, others might result from factors beyond the control of an institution, such as temporary variations in data reporting or interpretation errors.
“Unlike some international ranking systems, such as the QS World University Rankings, which utilise a damping mechanism to spread large, interannual swings in data, the NIRF rankings lack a similar mechanism,” it said.
The absence of a comparable mechanism in the NIRF framework raises questions about the ability of the system to correct for anomalies and errors, potentially impacting the stability and reliability of the rankings, the study said.
The paper noted that reliance on self-reported data raises pertinent questions regarding the consistency and accuracy of the information presented.
“Institutions varying in size, structure and resources may interpret and report data differently, potentially leading to disparities in the ranking outcomes. The absence of stringent mechanisms for verifying the accuracy and uniformity of the submitted data introduces an element of uncertainty into the rankings,” it said.
Without standardised reporting practices, the rankings may inadvertently favour institutions adept at presenting data in a favourable light rather than those genuinely excelling in academic parameters.
This, in turn, has profound implications on the credibility of the rankings, as their reliability depends on the accuracy and consistency of the data underpinning the evaluation process, it added.
The study noted that while the NIRF rankings have certainly proven to be a a valuable tool for assessing and comparing educational institutions in India, a cautious and discerning approach is essential.
“The identified inconsistencies underscore the need for an ongoing dialogue and refinement of the ranking framework. It is essential to acknowledge that the rankings, by their nature, subtly influence perceptions.
“Thus, the identified issues, if left unaddressed, may impact the credibility and relevance of the NIRF rankings, potentially affecting the perceptions of stakeholders such as students, parents and policymakers,” it said.
