Scoring Short Answer Questions of Five Borderline Medical Students

Abstract

Background: The assessment of medical knowledge is integral to becoming a medical practitioner in Australia, and Short Answer Questions (SAQs) are frequently used in this process. This paper compares the use of Classical Test Theory (CTT) and the Rasch rating scale measurement framework in scoring SAQs to evaluate the competence of borderline candidates in Australian medical students. Aims: The aim of this study was to utilise two scoring paradigms to compare the results of borderline medical students on SAQs. Methods: Forty SAQs were administered to 140, fifth year medical students at an Australian university in an online practice examination. Aligned with CTT, each student’s performance was expressed as the sum of the question scores. The data was then also analysed within the Rasch rating scale measurement framework and measures of performance were obtained. The two sets of results were compared across borderline students. Results: According to CTT, five students were identified as being exactly at the pass mark of 50 per cent. Rasch analysis indicated however that although the students had the same ability estimates, their approach to answering SAQs were vastly different, altering the interpretation of their overall performance.  Conclusion: The sole use of CTT in the analysis of examination data may result in issues of validity and reliability when measuring clinical competence. The Rasch rating scale measurement framework may be invaluable in informing the analysis of performance in high stakes scenarios to ensure fair decisions of clinical competence.