Introduction: Communication skills assessment is complex. Standardised patient use is widespread, but anxiety exists around the use of role players as assessors of competence in high stakes examinations. This study measures the level of agreement between scoring examiners and role players, and considers their influence on each other. Examiner status and question choices are analysed as variables.
Method: The valid oral interactive contextualised examination (VOICE) is a general practice examination styled as an objective structured clinical examination (OSCE) of six 15-minute stations, which include two role-played consultations with professional role players. The examination candidates are final year medical students. Clinical components are examined by a general practitioner (GP). Communication skills are assessed by these examiners in conjunction with the role players, through a process of negotiation. Descriptive professionalism/attitude bandings are used as percentage-scoring guidelines. Checklists are not used. For this study, the initial (independently) perceived marks of the two scoring groups and their agreed final (awarded) marks were recorded, along with other variables including gender, performance factors, demographics and the nature of the question. Data represents 512 students undertaking 1024 simulated consultations, examined by 28 role players and 46 examiners. Analysis was carried out using SPSS Version 10.
Results: Results show that the examination and negotiation process is consistent. Role players have a direct influence on scoring. The examiner's background is a significant variable [F9,1014 = 4.207, P < 0.001]. Students perform less well on questions involving higher degrees of clinical information giving. Question choice is not significant [F30,3039 = 1.397, P=0.074].
Discussion: The variables in the examination do not indicate any discrepancy substantial enough to bias a student's grade. Negotiated marking in this context is considered safe and reliable.