Table 2

Inter-rater agreement scores for screening

Screening stageRater 1Rater 2Inter-rater agreement (%)Expected agreement (%)Cohen’s kappaSE
Title and abstractAASKL71.9155.310.43670.0804
Full textMSNK92.8656.380.83630.1890
  • Pilot phase for title and abstract screening carried out over 90 papers with each reviewer assigned to 60 papers, with overlap such that each paper was scored by two reviewers.