Inter-rater agreement scores for screening
| Screening stage | Rater 1 | Rater 2 | Inter-rater agreement (%) | Expected agreement (%) | Cohen’s kappa | SE |
| Title and abstract | AAS | KL | 71.91 | 55.31 | 0.4367 | 0.0804 |
| AAS | MS | 83.03 | 55.31 | 0.6247 | 0.1025 | |
| MS | KL | 84.27 | 55.31 | 0.6480 | 0.1060 | |
| Full text | MS | NK | 92.86 | 56.38 | 0.8363 | 0.1890 |
Pilot phase for title and abstract screening carried out over 90 papers with each reviewer assigned to 60 papers, with overlap such that each paper was scored by two reviewers.