Abstract:

In this study, it was aimed to compare the coefficients of interrater agreement according to classical statistic theories. The difference between the calculated agreement coefficients according to different scaling types has been focused and the importance of deciding on the scaling type has been revealed. It was examined whether the change in the number of raters and rateds affected the values, and the most appropriate sample size to be used for the interrater agreement was calculated by using the optimization analysis of the generalizability theory. In the study, 9 cards of the Bender-Gestalt motor skill test, which can be seen by everyone, were applied to 35 children in total, and the measurements were evaluated by 8 raters blindly to each other. Accordingly, the highest agreement value was calculated in the intra-class correlation coefficient and this value was followed by Krippendorff alpha, Fleiss kappa and Cohen kappa, respectively. It has been determined that as both the number of rateds and raters decrease, the agreement values tend to decrease. On the other hand, it was determined that the difficulty level of the cards did not have a significant effect. The high generalizability coefficients showed that the test was reliably scored by the raters. When the optimization analysis was examined, it was seen that the most suitable sample size for this test was 50. Having more than 50 participants did not increase agreement.

Keywords: