Abstract:

The goal of this study was to develop a "Textuality Rubric in PISA Criterion" (TRIPC) in order to evaluate schoolbook texts. The data were obtained by using the analytical rubric developed for the study in order to score 20 reading texts with continuous text characteristics of PISA which were included in Turkish textbooks used in 5th, 6th, 7th, and 8th grades during the 2019-2020 school year. Cognitive processes and reading proficiency levels within the framework of PISA 2018 reading skills assessment were taken into consideration when setting the criteria of the analytical rubric. Three different degrees of success were defined for each criterion. First, 8 experts were consulted in order to obtain validity proof for the developed analytical rubric. Content validity rates were calculated for each criterion based on the responses of these experts and a principle components analysis was also conducted in order to obtain proof of validity. Intraclass correlation coefficient and Cronbach alpha value calculated for proof of reliability for the analytical rubric were 0.96 and 0.96, respectively. Weighted Kappa was calculated by using scores given by two Turkish teachers in order to obtain proof of inter-rater reliability. According to the obtained weighted Kappa coefficient test, the analytical rubric criteria showed good and very good agreement. "Textuality Rubric in PISA Criterion" consisting of 3 main criteria, namely "obtaining information”, "understanding” and "reflecting and evaluating” and 8 sub-criteria were developed. It was concluded that the analytical rubric developed is a highly valid and reliable measurement tool.

Keywords: