Statistical and qualitative analysis of ChatGPT and human raters in preservice teachers' writing assessment
Küçük Resim Yok
Tarih
2026
Dergi Başlığı
Dergi ISSN
Cilt Başlığı
Yayıncı
Izzet Kara
Erişim Hakkı
info:eu-repo/semantics/openAccess
Özet
Teachers spend a significant amount of time providing feedback. This study compared expert and ChatGPT assessments and feedback on written texts to determine the suitability of AI for writing skill assessments that are time-consuming to assess and provide feedback. Three experts and ChatGPT graded 14 Turkish undergraduate students' assignments using rubric that included content, language use, vocabulary, organization, and mechanics, and justified their decisions. The study involved document review and triangulation, a qualitative design. In addition, an intraclass correlation coefficient was used to assess the consistency of the ChatGPT and the experts' scores. All feedback was qualitatively analyzed to identify the strengths and weaknesses of the experts and their similarities with ChatGPT. Experts and ChatGPT had moderate to weak consistency in the writing subscales, while good reliability was found in the total score. Experts excelled in 'explanatory feedback', 'interpretation' and 'experience', while ChatGPT excelled in 'automation and continuity' and 'data processing capacity'. Experts' weaknesses included 'limited time and energy' and 'comparison bias', while ChatGPT's weaknesses were 'ambiguous expressions' and 'repetition'. The study also found that experts and ChatGPT preferred to provide constructive and supportive feedback.
Açıklama
Anahtar Kelimeler
Artificial Intelligence, ChatGPT, Writing feedback, Human-raters
Kaynak
International Journal of Assessment Tools in Education
WoS Q Değeri
Q3
Scopus Q Değeri
Cilt
13
Sayı
1
