Korea Business Review

Korea Business Review

Edited by: Hyoung-Goo Kang Print ISSN: 2951-3596 Online ISSN: 2951-360X © THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION. Frequency: Quarterly (four times per year): February 28, May 31, August 31, and November 30
Cover
Latest Issue Vol. 60, No. 5 2025

Article Info

Comparing Native and Non-native Rater Assessments of Korean Oral Proficiency: A FACETS analysis

Kim, Hyunah

University of Toronto

Korea Business Review 51Vol. 5No. pp.83-113 (2016)

DOI: 10.20880/kler.2016.51.5.83

Abstract

Korean as a foreign language has experienced significant growth rates in the past few decades due in part to the worldwide popularity of Korean pop and entertainment culture, as well as its strengthening economic status. This new demand has led to a significant increase in the number of non-native Korean-speaking teachers. Despite their competence, many of these instructors report gaps in their confidence and abilities when it comes to evaluating students. Rater biases stemming from features present or absent in their linguistic and cultural backgrounds are of particular concern and may be more prevalent in assessments or oral proficiency. Despite the fact that rater bias in oral assessment has been explored extensively in the English-as-a-Foreign-Language context, there exists no relevant study of rater bias on speaking assessments in the Korean-as-a-foreign-language context. This paper reports the findings of an empirical study, which investigated possible systematic differences in evaluation of native Korean-speaking and non-native raters on speaking samples of learners of Korean and the effects of devices commonly used to enhance inter-rater reliability, such as detailed rubrics and rater training. The data were derived from four native and five non-native teachers of Korean, who were asked to evaluate five speech samples using both holistic and analytic rating scales. In the first round, the participants evaluated the samples without any rubrics. After being provided with detailed rubrics and rater training, the participants again engaged in the second round of rating using the same speech samples. The results, analyzed by many-facet Rasch measurement, revealed that the rating patterns of the two rater groups were significantly different in terms of severity, especially for holistic rating and accuracy and range features for analytic rating, while both groups maintained the acceptable level of rating consistency. The analysis also showed that rating criteria and rater training substantially settled the difference in rating severity between the two rater groups. The paper concludes with implications of the study on NNS raters’ rating pattern and future directions for rater training for Korean oral proficiency test.

Keywords

Korean speaking assessmentrater variablesrating patternsnative and non-native speakersNS and NNSmany-facet Rasch measurement

References

  1. [1] [학술지(정기간행물)] 백현영/중학교 영어교사의 말하기평가 채점경향 분석/언어과학/2011/18(4)/77~99/한국언어과학회/
  2. [2] [학술지(정기간행물)] Brown, A/The effect of rater variables in the development of an occupation-specific language performance test/Language Testing/1995/12/1~15//
  3. [3] [학술지(정기간행물)] Fayer, J. M/Native and nonnative judgments of intelligibility and irritation/Language Learning/1987/37/313~326//
  4. [4] [단행본] FACETS program for language education assessment research : basic/Jang, S. Y/Global Contents/2009/~///
  5. [5] [학술지(정기간행물)] 강석한/외국인 한국어 말하기 시험의 평가자 요소가 채점에 미치는 영향/이중언어학/2014/(55)/1~29/이중언어학회/
  6. [6] [학술지(정기간행물)] Kim, Hyun-Ju/Rater Reliability in L2 Oral Proficiency Tests/영어교육/2006/61(3)/105~118/한국영어교육학회/
  7. [7] [학술지(정기간행물)] Kim, Y. H/An investigation into native and non-native teachers’ judgments of oral English performance : A mixed methods approach/Language Testing/2009/26/187~217//
  8. [8] [학술지(정기간행물)] 이향/한국어 말하기 평가의 발음 영역 채점에서의 채점자 특성에 따른 채점 경향 연구 -한국어 교육 경험과 전공을 중심으로-/외국어로서의 한국어교육/2013/39/213~245/언어연구교육원 한국어학당/
  9. [9] [학술지(정기간행물)] 이향/한국어 학습자의 발화에 대한 한국어 교사와 비교사 원어민 간의 평가 차이 연구/한국어교육/2014/25(4)/163~188/국제한국어교육학회/
  10. [10] [단행본] A guide to English language testing/Lee, W/Moonjin Media/2012/~///
  11. [11] [단행본] Understanding language assessment/Lee, Y. S/Seoul National University Press/2003/~///
  12. [12] [학술지(정기간행물)] 이영식/다국면 Rasch 측정의 Facets 프로그램을 활용한 영어 작문 평가의 원어민 채점 검증/영어어문교육/2014/20(1)/475~496/한국영어어문교육학회/
  13. [13] [학술지(정기간행물)] 이영식/한국어 작문 평가 채점의 타당도 검증: 다국면 Rasch 측정 모형 활용/Foreign Languages Education/2014/21(3)/355~375/한국외국어교육학회/http://dx.doi.org/10.15334/FLE.2014.21.3.355
  14. [14] [학술지(정기간행물)] Lunz, M. E/Judge consistency and severity across grading periods/Evaluation & the Health Professions/1990/13/425~444//
  15. [15] [단행본] Assessing speaking/Luoma, S/Cambridge University Press/2004/~///
  16. [16] [단행본] Measuring second language performance/McNamara, T. F/Longman/1996/~///
  17. [17] [학술지(정기간행물)] Park, D. H/Question type development for CBT/IBT-based Korean proficiency test/National Institute for International Education/2012//~//
  18. [18] [학술지(정기간행물)] Shin, D. I/Exploring rating patters with Rasch measurement techniques: Implications for training/Foreign Language Education/2001/8/249~272//
  19. [19] [단행본] English language testing in Korea II: Speaking test/Shin, D. I/Hankook Munhwasa/2006/~///
  20. [20] [학술지(정기간행물)] Stassenko, I/Investigating cultural variability in rater judgements of oral proficiency interviews/Eesti Rakenduslingvistika Ühingu aastaraamat/2014/10/269~281//
  21. [21] [학술지(정기간행물)] Wei, J/Investigating Differences Between American and Indian Raters in Assessing TOEFL iBT Speaking Tasks/Language Assessment Quarterly/2015/12/283~304//
  22. [22] [학위논문] Yu, K. A/Effects of nonnative English-speaking raters’ oral proficiency levels on English speaking test ratings and rating processes/Ph. D/Ewha Woman’s University//2011
  23. [23] [학술지(정기간행물)] Zhang, Y/Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs/Language Testing/2010/28/31~50//

Korea Business Review · THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION.

Print ISSN 2951-3596 · Online ISSN 2951-360X

Privacy PolicyTerms of UseContact UsKMA Homepage

TEL : 010-2672-3919 EMAIL : kbr2editor@gmail.com 계좌번호 : 신한 140-009-175743 예금주 : (사)한국경영학회 COPYRIGHT ⓒ 2023 THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION. All rights reserved