Korea Business Review

Korea Business Review

Edited by: Hyoung-Goo Kang Print ISSN: 2951-3596 Online ISSN: 2951-360X © THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION. Frequency: Quarterly (four times per year): February 28, May 31, August 31, and November 30
Cover
Latest Issue Vol. 60, No. 5 2025

Article Info

Investigating Rater’s Cognitive Processes in the Writing Assessment: An Eye-Movement Study

LEE, Ji-won Park, Young-min

한국교원대학교 한국교원대학교

Korea Business Review 50Vol. 4No. pp.395-424 (2015)

DOI: 10.20880/kler.2015.50.4.395

Abstract

The purpose of this study was to investigate online processes of assessing writing through rater’s the eye-movements. Rater’s eyes-movements are measured during rating essays, and then their fixation counts and fixation duration between a rubric and essays are analysed that change as a time-series. Our data suggests that raters showed about 58% of fixation counts and 28% of fixation duration in rubric, and they fixed during whole rating processes 41.4 seconds at essay(AOI 2) and 16.2 seconds at rubric(AOI 1); fixation counts have been in sharp decline until they read the 10th essay, and then changed gentle decline trends following. We made a regression model of the log function base on relative fixation count and fixation duration data. We estimated a 88th essay as the predicted value that fixation rate is a zero. This means a time point that raters construct a cognitive representation of the rubric. And also we discussed that practices should require about 25 minimum rating sample essay for rater training that start long-tail trend line on our non-linear regression. This means a practice effect is slight that is represent the inclination declines on the graph.

Keywords

안구운동시선추적시계열작문 평가루브릭쓰기 채점평가자채점자예시문

References

  1. [1] [단행본] 작문교육론/박영목/역락/2008/~///
  2. [2] [단행본] 국어학과 국어교육학/박영민/채륜/2015/~///작문 평가의 평가자 신뢰도
  3. [3] [학술지(정기간행물)] 박영민/‘손글씨 설명문’과 ‘워드 설명문’ 평가 과정에서 나타나는 국어교사 눈동자 움직임의 차이/국어교육학연구/2014/49(2)/193~224/국어교육학회/
  4. [4] [학술지(정기간행물)] 박영민/쓰기 효능감 및 성별 차이가 중학생의 쓰기 수행에 미치는 효과/국어교육학연구/2007/32(28)/327~360/국어교육학회/
  5. [5] [단행본] 인지심리학/이정모/학지사/2002/~///
  6. [6] [단행본] Cognitive psychology and its implications/Anderson, J. R./WH Freeman/Times Books/Henry Holt & Co/1990/~///
  7. [7] [학술지(정기간행물)] Andrade, H. G./Understanding rubrics/Educational Leadership/1997/54(4)/14~17//
  8. [8] [단행본] Researching practice:Evaluating assessment essays/Barritt, L./College Composition and Communication/1986/315~327///
  9. [9] [학술대회논문] Boyd, P./Grading student work: Using think aloud to investigate the assessment practices of university lecturers/British Educational Research Association conference, Manchester, UK/2009//~/
  10. [10] [보고서] Reliability issues with performance assessments: A collection of papers/Colton, D. A.//1997/~/ACT Research Report Series, 97-3
  11. [11] [학술지(정기간행물)] Cumming, A./Decision making while rating ESL/EFL writing tasks : A descriptive framework/The Modern Language Journal/2002/86(1)/67~96//
  12. [12] [학술지(정기간행물)] DeRemer, M. L./Writing assessment : Raters' elaboration of the rating task/Assessing Writing/1998/5(1)/7~29//
  13. [13] [학술지(정기간행물)] Ericsson, K. A./Verbal reports as data/Psychological Review/1980/87(3)/215~//
  14. [14] [단행본] Holistic assessment of writing:Experimental design and cognitive theory/Freedman, S. W./Research on Writing: Principles and Methods/1983/75~98///
  15. [15] [학술지(정기간행물)] Hamp-Lyons, L./Communicative writing profiles : An investigation of the transferability of a Multiple-Trait scoring instrument across ESL writing assessment contexts/Language Learning/1991/41(3)/337~373//
  16. [16] [단행본] Reliability, validity, and holistic scoring: What we know and what we need to know/Huot, B./College Composition and Communication/1990/201~213///
  17. [17] [단행본] Assessing second language writing : The rater’s perspective Vol. 3/Lumley, T./Peter Lang/2005/~///
  18. [18] [학술지(정기간행물)] Lunz, M. E./Measuring the impact of judge severity on examination scores/Applied Measurement in Education/1990/3(4)/331~345//
  19. [19] [단행본] A study of the decision-making behaviour of composition markers/Milanovic, M./Performance Testing, Cognition and Assessment/1996/92~114///
  20. [20] [학술지(정기간행물)] Moskal, B. M./Scoring rubric development : Validity and reliability. Practical Assessment/Research &Evaluation/2000/7(10)/71~81//
  21. [21] [단행본] Insight from the eyes/Paulson, E. J./Heinemann Educational Books/2003/~///
  22. [22] [단행본] The Oxford handbook of eye movements/Peterson/Oxford University Press/2011/~///Eye movement and memory
  23. [23] [학술대회논문] Sakyi, A. A./Validation of holistic scoring for ESL writing assessment: How raters evaluate compositions/Fairness and validation in language assessment: selected papers from the 19th language testing research colloquium, Orlando, Florida/2000//129~152/Cambridge University Press
  24. [24] [학술지(정기간행물)] Stahl, J. A./Answering the Call for a New Psychometrics/Rasch Measurement Transactions/1991/5(1)/127~//
  25. [25] [학술대회논문] Stahl, J. A./Equating examinations that include judges (multiple facets)/annual meeting of the National Council of Measurement in Education, Chicago/1991//~/
  26. [26] [학술지(정기간행물)] Weigle, S. C./Using FACETS to model rater training effects/Language Testing/1998/15(2)/263~287//
  27. [27] [학술지(정기간행물)] White, E. M./Holisticism/College Composition and Communication/1984/35/400~409//
  28. [28] [학술지(정기간행물)] Winke, P./ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric : An eye-movement study/Assessing Writing/2015/25/37~53//
  29. [29] [학술지(정기간행물)] Wolfe, E. W./The relationship between essay reading style and scoring proficiency in a psychometric scoring system/Assessing Writing/1997/4(1)/83~106//
  30. [30] [단행본] Identifying rater effects in performance ratings/Wolfe, E. W./Performance appraisals: A critical view/2005/91~103///
  31. [31] [학술지(정기간행물)] Wolfe, E. W./Cognitive differences in proficient and nonproficient essay scorers/Written Communication/1998/15(4)/465~492//

Korea Business Review · THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION.

Print ISSN 2951-3596 · Online ISSN 2951-360X

Privacy PolicyTerms of UseContact UsKMA Homepage

TEL : 010-2672-3919 EMAIL : kbr2editor@gmail.com 계좌번호 : 신한 140-009-175743 예금주 : (사)한국경영학회 COPYRIGHT ⓒ 2023 THE KOREAN ACADEMIC SOCIETY OF BUSINESS ADMINISTRATION. All rights reserved