This study investigates the validity of the method used to measure coherence by analyzing the correlation between cohesion scores and the coherence of student texts calculated using topic modeling. For this purpose, cohesion through topic similarity was categorized into local, global, and overall text coherence. Next, the results were compared with teacher-generated coherence scores. The results showed that local cohesion was negatively correlated with the coherence scores. Global cohesion was not significantly correlated with teachers’ grading results, and overall text cohesion showed a positive correlation. This study shows that coherence is related to the consistency of the whole text and that topic modeling can be a useful tool for providing objective information about the text.