Do captions benefit everyone? Comparing the effects of automatic versus human captions on learner comprehension

Abstract

Captions have been found to benefit diverse learners, supporting comprehension, memory for content, vocabulary acquisition, and literacy (Gernsbacher, 2015). In this study we compare human-corrected captions (HCs) with automatic captions (ACs) on unscripted lecture content. We also ask whether the effects of captions on comprehension are modulated by viewers’ language proficiency.

To compare ACs with HCs we used the NLTK, jiwer packages and GloVe embedding-method with cosine similarity in Python. ACs and HCs were equivalent on measures of presentation speed (12 cps) and cosine similarity (AC = 98.7% and HC = 99.9%). On measures of accuracy, Word Error Rate (WER) was higher for HCs (15% versus 7%) but lower on content words only (10% versus 20%). One final difference was presentation format: ACs appeared on one line, one word at a time and HCs on two (Figure 1). For the experimental investigation, 80 Italian-native English speaking university students watched a 10-minute video of a seminar style lecture in English, downloaded from MIT Courseware. Participants were assigned to one of three conditions: lecture with ACs (n = 26), with HCs (n = 28) or with no captions (NCs) (n = 27). They then completed a comprehension test on the content of the lecture. Proficiency was assessed with the Michigan Test of English Language Proficiency.

To examine the relationship between comprehension, captions, and proficiency we ran generalised linear mixed models with comprehension score as the outcome variable. The best fit model included condition, proficiency, and their interaction. As can be seen from Figure 1, participants’ comprehension scores were positively correlated with language proficiency. At lower proficiency levels participants performed worse with HCs while at higher proficiency levels there was an advantage.

In this study we did not find evidence that captions always benefit everyone. Whether learners benefited from captions depended on two factors: caption type, and language proficiency. The fact that at lower proficiency levels comprehension was worse with HCs is attributed to the way text was displayed in the captions conditions (Figure 2). When language proficiency is low, tracking written information while simultaneously trying to follow speech is cognitively effortful and may backfire, hindering comprehension. High proficiency speakers don’t face this bottleneck. The advantage of HCs at the highest levels of proficiency is likely to be due to the greater accuracy of this captioning type. Our study highlights the importance of testing captioning systems with diverse learners, under different conditions, to better understand what factors are beneficial for whom and when.