Subtitling and Eye Movements: Insights from Proficient Viewers

Background/purpose. The popularity of foreign-language films and shows with subtitles has been driven by their cost-effectiveness in comparison to audio dubbing. The primary objective of this research was to explore the reading and viewing behaviors of proficient learners in the contemporary era, characterized by significant progress in eye-tracking technologies. These advancements have provided researchers with the capability to precisely analyze eye fixation patterns, offering valuable insights into how individuals engage with written and visual content.

Materials/methods. The study’s focus was on adults between the ages of 20 and 40 years old who watched videos with English language soundtracks and Standard Arabic subtitles, as well as videos with Moroccan Arabic and Standard Arabic soundtracks and subtitles. To assess the impact of different subtitling speeds on reading behavior, participants were exposed to varying speeds. Also, a detailed questionnaire was administered to the participants after the sessions in order to obtain further information and the collected data were then analyzed using descriptive statistics.

Practical implications. The findings of this study revealed that doctoral candidates spent less time focusing on the subtitles, and some even completely ignored them. Master’s students exhibited similar eye movement patterns to the doctoral candidates, while undergraduates had longer fixation times on the subtitles.

Conclusion. This research provides valuable insights into how the speed of subtitles influences viewer behavior and has implications for language learning and the production of audiovisual content. By understanding how different subtitling speeds affect reading behavior, language learners and content creators can make informed decisions to enhance the learning experience

Keywords: Eye movement, fixation, advanced learner, viewer, subtitle

References

Al-Omari, A. (2024). The impact of artificial intelligence on the school management: a study of opportunities and challenges in Jordan. INTED2024 Proceedings, 5478-5486. https://doi.org/10.21125/inted.2024.1418

Aderdor, R. (2022). The Use of Social Media by Rural Moroccan Women: Communication, Learning and Identity [Master’s thesis, Hamad Bin Khalifa University, Qatar]. https://www.proquest.com/openview/799d73042f1bac8db2fc790341e2fe8b/1?pq-origsite=gscholar&cbl=2026366&diss=y

Amine, L. S., & Cavusgil, S. T. (1983). Mass media advertising in a developing country: the case of Morocco. International Journal of Advertising, 2(4), 317-330. https://doi.org/10.1080/02650487.1983.11104985

Bisson, M. J., Van Heuven, W. J., Conklin, K., & Tunney, R. J. (2014). Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, 35(2), 399-418. https://doi.org/10.1017/S0142716412000434

Carpenter, P. A., & Just, M. A. (1983). What your eyes do while your mind is reading. In K. Rayner (Ed.), Eye movements in reading (pp. 275-307). Academic Press. https://doi.org/10.1016/b978-0-12-583680-7.50022-9

Choy, S., Li, M., & Singh, P. (2015). The Australian doctorate curriculum: Responding to the needs of Asian candidates. International Journal for Researcher Development, 6(2), 165-182. https://doi.org/10.1108/IJRD-08-2014-0024

Danan, M. (2004). Captioning and subtitling, undervalued language learning strategies. Meta, 49(1), 67-77. https://doi.org/10.7202/009021ar

De Bruycker, W., & d’Ydewalle, G. (2003). Reading native and foreign language television subtitles in children and adults. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The Mind’s Eye,(pp. 671-684). Elsevier. https://doi.org/10.1016/B978-044451020-4/50036-0

Diao, Y., Chandler, P., & Sweller, J. (2007). The effect of written text on comprehension of spoken English as a foreign language. The American Journal of Psychology, 120(2), 237-261. https://doi.org/10.2307/20445397

Dvorghets, O. S., & Shaturnaya, Y. A. (2015). Developing students’ media literacy in the English language teaching context. Procedia-Social and Behavioral Sciences, 200, 192-198. https://doi.org/10.1016/j.sbspro.2015.08.051

D’Ydewalle, G. (2002). Foreign-language acquisition by watching subtitled television programs. Journal of Foreign Language Education and Research, 12, 59-77.

D’Ydewalle, G., & De Bruycker, W. (2007). Eye movements of children and adults while reading television subtitles. European Psychologist, 12(3), 196-205. https://doi.org/10.1027/1016-9040.12.3.196

D’Ydewalle, G., Muylle, P., & Van Rensbergen, J. (1985). Attention shifts in partially redundant information situations. In R. Groner, G. W. McConkie, & C. Menz (Eds.), Eye movements and human information processing (pp. 375-384). North-Holland.

D’Ydewalle, G., Praet, C., Verfaillie, K., & Rensbergen, J. V. (1991). Watching subtitled television: Automatic reading behavior. Communication Research, 18(5), 650-666. https://doi.org/10.1177/009365091018005005

D’Ydewalle, G., & Van Rensbergen, J. (1989). 13 Developmental Studies of Text-Picture Interactions in the Perception of Animated Cartoons with Text. Advances in Psychology, 58, 233-248. https://doi.org/10.1016/S0166-4115(08)62157-3

D’Ydewalle, G., Van Rensbergen, J., & Pollet, J. (1987). Reading a message when the same message is available auditorily in another language: The case of subtitling. In J. K. O’Regan & A. Levy-Schoen (Eds.), Eye movements from physiology to cognition (pp. 313-321). Elsevier. https://doi.org/10.1016/B978-0-444-70113-8.50047-3

Ed-Dali, R., & Elasri, K. (2020). Influence and Dissemination of Solecism among Moroccan High School and University Students. International Journal of Cognitive and Language Sciences, 14(9), 747-754.

Ghia, E. (2012). The impact of translation strategies on subtitle reading. In E. Perego (Ed.), Eye tracking in audiovisual translation (pp. 157-182). Aracne.

Hauss, K. (2021). What are the social and scientific benefits of participating at academic conferences? Insights from a survey among doctoral students and postdocs in Germany. Research Evaluation, 30(1), 1-12. https://doi.org/10.1093/reseval/rvaa018

Irwin, D. E. (2004). Fixation location and fixation duration as indices of cognitive processing. In J. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (Chap. 3, pp. 105-133). Psychology Press.

Jensema, C. J., El Sharkawy, S., Danturthi, R. S., Burch, R., & Hsu, D. (2000). Eye movement patterns of captioned television viewers. American Annals of the Deaf, 145(3), 275-285. https://doi.org/10.1353/aad.2012.0093

Krejtz, I., Szarkowska, A., & Krejtz, K. (2013). The effects of shot changes on eye movements in subtitling. Journal of Eye Movement Research, 6(5), Article 3. https://doi.org/10.16910/jemr.6.5.3

Kruger, J.-L., Hefer, E., & Matthew, G. (2014). Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2. Journal of Eye Movement Research, 7(5), Article 4. https://doi.org/10.16910/jemr.7.5.4

Kruger, J.-L., & Steyn, F. (2014). Subtitles and eye tracking, reading and performance. Reading Research Quarterly, 49(1), 105-120. https://doi.org/10.1002/rrq.59

Kruger, J.-L., Szarkowska, A., & Krejtz, I. (2015). Subtitles on the moving image: An overview of eye tracking studies. Refractory: A Journal of Entertainment Media, 25, 1-14.

Langton, S. R., Law, A. S., Burton, A. M., & Schweinberger, S. R. (2008). Attention capture by faces. Cognition, 107(1), 330-342. https://doi.org/10.1016/j.cognition.2007.07.012

Laskowska, M., Szarkowska, A., Pilipczuk, O., & Oliver, A. (2015, March 5-6). The inﬂuence of the original language of the ﬁlm on subtitle reading [Conference presentation]. III Polish Eye tracking Conference, Warsaw, Poland.

Liao, S., Yu, L., Reichle, E. D., & Kruger, J. L. (2021). Using eye movements to study the reading of subtitles in video. Scientific Studies of Reading, 25(5), 417-435. https://doi.org/10.1080/10888438.2020.1823986

Lin, J. J., Lee, Y.-H., Wang, D.-Y., & Lin, S. S. (2016). Reading subtitles and taking enotes while learning scientific materials in a multimedia environment: Cognitive load perspectives on EFL students. Journal of Educational Technology & Society, 19(4), 47-58. https://www.j-ets.net/collection/published-issues/19_4

Markham, P., & Peter, L. (2003). The inﬂuence of English language and Spanish language captions on foreign language listening/reading comprehension. Journal of Educational Technology Systems, 31(1), 331-341. https://doi.org/10.2190/BHUH-420B-FE23-ALA0

Martinho, G., Magalhães, D., & Pires, A. (2017). Consumer behavior with respect to the consumption and recycling of smartphones and tablets: An exploratory study in Portugal. Journal of Cleaner Production, 156, 147-158. https://doi.org/10.1016/j.jclepro.2017.04.039

Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University.

Mestres, E. T., & Pellicer-Sánchez, A. (2019). Young EFL learners’ processing of multimodal input: Examining learners’ eye movements. System, 80, 212-223. https://doi.org/10.1016/j.system.2018.12.002

Miller, L. R. (1969). Eye movement latency as a function of age, stimulus uncertainty, and position in the visual field. Perceptual and Motor Skills, 28(2), 631-636. https://doi.org/10.2466/pms.1969.28.2.631

Mitterer, H., & McQueen, J. M. (2009). Foreign subtitles help but native-language subtitles harm foreign speech perception. PlosOne, 4(11). Article e7785. https://doi.org/10.1371/journal.pone.0007785

Montero Perez, M., Van Den Noorgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning, A meta-analysis. System, 41(3), 720-739. https://doi.org/10.1016/j.system.2013.07.013

Noar, S. M. (2006). A 10-year retrospective of research in health mass media campaigns: where do we go from here? Journal of Health Communication, 11(1), 21-42. https://doi.org/10.1080/10810730500461059

Paek, H.-J., & Pan, Z. (2004). Spreading global consumerism: Effects of mass media and advertising on consumerist values in China. Mass Communication & Society, 7(4), 491-515. https://doi.org/10.1207/s15327825mcs0704_7

Paivio, A. (1986). Mental representations, A dual coding approach. Oxford University.

Perego, E., Del Missier, F., Porta, M., & Mosconi, M. (2010). The cognitive effectiveness of subtitle processing. Media Psychology, 13(3), 243-272. https://doi.org/10.1080/15213269.2010.502873

Price, K. (1983). Closed-captioned TV, an untapped resource. Massachusetts Association of Teachers of Speakers of Other Languages (MATESOL) Newsletter, 12(2), 1/8.

Rajendran, P., Rengarajan, T., Thangavel, J., Nishigaki, Y., Sakthisekaran, D., Sethi, G., & Nishigaki, I. (2013). The vascular endothelium and human diseases. International Journal of Biological Sciences, 9(10), 1057-1069. https://doi.org/10.7150%2Fijbs.7502

Ravindran, N., & Myers, B. J. (2012). Cultural influences on perceptions of health, illness, and disability: A review and focus on autism. Journal of Child and Family Studies, 21, 311-319. https://doi.org/10.1007/s10826-011-9477-9

Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372-422. https://psycnet.apa.org/doi/10.1037/0033-2909.124.3.372

Rayner, K., & Well, A. D. (1996). Effects of contextual constraint on eye movements in reading: A further examination. Psychonomic Bulletin & Review, 3(4), 504-509. https://doi.org/10.3758/BF03214555

Roberts, L., & Siyanova-Chanturia, A. (2013). Using eye-tracking to investigate topics in L2 acquisition and L2 processing. Studies in Second Language Acquisition, 35(2), 213-235. https://doi.org/10.1017/S0272263112000861

Robson, G. D. (2004). The closed captioning handbook. Elsevier.

Romero-Fresco, P., & Pérez, J. M. (2015). Accuracy rate in live subtitling: The NER model. In R. B. Piñero & J. D. Cintas (Eds.), Audiovisual translation in a global context (pp. 28-50). Palgrave Macmillan.

Specker, E. A. (2008). L1/L2 eye movement reading of closed captioning: A multimodal analysis of multimodal use [Doctoral dissertation, University of Arizona]. https://www.proquest.com/openview/c93734f7b91765c93e411d72960effc3/1?pq-origsite=gscholar&cbl=18750

Szarkowska, A. (2011). Text-to-speech audio description: towards wider availability of AD. The Journal of Specialised Translation, 15, 142-162. https://jostrans.soap2.ch/issue15/art_szarkowska.php

Szarkowska, A., Krejtz, I., Pilipczuk, O., Dutka, Ł., & Kruger, J. L. (2016). The effects of text editing and subtitle presentation rate on the comprehension and reading patterns of interlingual and intralingual subtitles among deaf, hard of hearing and hearing viewers. Across Languages and Cultures, 17(2), 183-204. https://doi.org/10.1556/084.2016.17.2.3

Vanderplank, R. (2016). ‘Effects of’ and ‘effects with’ captions. How exactly does watching a TV programme with same-language subtitles make a difference to language learners? Language Teaching, 49(2), 235-250. https://doi.org/10.1017/S0261444813000207

Wang, S., & Fu, R. (2020). Research on the influencing factors of the communication effect of TikTok short videos about intangible cultural heritage. In E. Markopoulos, R. Goonetilleke, A. Ho, & Y. Luximon (Eds.), Advances in Creativity, Innovation, Entrepreneurship and Communication of Design: AHFE 2020 (pp. 275-282). Springer.

Winke, P., Gass, S., & Sydorenko, T. (2010). The effects of captioning videos used for foreign language learning activities. Language Learning &Technology, 14(1), 65-86. http://llt.msu.edu/vol14num1/winkegasssydorenko.pdf

Winke, P., Sydorenko, T., & Gass, S. (2013). Factors inﬂuencing the use of captions by foreign language learners, an eye-tracking study. The Modern Language Journal, 97(1), 254-275. https://doi.org/10.1111/j.1540-4781.2013.01432.x

Woolley, G. (2010). Developing reading comprehension: Combining visual and verbal cognitive processes. The Australian Journal of Language and Literacy, 33(2), 108-125. https://doi.org/10.1007/BF03651827

Announcement

UNIBULLETIN News!

► New issue coming soon! (Volume 13 Issue 2, 2024)

► Call for Papers

UNIBULLETIN is calling for submissions. Authors are invited to submit papers from the all fields of the Education (General) and Social Sciences (General) in the international context. All submissions should be presented only in English.