“How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos”

ANNOTATION: Philip J. Guo, Juho Kim, and Rob Rubin. 2014. How video production affects student engagement: an empirical study of MOOC videos. In Proceedings of the first ACM conference on Learning @ scale conference (L@S ’14). Association for Computing Machinery, New York, NY, USA, 41–50. https://doi.org/10.1145/2556325.2566239

This study evaluated engagement via student behaviors while watching video within edX^® MOOCs. The authors used data from 6.9 million video-watching sessions across four edX^® courses: Intro to Computer Science and Programming (MIT), Statistics for Public Health (Harvard), Artificial Intelligence (Berkeley), and Solid State Chemistry (MIT). As of 2014 when this study was published, it represented the largest-scale study of video engagement to date. The goal was to determine engagement, defined by two conditions: engagement time and problem attempt (whether the student completed an assessment question after the video). The authors noted that they did not measure true “difficult-to-measure” engagement, as that would require direct observation and questioning to determine true engagement, such as whether a student was actively watching an entire video or rather was playing it in the background while multitasking.

Data was gathered using a mixed methods approach, combining quantitative data from video watching sessions from the four courses and qualitative data via interviews with six edX^® staff who were part of the production team for those courses. The quantitative data included: start and end times, video play speed, numbers of times the student pressed play/pause, and whether the student completed an assessment question after watching the video. The researchers also looked at video properties such as length, speaking rate of the instructor, video type (lecture or tutorial), production style (slides, code, Khan-style whiteboard, live classroom, studio, office desk), and finally interviews with staff for feedback and input. The six staff members interviewed included four video experts at edX^®, as well as two program managers who coordinated among edX^® and university faculty.

The results provide a helpful guide to producing video for any asynchronous course. The findings are as follows:

1. Shorter videos are more engaging, with a target of six minutes. Videos of nine minutes or longer were rarely watched more than 50% through.
2. Videos that intersperse the author’s “talking head” along with slides were more engaging than slides alone.
3. Videos that had a more personal feel, such as a low-production personal phone camera, can be more engaging than high-production-value shoots.
4. Khan-style live whiteboard type of videos are more engaging than static slides with narration. Motion and continuous visual flow are engaging.
5. Classroom lectures, even when specifically recorded for a MOOC, were not engaging. Chopping them up into well-planned parts did not make a significant difference.
6. Students are highly engaged by instructors who speak somewhat fast and have high enthusiasm.
7. Students engage different with lecture than tutorial videos, so for lectures, focus on the first-watch experience, but for tutorials, include support for rewatching and skimming.

It would be well worth looking at similar data from humanities- and arts-focused courses at edX^®, Coursera, or other large-scale MOOCs to determine whether students of non-scientific domains exhibit similar behavior and tendencies. Also, it would be telling to see whether younger viewers’ video-watching behavior in an online educational setting has or has not been affected by the revolution in short-form video such as TikTok and Instagram.